WAPGW active with 0 active sessions issue

WAPGW active with 0 active sessions issue SearchSearch
Author Message
ydnar lim
New member
Username: Ramil

Post Number: 17
Registered: 09-2010
Posted on Wednesday, January 05, 2011 - 03:33 am:   

Hi Support,

Today at 7:20pm we had an unusual event with Wap Gateway server
application, the service was up according to the WAPGW screen but there
were no active session from our users, which by that time have already
reported no internet access from their handsets.

The service was deactivated and reactivated several times with no
changes in the service (no internet access) it was necessary to
perform a restart to the server to have the service back to normal operation.

Can you please check the reason of this behavior.

Due to the emergency on that specific moment we were not able to get any
screenshot, we did a check of the logs of this server and found no
relevant error message. The only application error was related with
notepad by 5:30pm (GMT-6) as a reference only I paste that error

Event Type: Error
Event Source: Application Hang
Event Category: (101)
Event ID: 1002
Date: 1/4/2011
Time: 5:24:28 PM
User: N/A
Computer: VAS003
Description:
Hanging application notepad.exe, version 5.2.3790.1830, hang module
hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 48 61 6e 67 ion Hang
0010: 20 20 6e 6f 74 65 70 61 notepa
0018: 64 2e 65 78 65 20 35 2e d.exe 5.
0020: 32 2e 33 37 39 30 2e 31 2.3790.1
0028: 38 33 30 20 69 6e 20 68 830 in h
0030: 75 6e 67 61 70 70 20 30 ungapp 0
0038: 2e 30 2e 30 2e 30 20 61 .0.0.0 a
0040: 74 20 6f 66 66 73 65 74 t offset
0048: 20 30 30 30 30 30 30 30 0000000
0050: 30 0


This is the first time that we have this specific behavior, 0 active
sessions wit WAPGW service running and the first time we do not see any
changes by deactivating/reactivating the service.

We will wait for your feedback.

Best regards,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2805
Registered: 08-2008
Posted on Wednesday, January 05, 2011 - 09:45 pm:   

Hi Randy,

I can only conclude that the system ran out of memory, possibly related to whatever happened to NOTEPAD.EXE.

I have no knowledge regarding "Application Hang" events. I've never encountered one before.

Did someone try to use view a large log file on that machine using NOTEPAD?

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 19
Registered: 09-2010
Posted on Tuesday, January 11, 2011 - 11:08 pm:   

Hi Support,

Thank you very much for the information.

Can I also request for the removal of this thread due to some risk to the confidentially agreement of our customer.

Thank you & regards,
Randy
ydnar lim
New member
Username: Ramil

Post Number: 22
Registered: 09-2010
Posted on Tuesday, January 18, 2011 - 12:56 am:   

Hi Support,

Once again, we have encountered WAPGW service active with 0 sessions. We have to restart the server itself because restarting the NOWwap service did not normalized the connections.

And also there were no error logs reported on the windows event viewer regarding the issue. So we are not sure on what was the root cause of the two consecutive problem that we have encountered on NOWwap service.

Can you help us on where to find the NOWwap logs that may tell us the reason for this issue or what can be the possible reason behind this. Any suggestion that might help us to avoid this kind of problem?


Version: Now.WAP Proxy v2010.06.15




Thank you,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2839
Registered: 08-2008
Posted on Wednesday, January 19, 2011 - 12:29 am:   

Hi Randy,

I would suggest enabling the WAPDEBUG.LOG, and then forwarding that log to us via email immediately after this problem is experienced. (email to nowsms@nowsms.com with Attention:Des in the subject line)

Another piece of information that can be very helpful to know, is if you go into Windows Task Manager, what is the total memory in use on your system? Is it close to the system max? If so, try to determine what application is using all of the memory.

If memory does not seem to be a problem, what is the memory usage, VM Size, thread count and handle count for the WAP* applications? This could provide us with some clues.

When this problem occurs, does the WAP gateway still accept connections to the HTTP/WAP2 port, or are they rejected.

Are you using the WTLS version?

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 23
Registered: 09-2010
Posted on Saturday, January 22, 2011 - 05:43 am:   

Hi Des,

Please see attached screenshot of the WAP gateway system and the current CPU/Memory usage of the server for your reference. Also can you provide us on how to enable the WAPDEBUG.LOG?

And yes, we are using WTLS version.



Regards,
Randyscreenshot
ydnar lim
New member
Username: Ramil

Post Number: 24
Registered: 09-2010
Posted on Wednesday, January 26, 2011 - 03:05 am:   

Hi Des,

Can we have a follow up with this regards.



Thank you,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2858
Registered: 08-2008
Posted on Wednesday, January 26, 2011 - 10:50 pm:   

I'm sorry Randy, I thought I had replied, but I did not.

The WAPDEBUG.LOG is enabled by editing WAPGW.INI and adding Debug=Yes under the [WAPGW] header.

There is no need to restart the service after adding this setting, as it will be activated automatically.

Let me give you some pointers on Task Manager as well.

Check the box "Show processes from all users". This will display some additional processes that start with "wap". It would be very helpful to see what information task manager shows about these processors should the gateway enter another unstable state.

Also, in "Task Manager", from the menu bar, select View/Select Columns. Add "VM Size", "Handle Count" and "Thread Count". (The options available vary a bit between versions of Windows. If "VM Size" is not in the version you are running, that's fine. The other information should be sufficient.)

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 25
Registered: 09-2010
Posted on Thursday, January 27, 2011 - 08:07 am:   

Hi Des,

Thank you for all the informations. We will inform you if we will encounter such problem again and we'll forward the logs.

By the way enabling the WAPDEBUG.LOG will eat up more disk space faster?



Thank you,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2860
Registered: 08-2008
Posted on Thursday, January 27, 2011 - 11:41 pm:   

Hi Randy,

Sorry, I should have mentioned that. The debug log will roll over to a new file every time its size would exceed 512MB. At that time, the old debug log becomes WAPDEBUG.BAK and a new WAPDEBUG.LOG is created.

This basically means 1GB of disk space use for the debug log.

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 26
Registered: 09-2010
Posted on Friday, January 28, 2011 - 09:04 am:   

Hi Des,

Once again, we have encountered an issue with NowWAP gateway. The service was in unknown state. As you can see from the logs below:


[customer_ip] [customer_number] [27/Jan/2011:22:02:56 -0600] "GET
http://search.4shared.com/q/1/mp3" "BlackBerry8520/4.6.1.274 Profile/MIDP-2.0
Configuration/CLDC-1.1 VendorID/373" "Error: Timeout waiting for response from content server"
[customer_ip] [customer_number] [27/Jan/2011:22:02:56 -0600] "CONNECT [customer_ip]:1974" ""
"Error: Access denied. This proxy will not allow tunnelled connections to port 1974"

[customer_ip] [customer_number] [27/Jan/2011:22:51:57 -0600] "CONNECT v3.cache8.c.youtube.com:554"
"" "Error: Access denied. This proxy will not allow tunnelled connections to port 554"
[customer_ip] [customer_number] [27/Jan/2011:22:52:00 -0600] "CONNECT v3.cache8.c.youtube.com:554"
"" "Error: Access denied. This proxy will not allow tunnelled connections to port 554"


We had to restart the server itself to normalized the system.




Hoping for your quick response for any findings.



Thank you,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2864
Registered: 08-2008
Posted on Friday, January 28, 2011 - 02:21 pm:   

Hi Randy,

Those errors do not appear to be relevant to anything.

Do you have "Allow SSL/TLS to non-standard ports?" checked?

If NOT, then those types of connections will be denied.

If CHECKED, then those connections will be allowed.

If CHECKED, and you are seeing those errors, then that is indeed a problem. Please clarify.

What other problems were observed?

Can I get access to the complete WAPDEBUG.LOG and/or WAPDEBUG.BAK? IF so, please e-mail details to nowsms@nowsms.com with "Attention: Des" in the subject line.

Perhaps I can get remote access (VNC or Remote Desktop) to this system for diagnosis? Please follow-up via e-mail for details.

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 27
Registered: 09-2010
Posted on Saturday, January 29, 2011 - 05:57 am:   

Hi Des,

Thank you for the informations, we have already rectified the concern of the logs.


For the other concern of sudden unknown state of the application, they have decided to moved the application to other physical server so if the server itself was the problem the issue shouldn't be happening again.

They have also noticed that in the application window that the service is running a few minutes ago every time they remote on the server to check the status, like if the application was being restarted all the time. And we don't see any restart logs on the windows event viewer. Could this be a software bug? is there any reported issues related to this case. If yes, do we have an update patch that can resolve this problem?


Current Version:

Now.WAP Proxy v2010.06.15 support WTLS



Hoping for your quick response.



Thank you & regards,
Randy
ydnar lim
New member
Username: Ramil

Post Number: 28
Registered: 09-2010
Posted on Saturday, January 29, 2011 - 03:01 pm:   

Hi Des,

Once again we have experienced some technical issue with NowWAP application. I was monitoring the WAP gateway server via remote desktop for about 10 hours and everything seems running fine without any issue until the next login to the server and found out the the Status was Active for 2 minutes only without any logs of service restart on the event viewer on the server.

Can you investigate this concern and revert back to us if there is some patch to fix this issue.


Thank you,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2867
Registered: 08-2008
Posted on Saturday, January 29, 2011 - 03:44 pm:   

Hi Randy,

I'm concerned about the restarts. If there is any way I can get remote access to this server for further troubleshooting, I would like to try to determine what is happening. If so, please e-mail details to nowsms@nowsms.com with "Attention: Des" in the subject line. I can provide an IP address from which I will be connecting so that only that IP is temporarily granted access.

That said, I escalated this problem earlier this week with our engineering team. They advised me of a potential problem that was identified in some late 2009 and 2010 versions. (I say potential because the problem could not be recreated, but a case was made that a race condition could potentially occur under load, and the code was fixed for this potential occurrence.)

Further reviews were conducted this week, and an additional potential problem was found in code that is activated only when the debug log is active.

I'm posting the update (WTLS version) to http://www.nowsms.com/download/wapwtls20110128.zip.

Hopefully fixes for the identified problems will address the issue.

If the WAPDEBUG.LOG is enabled, I would also suggest disabling it for now. (Remove Debug=Yes from the WAPGW.INI file.)

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 30
Registered: 09-2010
Posted on Wednesday, February 02, 2011 - 03:03 pm:   

Hi Des,

We have update the wap service but still the same issue Status still shows the service was active for 10 Minutes ago when it should be 9-10 Hours at least.

Every 10 hours I think its restarting the way we noticed it.


Regards,
Randy
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2875
Registered: 08-2008
Posted on Wednesday, February 02, 2011 - 03:40 pm:   

Hi Randy,

Is it possible to get remote access to the system for further diagnosis?

--
Des
NowSMS Support
ydnar lim
New member
Username: Ramil

Post Number: 31
Registered: 09-2010
Posted on Thursday, February 03, 2011 - 03:47 am:   

Hi Des,

We are communicating with top management if you can remote access to the server since we are using VPN to connect to the server. If not possible, can you give us the proper procedure so we can do further diagnostic ourselves.

Thank you,
Randy
ydnar lim
New member
Username: Ramil

Post Number: 33
Registered: 09-2010
Posted on Thursday, February 03, 2011 - 04:23 am:   

Hi Des,

The issue is getting worst. Less than 5 to 10 minutes all of a sudden the Status became Not Active. See attached screenshot.

And also we have found some error logs on ERROR-20110202.LOG:


127.0.0.1 - [02/Feb/2011:22:04:02 -0600] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory"
127.0.0.1 - [02/Feb/2011:22:04:02 -0600] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory"



Regards,
Randywap errorwap issue
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2880
Registered: 08-2008
Posted on Thursday, February 03, 2011 - 12:51 pm:   

Hi Randy,

I'm concerned that there is more going on than meets the eye.

Remember before when you had to reboot the server to get things working again? (And there was an event log message about memory that seemed unrelated to NowWAP?)

If you have not already, I would strongly suggest a reboot.

I think some application is eating up memory and/or resources.

I don't think it is NowWAP. It could be. However, it appears that NowWAP is encountering problems that are forcing it to stop and restart. If NowWAP were eating up resources, those resources would be released, however that does not seem to be happening here.

What I'd be doing right now is looking closely at Task Manager, monitoring overall system memory usage ... and also looking at memory usage and handle counts for the NowWAP related processes to make sure that they are not seeing unusual spikes.

--
Des
NowSMS Support

Add Your Message Here, or click here to start a new topic.
Post:
Bold text Italics Underline Create a hyperlink Insert a clipart image
Options: Automatically activate URLs in message
Action: