WAPGW active with 0 active sessions issue | Search |
NowSMS Support Forums ⬆ Now.WAP Proxy Support ⬆ |
◄ ► |
Author | Message | |||
ydnar lim New member Username: Ramil Post Number: 17 Registered: 09-2010 |
Hi Support, Today at 7:20pm we had an unusual event with Wap Gateway server application, the service was up according to the WAPGW screen but there were no active session from our users, which by that time have already reported no internet access from their handsets. The service was deactivated and reactivated several times with no changes in the service (no internet access) it was necessary to perform a restart to the server to have the service back to normal operation. Can you please check the reason of this behavior. Due to the emergency on that specific moment we were not able to get any screenshot, we did a check of the logs of this server and found no relevant error message. The only application error was related with notepad by 5:30pm (GMT-6) as a reference only I paste that error Event Type: Error Event Source: Application Hang Event Category: (101) Event ID: 1002 Date: 1/4/2011 Time: 5:24:28 PM User: N/A Computer: VAS003 Description: Hanging application notepad.exe, version 5.2.3790.1830, hang module hungapp, version 0.0.0.0, hang address 0x00000000. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 41 70 70 6c 69 63 61 74 Applicat 0008: 69 6f 6e 20 48 61 6e 67 ion Hang 0010: 20 20 6e 6f 74 65 70 61 notepa 0018: 64 2e 65 78 65 20 35 2e d.exe 5. 0020: 32 2e 33 37 39 30 2e 31 2.3790.1 0028: 38 33 30 20 69 6e 20 68 830 in h 0030: 75 6e 67 61 70 70 20 30 ungapp 0 0038: 2e 30 2e 30 2e 30 20 61 .0.0.0 a 0040: 74 20 6f 66 66 73 65 74 t offset 0048: 20 30 30 30 30 30 30 30 0000000 0050: 30 0 This is the first time that we have this specific behavior, 0 active sessions wit WAPGW service running and the first time we do not see any changes by deactivating/reactivating the service. We will wait for your feedback. Best regards, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2805 Registered: 08-2008 |
Hi Randy, I can only conclude that the system ran out of memory, possibly related to whatever happened to NOTEPAD.EXE. I have no knowledge regarding "Application Hang" events. I've never encountered one before. Did someone try to use view a large log file on that machine using NOTEPAD? -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 19 Registered: 09-2010 |
Hi Support, Thank you very much for the information. Can I also request for the removal of this thread due to some risk to the confidentially agreement of our customer. Thank you & regards, Randy | |||
ydnar lim New member Username: Ramil Post Number: 22 Registered: 09-2010 |
Hi Support, Once again, we have encountered WAPGW service active with 0 sessions. We have to restart the server itself because restarting the NOWwap service did not normalized the connections. And also there were no error logs reported on the windows event viewer regarding the issue. So we are not sure on what was the root cause of the two consecutive problem that we have encountered on NOWwap service. Can you help us on where to find the NOWwap logs that may tell us the reason for this issue or what can be the possible reason behind this. Any suggestion that might help us to avoid this kind of problem? Version: Now.WAP Proxy v2010.06.15 Thank you, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2839 Registered: 08-2008 |
Hi Randy, I would suggest enabling the WAPDEBUG.LOG, and then forwarding that log to us via email immediately after this problem is experienced. (email to nowsms@nowsms.com with Attention:Des in the subject line) Another piece of information that can be very helpful to know, is if you go into Windows Task Manager, what is the total memory in use on your system? Is it close to the system max? If so, try to determine what application is using all of the memory. If memory does not seem to be a problem, what is the memory usage, VM Size, thread count and handle count for the WAP* applications? This could provide us with some clues. When this problem occurs, does the WAP gateway still accept connections to the HTTP/WAP2 port, or are they rejected. Are you using the WTLS version? -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 23 Registered: 09-2010 |
Hi Des, Please see attached screenshot of the WAP gateway system and the current CPU/Memory usage of the server for your reference. Also can you provide us on how to enable the WAPDEBUG.LOG? And yes, we are using WTLS version. Regards, Randy | |||
ydnar lim New member Username: Ramil Post Number: 24 Registered: 09-2010 |
Hi Des, Can we have a follow up with this regards. Thank you, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2858 Registered: 08-2008 |
I'm sorry Randy, I thought I had replied, but I did not. The WAPDEBUG.LOG is enabled by editing WAPGW.INI and adding Debug=Yes under the [WAPGW] header. There is no need to restart the service after adding this setting, as it will be activated automatically. Let me give you some pointers on Task Manager as well. Check the box "Show processes from all users". This will display some additional processes that start with "wap". It would be very helpful to see what information task manager shows about these processors should the gateway enter another unstable state. Also, in "Task Manager", from the menu bar, select View/Select Columns. Add "VM Size", "Handle Count" and "Thread Count". (The options available vary a bit between versions of Windows. If "VM Size" is not in the version you are running, that's fine. The other information should be sufficient.) -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 25 Registered: 09-2010 |
Hi Des, Thank you for all the informations. We will inform you if we will encounter such problem again and we'll forward the logs. By the way enabling the WAPDEBUG.LOG will eat up more disk space faster? Thank you, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2860 Registered: 08-2008 |
Hi Randy, Sorry, I should have mentioned that. The debug log will roll over to a new file every time its size would exceed 512MB. At that time, the old debug log becomes WAPDEBUG.BAK and a new WAPDEBUG.LOG is created. This basically means 1GB of disk space use for the debug log. -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 26 Registered: 09-2010 |
Hi Des, Once again, we have encountered an issue with NowWAP gateway. The service was in unknown state. As you can see from the logs below: [customer_ip] [customer_number] [27/Jan/2011:22:02:56 -0600] "GET http://search.4shared.com/q/1/mp3" "BlackBerry8520/4.6.1.274 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/373" "Error: Timeout waiting for response from content server" [customer_ip] [customer_number] [27/Jan/2011:22:02:56 -0600] "CONNECT [customer_ip]:1974" "" "Error: Access denied. This proxy will not allow tunnelled connections to port 1974" [customer_ip] [customer_number] [27/Jan/2011:22:51:57 -0600] "CONNECT v3.cache8.c.youtube.com:554" "" "Error: Access denied. This proxy will not allow tunnelled connections to port 554" [customer_ip] [customer_number] [27/Jan/2011:22:52:00 -0600] "CONNECT v3.cache8.c.youtube.com:554" "" "Error: Access denied. This proxy will not allow tunnelled connections to port 554" We had to restart the server itself to normalized the system. Hoping for your quick response for any findings. Thank you, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2864 Registered: 08-2008 |
Hi Randy, Those errors do not appear to be relevant to anything. Do you have "Allow SSL/TLS to non-standard ports?" checked? If NOT, then those types of connections will be denied. If CHECKED, then those connections will be allowed. If CHECKED, and you are seeing those errors, then that is indeed a problem. Please clarify. What other problems were observed? Can I get access to the complete WAPDEBUG.LOG and/or WAPDEBUG.BAK? IF so, please e-mail details to nowsms@nowsms.com with "Attention: Des" in the subject line. Perhaps I can get remote access (VNC or Remote Desktop) to this system for diagnosis? Please follow-up via e-mail for details. -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 27 Registered: 09-2010 |
Hi Des, Thank you for the informations, we have already rectified the concern of the logs. For the other concern of sudden unknown state of the application, they have decided to moved the application to other physical server so if the server itself was the problem the issue shouldn't be happening again. They have also noticed that in the application window that the service is running a few minutes ago every time they remote on the server to check the status, like if the application was being restarted all the time. And we don't see any restart logs on the windows event viewer. Could this be a software bug? is there any reported issues related to this case. If yes, do we have an update patch that can resolve this problem? Current Version: Now.WAP Proxy v2010.06.15 support WTLS Hoping for your quick response. Thank you & regards, Randy | |||
ydnar lim New member Username: Ramil Post Number: 28 Registered: 09-2010 |
Hi Des, Once again we have experienced some technical issue with NowWAP application. I was monitoring the WAP gateway server via remote desktop for about 10 hours and everything seems running fine without any issue until the next login to the server and found out the the Status was Active for 2 minutes only without any logs of service restart on the event viewer on the server. Can you investigate this concern and revert back to us if there is some patch to fix this issue. Thank you, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2867 Registered: 08-2008 |
Hi Randy, I'm concerned about the restarts. If there is any way I can get remote access to this server for further troubleshooting, I would like to try to determine what is happening. If so, please e-mail details to nowsms@nowsms.com with "Attention: Des" in the subject line. I can provide an IP address from which I will be connecting so that only that IP is temporarily granted access. That said, I escalated this problem earlier this week with our engineering team. They advised me of a potential problem that was identified in some late 2009 and 2010 versions. (I say potential because the problem could not be recreated, but a case was made that a race condition could potentially occur under load, and the code was fixed for this potential occurrence.) Further reviews were conducted this week, and an additional potential problem was found in code that is activated only when the debug log is active. I'm posting the update (WTLS version) to http://www.nowsms.com/download/wapwtls20110128.zip. Hopefully fixes for the identified problems will address the issue. If the WAPDEBUG.LOG is enabled, I would also suggest disabling it for now. (Remove Debug=Yes from the WAPGW.INI file.) -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 30 Registered: 09-2010 |
Hi Des, We have update the wap service but still the same issue Status still shows the service was active for 10 Minutes ago when it should be 9-10 Hours at least. Every 10 hours I think its restarting the way we noticed it. Regards, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2875 Registered: 08-2008 |
Hi Randy, Is it possible to get remote access to the system for further diagnosis? -- Des NowSMS Support | |||
ydnar lim New member Username: Ramil Post Number: 31 Registered: 09-2010 |
Hi Des, We are communicating with top management if you can remote access to the server since we are using VPN to connect to the server. If not possible, can you give us the proper procedure so we can do further diagnostic ourselves. Thank you, Randy | |||
ydnar lim New member Username: Ramil Post Number: 33 Registered: 09-2010 |
Hi Des, The issue is getting worst. Less than 5 to 10 minutes all of a sudden the Status became Not Active. See attached screenshot. And also we have found some error logs on ERROR-20110202.LOG: 127.0.0.1 - [02/Feb/2011:22:04:02 -0600] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory" 127.0.0.1 - [02/Feb/2011:22:04:02 -0600] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory" Regards, Randy | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2880 Registered: 08-2008 |
Hi Randy, I'm concerned that there is more going on than meets the eye. Remember before when you had to reboot the server to get things working again? (And there was an event log message about memory that seemed unrelated to NowWAP?) If you have not already, I would strongly suggest a reboot. I think some application is eating up memory and/or resources. I don't think it is NowWAP. It could be. However, it appears that NowWAP is encountering problems that are forcing it to stop and restart. If NowWAP were eating up resources, those resources would be released, however that does not seem to be happening here. What I'd be doing right now is looking closely at Task Manager, monitoring overall system memory usage ... and also looking at memory usage and handle counts for the NowWAP related processes to make sure that they are not seeing unusual spikes. -- Des NowSMS Support |