Memory Exception Error during malloc - insufficient memory

Memory Exception Error during malloc - insufficient memory SearchSearch
Author Message
Adi Wong
New member
Username: Adiwong

Post Number: 20
Registered: 05-2011
Posted on Wednesday, March 21, 2012 - 03:30 pm:   

It seems that after we receive the following error line, NowWap will just hang there and handles 0 session. Only after a 'stop' and 'start' of the process, it begins to handles traffic again.

ERROR-[DATE].log

127.0.0.1 - [xxxxxxxxxxx] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory"

Monitoring from the Task Manager, it seems that wap3gxs-wtls.exe is growing slowly.

I have 2 servers, both are having 4GB with only 750 active license. I believe that should be more than sufficient, but somehow the file just keeps growing. Kindly advise on how to tackle this.

Thank you in advance.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3848
Registered: 08-2008
Posted on Wednesday, March 21, 2012 - 09:14 pm:   

Hi Adi,

That is very concerning. How much memory is wap3gxs-wtls.exe using? At what pace does it seem to be growing? (Can you give me thread count and handle counts as well.)

Also, what version of NowWAP are you running?

And a side question ... would it be feasable to move/redirect WTLS sessions to a separate server?

At present, our WTLS implementation requires the entire process to run in 32-bit mode. Without WTLS, everything can run in 64-bit mode, which makes a larger address space available. (That said, this should not be an issue for 750 sessions, there must be something else occurring.)

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 21
Registered: 05-2011
Posted on Thursday, March 22, 2012 - 02:30 pm:   

Hi Des,

Thank you for your response. Currently, I do not have ways to track the memory usage over time. We observe the memory for wap3gxs-wtls.exe as follows:-

Server 1 - Process Hangs, Restarted NowWap 24 hours ago
Yesterday Memory Usage : 180mb
Today Memory Usage : 180mb

Server 2 - Last restart was 6 days ago
Yesterday Memory Usage : 800mb
Today Memory Usage : 1000MB

Both server are load balance, handling approximately 480 current user.

NowWap Version: v2010.06.15

Running under Windows Server 2003 (32 bit)

Seems like the memory usage is very uneven although they may be handling the same amount of traffic. And Server 1 is not observing any increase at the moment.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3859
Registered: 08-2008
Posted on Thursday, March 22, 2012 - 09:56 pm:   

Hi Adi,

Server 2 memory usage is definitely higher than it should be. I could see it peaking that high, but not sustained.

I've discussed this with my colleagues, and as a starting point, we're going to ask you to update to v2011.11.11.

A WTLS Version is at www.nowsms.com/download/wapwtls20111111.zip

Start doing a two or three times a day memory usage check.

It is possible that this update may resolve the issues, as it did reduce memory usage compared to 2010 versions.

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 22
Registered: 05-2011
Posted on Monday, March 26, 2012 - 05:20 pm:   

Thank you for the suggestion, I am arranging them to be upgraded. I would like to know if you have recommended steps for backup in case we want to perform a rollback?

At the same time, we encounter a freeze today. Unfortunately there is no error observed this time and memory usage is still low. Again after restart of the process it is fine again.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3875
Registered: 08-2008
Posted on Wednesday, March 28, 2012 - 12:33 am:   

Hi Adi,

To rollback an update, run the install for the older version.

If you do not have that install, just back up Program Files\NowWAP (or Program Files (x86)) and restore those contents if you need to rollback. (Stop the service and exit the program to allow such a rollback.)

There were sporadic reports of random freezes in NowWAP 2010 versions. While we were never able to recreate the problem, we suspected a particular code change, and removed that change in 2011 So I would strongly recommend applying the update.

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 23
Registered: 05-2011
Posted on Thursday, March 29, 2012 - 09:29 am:   

Hi Des,

We have upgraded to v2011.11.11 but unfortunately we are still encountering the same error and as you can see from the log, it freeze for around 34 minutes until we manually restart the service.

From other WAPGW.logs and RADIUS.log we also see the same freeze whereby no traffic is written during this period of time.

127.0.0.1 - [29/Mar/2012:07:27:00 +0400] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory"
127.0.0.1 - [29/Mar/2012:07:27:00 +0400] "ERROR GWmalloc" "" "Memory Exception Error during malloc - insufficient memory"
xxxxxxx - [29/Mar/2012:08:01:23 +0400] "CONNECT mmo.[websiteurl].com:443" "" "Error: Unable to determine phone number from dial-in server."

Kindest regards,
Adi Wong
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3880
Registered: 08-2008
Posted on Thursday, March 29, 2012 - 04:03 pm:   

Hi Adi,

There is obviously something very unexpected and unusual going on.

Is it possible for us to arrange remote access (RDP, VNC, other) for us to monitor and diagnose the problem?

Please e-mail nowsms@nowsms.com with Attention: Des in the subject line so that we can accelerate resolving this problem. I can provide you with details of a fixed IP from which we will connect, and we can arrange other particulars.

As a sanity check, is there a virus scanner actively running on the system? If there is, it could be the problem. We have seen numerous issues where virus scanners are not designed for large number of connections, and they end up leaking memory.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3881
Registered: 08-2008
Posted on Thursday, March 29, 2012 - 04:12 pm:   

Hi Adi,

An additional thought. When running the 32-bit version, to make more memory address space available, running Windows with the /3GB switch is advisable.

The following Microsoft Knowlegebase article explains how to do this:

http://technet.microsoft.com/en-us/library/bb124810(v=exchg.65).aspx

If you cannot migrate to a 64-bit OS, this configuration parameter increases addressable memory in the 32-bit OS from 2GB to 3GB.

Also, can you give me an idea of typical max active users and max active requests that you see reported by NowWAP?

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3883
Registered: 08-2008
Posted on Thursday, March 29, 2012 - 04:19 pm:   

I notice the discussion board software truncated the Microsoft Knowledgebase URL link because of parentheses characters.

Here is an alternative URL:

http://technet.microsoft.com/en-us/library/bb124810%28v=exchg.65%29.aspx
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3897
Registered: 08-2008
Posted on Monday, April 09, 2012 - 02:18 pm:   

Hi Adi,

I wanted to offer some additional follow-up.

Our engineers have been looking for any potential reasons for the problems you are experiencing.

As I mentioned earlier, the top two suggestions are:

1.) Avoid running a virus scanner on the PC running NowWAP (particularly any type of real-time virus scanner), as most scanners are not prepared to keep up with the amount of traffic generated.

2.) On 32-bit Windows systems, use the /3GB switch with Windows as per http://technet.microsoft.com/en-us/library/bb124810%28v=exchg.65%29.aspx

--

That said, we have done some more engineering investigation. Through this, we did discover a potential scenario that could lead to excessive memory use, and an inability to recover from this problem once encountered.

A fix for this, in a WTLS build is at http://www.nowsms.com/download/wapwtls20120330.zip

This updated version also fixes the issue where WTLS versions of NowWAP are unable to take advantage of the increased memory address space of 64-bit mode. So, I would also encourage migration to a 64-bit platform. (On a 64-bit platform, wap3gxs64.exe handles WAP2 and non-WTLS WAP1 connections, wap3gxs-wtls.exe handles only WAP1 WTLS. On 32-bit platforms, wap3gxs-wtls.exe handles all connections.)

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 24
Registered: 05-2011
Posted on Friday, April 20, 2012 - 12:57 pm:   

Hi Des,

Thanks for the support. I confirm that we currently do not have any virus scanner running.

Upgrading the platform to a 64-bit will not be possible as it is a hardware change.

We will proceed with your recommendation with /3GB and also install with the WTLS latest built.

Kindest regards,
Adi Wong
Adi Wong
New member
Username: Adiwong

Post Number: 25
Registered: 05-2011
Posted on Thursday, August 16, 2012 - 09:47 pm:   

We have done the upgrade and also recommended 3GB setting but unfortunately we still faced the same issue again on July after months of stability.

127.0.0.1 - [23/Jul/2012:08:22:42 +0400] "ERROR GWmalloc" "" "Memory Exception Error during realloc - insufficient memory - 38602 blocks, attempted 1293 bytes"
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4073
Registered: 08-2008
Posted on Wednesday, August 22, 2012 - 12:40 am:   

Hi Adi,

Has that been the only issue during this timeframe?

Did I previously suggest to you changing the keep-alive settings?

On the “HTTP” page of the NowWAP configuration, there are two “keep-alive” settings. Change both values to 0, and click Apply. (It is not necessary to restart the service.)

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 26
Registered: 05-2011
Posted on Thursday, August 23, 2012 - 10:23 pm:   

Hi Des,

Before I try this setting, can I know what is the impact to the subscriber. Doesn't that means if a subscriber is idle, they are disconnected and requires to make new connection?

Also previously mentioned about the 64-bit, my customer are considering it, but it will require a migration to a new machine, how could we ensure that that it will actually solve the issue after proposing this?

Kindest regards,
Adi Wong
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4077
Registered: 08-2008
Posted on Friday, August 24, 2012 - 11:38 am:   

Hi Adi,

Subscribers will not see any change with this setting, perhaps a small number of milliseconds slower per request, but nothing perceptible.

To make sure I am not jumping to the wrong solution regarding the 64 bit recommendation, can you monitor statistics for several days and advise typical values for peak active users and requests.

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 27
Registered: 05-2011
Posted on Tuesday, August 28, 2012 - 04:38 pm:   

Hi Des,

They are hitting their limit on their license for some day with 750 Max Active Users at some day (750 x 2). Two servers serving as WapGW on 50/50 load balance.

A sample of a day logs as follows:-

751 - Max Active Users
424 - Max Active Requests
14163 - Total Unique MSISDN
61967 - Total Sessions
2957931 - Total Requests
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4079
Registered: 08-2008
Posted on Tuesday, August 28, 2012 - 08:12 pm:   

Hi Adi,

Ok ... that does give me one other possibility to check.

At one time in the past, we did confirm a memory leak when hitting max users. That was fixed, but perhaps the problem was not fully addressed. We will run some tests on this scenario.

--
Des
NowSMS Support
Adi Wong
New member
Username: Adiwong

Post Number: 28
Registered: 05-2011
Posted on Wednesday, September 26, 2012 - 05:18 pm:   

Hello Des,

Would like to know if you manage to find anything wrong with the memory leak when hitting max users.

Kindest regards,
Adi Wong

Add Your Message Here, or click here to start a new topic.
Post:
Bold text Italics Underline Create a hyperlink Insert a clipart image
Options: Automatically activate URLs in message
Action: