Still a cpu leak in 4.2

Still a cpu leak in 4.2 SearchSearch
Author Message
lars mservice
Posted on Friday, June 27, 2003 - 08:24 am:   

Hi Bryce.
Remember i wrote a thing about smsgws.exe using 100% cpu?
Yesterday we had very high traffic on nowsms server, wich meant that sometimes there would be sent more messages per minute than our license is for(30msg/min.).
The exact same thing happend again.
smsgws.exe takes 100% cpu and i almost cannot reach it via pc anywhere or terminal service.
If i post a mesage to it, it takes nowsms about 3 minutes to progress it.

My guess is that when there is coming more messages into nowsms per minute, than the license is for, nowsms starts putting things i que.
What happends from there i dont know.
All i know id that smsgws.exe takes 100% of my cpu.

Could it be an issue of using the extrahttpargs?
Or maybe debug=yes is not good when having high traffic?

The irritating thing in this is actually the delay that occurs from a message comes in untill nowsms handles that and sends out. It takes bout 3 minutes wich is way too long for a customer to feel that it is a good service.

I know this message is a bit confusing to read, but its friday morning and i am tired.
Hope u undestand the most of it.
lars mservice
Posted on Friday, June 27, 2003 - 08:39 am:   

Just thought of another scene that could be what happend.
Last night, when it used 100% cpu, i let the server be. In other words i did not restart it or do anything.
This morning it still used 100% cpu, but when i restarted the machine the server ran/runs perfect.

So maybe its because when a large que occurs in nowsms, the smsgws.exe takes a lot of cpu. When the que is gone, then smsgws.exe ought to release this usage but it does not. So maybe thats where the bug is? It does not release the usage.
I owe it to tell you that the server has only 256Mb ram, and 2Gb free space on harddisk, so there is not much memory to work with. This can have an impact on the cpu usage.
How to test for you guys was to create a scenario using not the latest server and not to use a server with too much memory and disks.
Then set make 4 or 5 extrahttpargs and remember to set debug=yes. And you could try to make 150 different connections in the smsgw.ini so the program has to use memory to store all these in.

The try posting 50 messages to nowsms every minute. Do this in like 5 minutes and see what happends. If you only post once, nowsms will use 75% cpu in about 3 0seconds, so here it is important to really be hard on it;-).

If you do not get the cpu usage i get, then i will go out and buy a new server.


Lars
lars mservice
Posted on Friday, June 27, 2003 - 02:04 pm:   

Version 3 of the error.
I have a script that posts some messages to nowsms. It made nowsms use 100% cpu on smsgws.exe. I think its because of the amount per second but i dont know.

This is what i posted via http to nowsms:


27-06-2003 14:30:07 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:30:14 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:30:21 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:30:28 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=kiua&appnr=0001

27-06-2003 14:30:35 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=kiua&appnr=0001

27-06-2003 14:30:42 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=kiua&appnr=0001

27-06-2003 14:30:49 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:30:56 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:03 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-30_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:11 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:19 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:26 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:33 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_4x/default.asp&Sender=00000000&text=Bryce%20is%20cool% 20yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:40 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001

27-06-2003 14:31:47 GET http://xxx.xxx.xxx.xxx/?Phonenumber=0000000000&WAPURL=http://mmsc.xxx.com/wapord er/unipic/27-6-2003-14-31_x/default.asp&Sender=00000000&text=Bryce%20is%20cool%2 0yes%2E&mediacode=sss&appnr=0001
lars
Posted on Friday, June 27, 2003 - 10:06 pm:   

It still does it.
I am beginning to bring Symantec norton antivirus into suspect too.
If smsgws.exe holds a que and norton checks files that changes, could this have an impact?

Anyone outthere had the same errors???
Heeeeelp!
Anonymous
Posted on Sunday, June 29, 2003 - 03:44 pm:   

Hello Lars,
I have the same 100% CPU utilization problem and my experience is that by disabling the schedular services of windows 2000 Server the problem frequency becomes down. Also i am using 2Gigs of RAM that may cause this reduction.

Babrus
lars mservice
Posted on Sunday, June 29, 2003 - 09:13 pm:   

Hello Anonymous.

I will try to disable my tastmanager. But i am against it. A server ought to run more than one app ;-)

;)Lars
lars mservice
Posted on Sunday, June 29, 2003 - 10:02 pm:   

Found the problem!!! Fix wanted!!!

No! disabling taskmanager did not do the trick. And thank God for that. I have quite a lot of scripts running to keep my server optimal.

smsgws.exe was taking all my cpu again, and i was desperate to try stuff.
First thing i did was disabling Real-Time Protection on my Norton Antivirus Coorporate edition. AND THAT DID IT!!!!
Yes thats it! Somehow nowsms and nortons realtime protection does not like each other when traffic is high on nowsms.
why?
Could it be something with the que that nowsms create if there is more incoming than the license allows? I dont know honestly.
But Bryce, it seems to be the problem. I have tried it again now, and as soon as i turn of the realtime protection, nowsms releases the cpu and everything runs smoothly again.

I hope you see the problem with this. The outgoing messages are up to 3 minutes delayed when smsgws.exe uses all the cpu, and my customers are mad at me for not coming up with a solution.

Please install Norton antivirus and enable realtime protection. Then push lots of messages into nowsms and see what happends.


Just did it again to make absolutely sure that this was the problem. And yes it is the problem.
Disabling norton realtime took the cpu from 100% down to 24% total, and smsgws.exe to almost nothing
lars
Posted on Sunday, June 29, 2003 - 10:14 pm:   

Bryce i have sent you a mail with some log stuff in it where you can see what happends when this error occurs.
lars
Posted on Sunday, June 29, 2003 - 10:16 pm:   

I also have Anti-Trojan 5.5 installed.
Anonymous
Posted on Monday, June 30, 2003 - 05:58 am:   

Thats good lars, but i am going to more precise now. By turning off the Windows scheduled Backup utility, the problem becomes erased and all set to normal.
Bryce Norwood - NowSMS Support
Posted on Monday, June 30, 2003 - 02:41 pm:   

Hmm...

From past experience, I wouldn't think that a real time virus scanner would have an issue with anything that NowSMS is doing. There is a file created for each queued SMS, but they are small, and it would not take long for a scanner to scan them.

I've done some tests with the system running at the max limit of its license, and I also don't believe that there is anything CPU intensive going on at that stage (also there is no file activity at all going on during the license throttling, so that shouldn't affect a real-time virus scanner).

I'm wondering if instead the issue is related to there being a lot of HTTP SMSC definitions on your system ... and the virus scanner is doing real time scanning of HTTP connections. But the scanner is not prepared to handle a lot of simultaneous outbound HTTP connections simultaneously ... and that is causing the problem.

I'll have to try to dig up some information on the Norton scanner in this regard. I actually have it running on my office desktop, but I'm on the road this week, and I use Macafee on my notebook.

Check to see if Norton has any settings in the scanner related to scanning HTTP connections, as my best guess is that whatever is eating up the CPU utilisation is related to that.

-bn
lars mservice
Posted on Monday, June 30, 2003 - 09:24 pm:   

Hi Bryce

thanx for your answer.
I did not know that norton antirirus handled http requests. I thought that norton had other products for that. As far as i know it only handles files and incoming mail if you have a mailprogram installed.

Well if i cannot use norton, what other antivirus can you recommend?
It is really important for me to know. I am willing to just about anything right now.
Bryce Norwood - NowSMS Support
Posted on Tuesday, July 01, 2003 - 02:45 am:   

Lars,

I can't say that I have run an active virus scanner on a busy system. I want to try some tests with Norton when I get back to my regular office next week.

-bn
Olaf
Posted on Tuesday, July 01, 2003 - 07:22 pm:   

Hi Lars,

You may want to try the Norman Virus scanner. I've seen it perform well in busy mail systems (but not in NowSMS - yet). I'll send you an eval if you like (we're Norman Partner...)
lars mservice
Posted on Tuesday, July 01, 2003 - 08:20 pm:   

I'd be happy o try anything right now ;-)

So if you would send it to lars@mService.dk i'd be happy to try it out.
lars mservice
Posted on Wednesday, July 02, 2003 - 08:03 am:   

Okay!
I have uninstalled my, very expendsive, norton antivirus and installed the recommended Norman antivirus.
Everything seems to be running much better now.
No 100% cpu usage anymore. I will keep on stressing it to see if i can make it do it ;)

Untill so far my conclusion is that norton antivirus is not the optimal thing to run on my server with nowsms. This is very sad since this is the most used antivirus in the world, and i really trust it 100%
Now i have to get familiar with Norman and see how good it works.