High CPU load

High CPU load SearchSearch
Author Message
Jon Berres
New member
Username: Jberres

Post Number: 2
Registered: 04-2009
Posted on Wednesday, June 10, 2009 - 02:33 pm:   

We are running the newest version of NowSMS (v2009.05.08) and we have noticed that the CPU usage on our server is running very high at times. Normally it runs at about 3.4 ghz sending about 4000 MMS & 5000 SMS messages a day. At times however the mmsc.exe process on the server will run at 100% using aprox 11.5 ghz. If we reboot the server or end the mmsc.exe task then the processor load goes back down. However, we have had several times now where the CPU load will go down after ending the mmsc.exe task but then 5 minutes later the CPU will be back at 100%. It will keep doing this for 5-7 more times after ending the task and then it will run fine for a few days. This problem has been repeating several days a week for about a month and a half now. We tried disabling the anti-virus and it made no difference.

We posted a similar issue previously (April 27, 09') which we where told that the new 2009 version was supposed to fix issues like this, however it has not.
This server is running on a VMware ESX server running two Xeon Quad cores. The MMSC virtual machine has 4 cores assigned to it(total 11.3 ghz).

Thanks,
Jon
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 922
Registered: 08-2008
Posted on Wednesday, June 10, 2009 - 03:37 pm:   

Hi Jon,

Refresh my memory on this ... this system is running as an MMSC, correct?

Are you using MMS routing callbacks by any chance?

The most common problem with this has been routing callbacks that return an invalid route. This results in queueing problems that lead to 100% utilisation.

These queued messages are in the VASPQ subdirectory. They are NOT messages queued for local recipients, but they are messages queued for remote recipients ("MMSC Routing" definitions).

The next time this problem surfaces, copy the contents of the VASPQ directory to another location so that we can analyse the content to get a better idea of what is going wrong.

Stopping the MMSC.EXE process is likely to make things worse. Each time it starts up, it is going to analyse the messages in the VASPQ directory ... I'm suspecting there may be quite a few, and that they are possibly misqueued.

How many "MMSC Routing" definitions do you have?

What are there types? What are the routing rules for which they are used?

--
Des
NowSMS Support
Jon Berres
New member
Username: Jberres

Post Number: 3
Registered: 04-2009
Posted on Wednesday, June 10, 2009 - 04:27 pm:   

Hi Des,

Yes our system is running as an MMSC. It only has one route and is being delivered as "direct delivery" (internal mmsc). It routes the messages using MM4(SMTP). It is 3GGP MMS Version 5.5.0.

We will watch for the server to peg out again and copy the contents of the VASPQ folder for you to view. Are there any other logs or files that would help you with this?

Thanks,
Jon
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 923
Registered: 08-2008
Posted on Wednesday, June 10, 2009 - 06:33 pm:   

Hi Jon,

Every other 100% utilisation problem in MMSC.EXE over the past couple of years has been related to problems with the outbound "MMSC Routing" queue.

However ... I went back and looked at the other thread and the directory scan that I asked you to send me. And I don't see any characteristics that would point to a problem in this area.

However, I did notice something unusual. Your previous thread was posted on April 27, when you killed the MMSC.EXE process multiple times.

Your directory scan shows some temporary files that didn't get cleaned up properly. While there are some situations where this will happen ... particularly if you kill the MMSC.EXE process ... in this case, it suggests that each time the MMSC.EXE process was terminated, NowSMS was performing content adaptation on the same message.

This suggest that perhaps NowSMS is going into a processing loop of some sort when processing a particular message.

This is only a theory. It is possible that something about the message structure is triggering an error that simply causes the temporary files not to be deleted ... which would be a minor annoyance, not the sort of critical problem that you are encountering.

Or it could just be coincidence (something else could be slowing the server down so much that clients are timing out and making repeated requests for the same message). But it looks very suspicious.

Here's what I'd suggest ...

Let's not forget the VASPQ scenario. But I'm suspicious about files in the MMSCDATA\TEMP directory.

When was the last time you had to kill MMSC.EXE?

Can you send me the contents of the MMSCDATA\TEMP directory so that I can look at it? (You can reply to the previous e-mail or send it to nowsms@nowsms.com with "Attention: Des" in the subject line ... but also please post a reply in this thread, as that will do better to trigger me to look for the e-mail.) I'm likely then going to ask you for several raw MMS message files, based upon what I see in MMSCDATA\TEMP.

--
Des
NowSMS Support
Jon Berres
New member
Username: Jberres

Post Number: 4
Registered: 04-2009
Posted on Wednesday, June 10, 2009 - 08:05 pm:   

Hi Des,

I am e-mailing you what is currently in our MMSCDATA\TEMP directory. We had to kill the MMSC.exe task twice today.

Thanks,
Jon
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 930
Registered: 08-2008
Posted on Wednesday, June 10, 2009 - 09:11 pm:   

Hi Jon,

I don't think the e-mail has made it to our system.

I'm going to reply to a previous e-mail with details on an FTP site drop box that can be used as an alternative.

--
Des
NowSMS Support