120mpm limit not working 100%?

120mpm limit not working 100%? SearchSearch
Author Message
Nikos Mavrakis
New member
Username: Nmavra

Post Number: 36
Registered: 05-2007
Posted on Friday, December 04, 2009 - 11:35 am:   

Hello,
we have the 120mpm license and I'm trying to see the real throughput in peak times where it matters the most.

I see that in 10 seconds intervals, the rate that NowSMS sends to the SMSCs is not 100% as it should be.

2009-12-03 10:53:00 ,4B10A6A4.req
2009-12-03 10:53:01 ,4B10A6A5.req
2009-12-03 10:53:01 ,4B10B4AF.req
2009-12-03 10:53:02 ,4B10A6A7.req
2009-12-03 10:53:02 ,4B10B4BB.req
2009-12-03 10:53:03 ,4B10A6AA.req
2009-12-03 10:53:03 ,4B10B4BE.req
2009-12-03 10:53:04 ,4B10A6AC.req
2009-12-03 10:53:05 ,4B10A6AF.req
2009-12-03 10:53:06 ,4B10A6B2.req
2009-12-03 10:53:07 ,4B10A6B5.req
2009-12-03 10:53:08 ,4B10A6B9.req
2009-12-03 10:53:09 ,4B10B4CC.req
2009-12-03 10:53:09 ,4B10A6BB.req
2009-12-03 10:53:09 ,4B10B4D0.req
2009-12-03 10:53:10 ,4B10A6BE.req

Another example

2009-12-03 10:53:30 ,4B10A6F8.req
2009-12-03 10:53:31 ,4B10A700.req
2009-12-03 10:53:32 ,4B10B517.req
2009-12-03 10:53:32 ,4B10A705.req
2009-12-03 10:53:33 ,4B10A706.req
2009-12-03 10:53:34 ,4B10A70B.req
2009-12-03 10:53:35 ,4B10A70D.req
2009-12-03 10:53:35 ,4B10B524.req
2009-12-03 10:53:36 ,4B10A70F.req
2009-12-03 10:53:36 ,4B10B528.req
2009-12-03 10:53:37 ,4B10A711.req
2009-12-03 10:53:38 ,4B10A718.req
2009-12-03 10:53:39 ,4B10B530.req
2009-12-03 10:53:39 ,4B10A71B.req
2009-12-03 10:53:39 ,4B10B531.req

and yet another

2009-12-03 10:53:40 ,4B10A71F.req
2009-12-03 10:53:40 ,4B10B53B.req
2009-12-03 10:53:41 ,4B10A723.req
2009-12-03 10:53:42 ,4B10A726.req
2009-12-03 10:53:42 ,4B10B548.req
2009-12-03 10:53:43 ,4B10A72A.req
2009-12-03 10:53:44 ,4B10A72E.req
2009-12-03 10:53:45 ,4B10A732.req
2009-12-03 10:53:46 ,4B10A738.req
2009-12-03 10:53:47 ,4B10A73E.req
2009-12-03 10:53:47 ,4B10B553.req
2009-12-03 10:53:48 ,4B10A748.req
2009-12-03 10:53:49 ,4B10A74A.req

Please also see a random 1min interval.

2009-12-03 10:55:00 ,4B10B65A.req
2009-12-03 10:55:00 ,4B10A835.req
2009-12-03 10:55:01 ,4B10A837.req
2009-12-03 10:55:02 ,4B10B662.req
2009-12-03 10:55:02 ,4B10A838.req
2009-12-03 10:55:02 ,4B10B667.req
2009-12-03 10:55:04 ,4B10A83B.req
2009-12-03 10:55:07 ,4B10B672.req
2009-12-03 10:55:08 ,4B10A840.req
2009-12-03 10:55:09 ,4B10A845.req
2009-12-03 10:55:10 ,4B10B67B.req
2009-12-03 10:55:10 ,4B10A849.req
2009-12-03 10:55:11 ,4B10A84C.req
2009-12-03 10:55:12 ,4B10A84E.req
2009-12-03 10:55:12 ,4B10B684.req
2009-12-03 10:55:13 ,4B10A850.req
2009-12-03 10:55:14 ,4B10A855.req
2009-12-03 10:55:15 ,4B10A859.req
2009-12-03 10:55:16 ,4B10A85A.req
2009-12-03 10:55:16 ,4B10B694.req
2009-12-03 10:55:17 ,4B10A85D.req
2009-12-03 10:55:18 ,4B10A85F.req
2009-12-03 10:55:19 ,4B10A867.req
2009-12-03 10:55:20 ,4B10A86B.req
2009-12-03 10:55:21 ,4B10A86D.req
2009-12-03 10:55:22 ,4B10A86F.req
2009-12-03 10:55:23 ,4B10A877.req
2009-12-03 10:55:24 ,4B10A878.req
2009-12-03 10:55:24 ,4B10B6B0.req
2009-12-03 10:55:25 ,4B10A87A.req
2009-12-03 10:55:26 ,4B10A87B.req
2009-12-03 10:55:27 ,4B10B6B5.req
2009-12-03 10:55:27 ,4B10A882.req
2009-12-03 10:55:27 ,4B10B6BA.req
2009-12-03 10:55:28 ,4B10A885.req
2009-12-03 10:55:29 ,4B10A887.req
2009-12-03 10:55:30 ,4B10B6BC.req
2009-12-03 10:55:30 ,4B10A889.req
2009-12-03 10:55:31 ,4B10B6BF.req
2009-12-03 10:55:31 ,4B10B6C2.req
2009-12-03 10:55:32 ,4B10B6C4.req
2009-12-03 10:55:32 ,4B10A891.req
2009-12-03 10:55:33 ,4B10B6C6.req
2009-12-03 10:55:33 ,4B10B6CA.req
2009-12-03 10:55:34 ,4B10A898.req
2009-12-03 10:55:35 ,4B10A89C.req
2009-12-03 10:55:36 ,4B10B6DB.req
2009-12-03 10:55:36 ,4B10A89F.req
2009-12-03 10:55:37 ,4B10A8A2.req
2009-12-03 10:55:38 ,4B10A8A4.req
2009-12-03 10:55:39 ,4B10B6E8.req
2009-12-03 10:55:39 ,4B10A8A6.req
2009-12-03 10:55:39 ,4B10B6EA.req
2009-12-03 10:55:41 ,4B10A8A8.req
2009-12-03 10:55:42 ,4B10B6F0.req
2009-12-03 10:55:42 ,4B10B6F3.req
2009-12-03 10:55:42 ,4B10A8AA.req
2009-12-03 10:55:43 ,4B10A8AE.req
2009-12-03 10:55:44 ,4B10A8AF.req
2009-12-03 10:55:44 ,4B10B6F9.req
2009-12-03 10:55:45 ,4B10A8B1.req
2009-12-03 10:55:45 ,4B10B6FC.req
2009-12-03 10:55:46 ,4B10A8BE.req
2009-12-03 10:55:47 ,4B10A8C0.req
2009-12-03 10:55:48 ,4B10B708.req
2009-12-03 10:55:48 ,4B10A8C5.req
2009-12-03 10:55:48 ,4B10B709.req
2009-12-03 10:55:49 ,4B10A8C6.req
2009-12-03 10:55:50 ,4B10A8C7.req
2009-12-03 10:55:51 ,4B10A8CB.req
2009-12-03 10:55:52 ,4B10B71A.req
2009-12-03 10:55:52 ,4B10A8D7.req
2009-12-03 10:55:53 ,4B10A8D9.req
2009-12-03 10:55:54 ,4B10B722.req
2009-12-03 10:55:57 ,4B10A8DB.req
2009-12-03 10:55:58 ,4B10A8E1.req
2009-12-03 10:55:59 ,4B10B730.req
2009-12-03 10:55:59 ,4B10A8F5.req
total for 1 min 78 sms.

At this time, our Q folder is full with 1500+ messages waiting to be sent. At the same time, the gateway is only receiving deliry reports, which do not count against our limit, no .in messages appear since it is a bulk sent from our side.

I can also email you the complete IN & OUT log if you need it.

Any ideas why this is happening?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1493
Registered: 08-2008
Posted on Friday, December 04, 2009 - 04:48 pm:   

Hi Nikos,

Are all of these messages going out the same SMSC connection, or is it a mix of connections?

SMPP? Or is it CIMD?

I ran some tests, and I'm not seeing any major problems with NowSMS throttling. With a 120 messages per minute license, I seem to have a problem hitting 120 in a minute and average between 118 and 119. So there does appear to be a minor problem, but not as severe as you are experiencing.

That's why I'm wondering if all of these messages are going to a particular SMSC connection, and what the connection type is ... because I recall you have at least 1 CIMD connection, and we've had performance issues with it before, especially with receiving messages causing delays sending. I'm wondering if we are revisiting that issue.

--
Des
NowSMS Support
Nikos Mavrakis
New member
Username: Nmavra

Post Number: 37
Registered: 05-2007
Posted on Monday, December 07, 2009 - 08:56 am:   

Hello Des,
these are messages being sent from multiple connections, in fact the largest amount of them (99% at the given time) are being sent from 2 connections, one SMPP and one CIMD.

If I sent you a debug log of say 10mins time during this peak time of bulk sent messages, do you think you would be able to identify the cause?
Nikos Mavrakis
New member
Username: Nmavra

Post Number: 38
Registered: 05-2007
Posted on Wednesday, December 16, 2009 - 01:44 pm:   

Des, do you think you could look into this matter please? :-)
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1561
Registered: 08-2008
Posted on Wednesday, December 16, 2009 - 07:35 pm:   

Hi Nikos,

Maybe a debug log would be a good thing.

I'm wondering if we're not having a problem similar to the problem that we were having before with inbound messages .... where the accounting callbacks that NowSMS is issuing are taking too long to get a response back.

There is an SMSOut accounting callback that occurs after each outbound SMS message submission. NowSMS doesn't move on to processing the next message until that callback completes.

I'd probably investigate that first. If you want to send an SMSDEBUG.LOG to nowsms@nowsms.com with Attention: Des in the subject line, I'll look at that.

For SMPP, I'd also check to see if you have the "Async Mode" enabled. (If not, try enabling it with a small window size of 5.) For SMPP, at least, it will also have the benefit of moving the accounting callbacks out of the critical path as they are processed in a separate thread.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1569
Registered: 08-2008
Posted on Thursday, December 17, 2009 - 07:40 pm:   

Hi Nikos,

I've looked over the logs, and the performance problem seems to be centered on two CIMD connections that the majority of these queued messages are being routed to.

Whenever you submit a message and the provider returns error 9 ... the provider waits between 1 and almost 2 seconds to return the error response.

If the provider accepts the message, they return a response much more quickly.

Over a CIMD connection, NowSMS submits messages in a synchronous fashion. It doesn't submit the next message until it receives an acknowledgment that the SMSC has received the previous message.

These "Error 9" rejections are outnumbering messages that are accepted 2 to 1. And the 1 to 2 second delay in the error response is severely impacting performance.

The only solutions that I could see are ...

1.) Identifying and preventing "error 9" situations so that there are not as many of these messages that end up in the queue.

2.) Adding additional CIMD connections. These delays are only affecting throughput for a connection that encounters an "error 9". Other connections are able to keep sending.

--
Des
NowSMS Support