NowSMS slow sending to SMPP SMSC

NowSMS slow sending to SMPP SMSC SearchSearch
Author Message
Blair
New member
Username: Bcotnam

Post Number: 29
Registered: 07-2011
Posted on Monday, March 19, 2012 - 03:17 pm:   

Hello,

I am going to explain our setup and then explain our problem.

We have 4 seperate installations of NowSMS running on different remote servers at different locations. All the locations have modems attached to send out the messages. We we have a main installatin of nowsms with modems, then have 3 SMSC routes with prefix routing to our remote locations of nowsms, which they all have modems to send out messages too.

So, we have our main installation of NowSMS running at the office locally, and then 3 remote installations of nowsms.

All our clients submit messages to our main NowSMS installation via smpp connections. ONce we receive the messages at our main installation we have prefix routing set to send certain messages to the remote location of nowsms.

It works sending prefix messages to our remote locations of nowsms route via HTTP over SMPP. The problem is our main installation of nowsms can't send messages to our remote locations very fast. In fact its extremely slow. The remote locations can only receive about 1sms a second.

When we submit a lot of messages and it matches the prefix for the remote location route we want to be able to send those messages fast.

Looks like nowsms is having a hard time sending messages to those remote installation of nowsms.

I have enabled smpp Async mode with a default value of 5 on those smsc routes for the remote locations, and messages were slow. I tried increasing this value to 20 (which is double the speed of what we want, we want to be able to send messages to all remote locations at a speed of 10msg/sec, and we are no where near that.

Our main installation we have a 100msg/sec license, and on all the remote locations of nowsms we have 50msg/sec license.

We need to be able to send messages to the remote locations of nowsms much faster. Any ideas?

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3843
Registered: 08-2008
Posted on Monday, March 19, 2012 - 09:49 pm:   

Hi Blair,

What kind of speed are the links involved?

If you do a PING from one server to another, what is the latency time for the ping?

That said, I can't imagine a high latency reducing throughput this much.

There's something else going on. It sounds like async mode is not actually active, combined with a high latency. Even then, I'd expect to see 3 to 5 messages per second.

So something just seems fundamentally off. I'd like to see debug logs so that I can understand the timing, in particular of the SMPP transmissions and acks.

Enable the SMSDEBUG.LOG.

Repeat a problem scenario, then email the SMSDEBUG.LOG, SMPPDEBUG.LOG and SMSOUT-yyyymmdd.LOG to nowsms@nowsms.com with Attention: Des in the subject line.

Please also give me an idea of a time period in particular where the transmissions are slow.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 31
Registered: 07-2011
Posted on Friday, March 23, 2012 - 01:10 pm:   

Hello Des,

I have sent you an email with our log files. The log files were too large to email, so have put them on our FTP server for you to downlaod. I sent you the credentials in the email.

I sent you the log files from our main installation of nowsms which receives all the messages. I have also given you the logs of the 3 remote locations of nowsms which we are sending only +55* prefix to.

It worked better this time. I could see around 4-5msg/sec coming into the remote sites, which is better, but when we send a blast of +55 prefix we want these messages to very fast go the remote site queues.

We submitted the messages from the HTTP interface between the hours of 8-9am EST, we sent just over 25000 messages from the user Claudino.

Plese investigate the logs and let usk now what we have to do to get the prefix rouing to work much faster. The faster we can get the prefix routing to the remote locations the better.

I haven't played around much the the setting smpp async mode, but have it set to 20 on all the SMSC connections to our remote sites in our main installation of nowsms. Not sure what this setting should be set to, but didn't see any difference when i was playing around with this setting.

Thanks for your help.
Blair
ZIM Corporation
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3860
Registered: 08-2008
Posted on Friday, March 23, 2012 - 02:25 pm:   

Hi Blair,

I took a quick look through the logs, and indeed NowSMS seems to be stuttering.

I believe you are experiencing a known SMPP performance problem that was introduced in version 2011.11.14 and fixed in 2011.12.15.

I apologize for not thinking of this sooner, as I thought you were running a more recent update because of other issues. But if I had gone back and read through the other thread (http://support.nowsms.com/discus/messages/1/70580.html), I would have realized what version you were running.

I'd suggest using either 2012.02.09 (http://www.nowsms.com/smpp-server-adds-characters-to-text-message) or the more recent special version in that thread.


--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 33
Registered: 07-2011
Posted on Friday, March 23, 2012 - 02:41 pm:   

Hi Des,

Thanks for your resposne. I can see how nowsms is stuttering when our smpp clients submit messages to us over smpp with the bug you suggested above, but the 25000 messages i sent yesterday from user 'Claudino' were sent via the nowsms web interface which is HTTP. So, is something else going on here?

98% of the time our customers are using SMPP to submit the messages so we need prefix routing for smpp messages to work well and fast to be able to support our message flow, so i will look at updating to version 2012-02-09, does this version have the ENROUTE option available so, nowsms will send the DR instead of the operator? As we need this option. I understand is doesn't have the customizable status code in this version as your stil working on it, but as long as nowsms can generate the DR thats ok until we get the newest build.

RIght now, we are running v2011-11-14 on our main server and v2012-02-16 on all our remote sites which is the beta build you gave us i think last week or the week before. The thing is when we tried the beta build you gave us (v2012-02-16)on our main server and remote sites we noticed the same performance problems.

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3861
Registered: 08-2008
Posted on Friday, March 23, 2012 - 04:36 pm:   

Hi Blair,

The performance problem that I mentioned is in the SMPP async transmitter. It doesn't matter how messages are submitted. The outbound sending stutters beginning in version 2011.11.14 and fixed in 2011.12.15.

If you are seeing that problem in other versions, then the underlying cause is different. I can tell from the debug logs that you sent, that this is the async stutter problem that was known to that particular version.

2012.02.09 does generate ENROUTE.

If it has performance issues, we'd like to see similar logs to diagnose.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 34
Registered: 07-2011
Posted on Friday, March 23, 2012 - 05:02 pm:   

Hi Des,

Ok, i understand now. My mistake. I will updated to 2012.02.09 on all nowsms installations as you suggest and try testing again. I will report back if i encounter any further probelms.

Thanks for your help.
Blair
Blair
New member
Username: Bcotnam

Post Number: 35
Registered: 07-2011
Posted on Friday, March 23, 2012 - 05:04 pm:   

I have one question. Generally, what should the SMPP async mode be set to? I just set it to double of the messages/second we want. Is that correct?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3862
Registered: 08-2008
Posted on Friday, March 23, 2012 - 05:50 pm:   


quote:

I have one question. Generally, what should the SMPP async mode be set to? I just set it to double of the messages/second we want. Is that correct?




Generally speaking, I think that is good advice, until you reach higher speeds.

We typically use a value of 50 when testing. I'm not sure there's any benefit to going any higher.
Blair
New member
Username: Bcotnam

Post Number: 40
Registered: 07-2011
Posted on Friday, April 20, 2012 - 07:42 pm:   

Hello Des,

We have installed the new build (v2012.04.04)and works fine, but we are still experiencing slow sending speed to our remote locations of nowsms.

I'll just refresh our setup.

We have a main installation of nowsms where all our customers send messages to. Then we have 2 remote locations of nowsms acting as a SMSC gateway and we send messages to these remote locations via prefix routing.

I'll send you the logs of the tests we did today. We sent 25388 messages to the main installation of nowsms via smpp. Then the main installation of nowsms is sending these messages to both remote locations with prefix routing.

The messages being submitted to the main installation of nowsms is working well, about 40msg/sec being submitted to nowsms. Nowsms is sending those messages slow to the remote locatons (about 4-6msg/sec to each site on average so, around 9-14msg/sec between the two sites which is very slow).

When i first submitted the messages, nowsms was sending the messages to the remote locatons at a good rate, about 30msg/sec between the two sites, then it really slows down to about 9-14 messages a second which is very weird.

I started sending the messages at 1:22pm. we are looking at sending between 30-40msg/sec between 3 remote sites, but we are no where near that.

Below is some examples of tests we did last week:
1. First test with 1 remote site:
Remote Site #1 - 17msg/sec (5000messages)

2. Second test with 2 remote sites
Remote Site #1 - 14msg/sec
Remote Site #2 - 14msg/Sec (5000messages)

3. Third test with 3 remote sites
Remote Site #1 - 8msg/sec
Remote Site #2 - 9msg/Sec
Remote Site #3 - 7msg/sec (5000messages)

4. Fourth test with 3 remote sites
Remote Site #1 - 2msg/sec
Remote Site #2 - 2msg/sec
Remote Site #3 - 2msg/sec (25000messages)

As you can see the more messages we submit the slow the sending is to remote sites. Those are just example, i don't have the log files for these tests.

We submitted the messages from user Claudino and all the messages were sent to +90 prefix. Text =Test 01.

Let me know if you need any further information and i will email you our logs.

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3915
Registered: 08-2008
Posted on Friday, April 20, 2012 - 08:45 pm:   

Hi Blair,

I'm going to ask one of my more experienced colleagues to take a closer look at this on Monday.

But my initial diagnosis is that the accounting callbacks seem to be the bottleneck. I believe that is what is limiting the throughput.

In particular, it seems to be the Type=SMSOUT processing.

I'd be curious if you disabled them. This can be done by adding AccountingExtended=No under the [SMSGW] header of SMSGW.INI. (Disables SMSIN and SMSOUT callbacks.)

Alternatively, there is another setting AsyncCallbackThreads=## (also [SMSGW] section). This causes NowSMS to allocate more threads for callback processing. However, the fact that you are seeing slower performance as you activate more remote sites suggests to me that this will not give you a performance impact. There is one thread allocated per connection, processing accounting callbacks for that connection as messages are sent out. If you see slower performance with more connections, it is likely that your callback has critical sections blocking access, for example with database updates. Adding more threads may just cause each accounting callback to take longer.

I could be wrong, but that's my best suspicion.

I can see from the log file that PreAuth callbacks are extremely fast ... under 1ms. SMSSend and SMSOut are considerably slower ... 30 ms is typical, but often considerably longer.

My colleagues may have other ideas, but I believe anything you can do to speed up these callbacks (or have them quickly write to a queue that can be processed outside of the critical path) will give the performance increase you need.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3924
Registered: 08-2008
Posted on Monday, April 23, 2012 - 09:35 pm:   

Hi Blair,

I wanted to give you some more follow-up.

My preliminary analysis appears to be holding up.

As a test, we created an accounting callback that simply waited for 30ms when processing SMSOut callbacks.

When running with a single connection, this limited throughput to around 32 messages per second. (32 * 30ms = 960ms ... which is fairly close to 1000ms = 1sec)

When running multiple connections in this test, the overall throughput involves, reaching the 30 to 32 messages per second range per connection (up to a point).

Similarly, using AsyncCallbackThreads=10 with a single connection bumps up throughput to over 300 messages per second.

As your throughput per connection decreases as the number of connections is added, I believe this is because your callback issue is more complex.

Our test callback just goes to sleep for 30ms.

If we have more threads processing these callbacks, the number of callbacks processed will increase.

However, I believe your callbacks are performing some database transactions, and as a result, there is blocking for exclusive access/updating occurring.

What I notice in your logs is that while 30ms is a typical delay for total time processing a callback, there are plenty of instances where it is hundreds of ms.

Example:


13:59:50:130 [61] RetrieveURL: GET /nowsms/index.php?Type=SMSOut...

13:59:50:551 [61] HttpResponseWait: Ok
13:59:50:551 [61] RetrieveURL: HTTP/1.1 200 OK

If you look for this sequence in the log, it shows you how long the callback is taking to respond. The time between "RetrieveURL: GET" and "HttpResponseWait: Ok" with a matching [number].

In this case, the response took 421ms, which is why the SMS processing is so slow.


Compare this to the delay processing PreAuth callbacks, and the response is almost instantaneous (< 1ms).

I'd suggest checking over the database activity that you perform in the callbacks. See if instead your callback can write to a log or transaction file that is processed outside of this critical path. If the callback delays can be brought down, the speed will increase accordingly.

I hope that explanation makes sense ... let me know if I can clarify further.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 41
Registered: 07-2011
Posted on Tuesday, April 24, 2012 - 02:14 pm:   

Hello,

Thanks for your reply.

Our accounting callback script looks for Type SMSSend or SMSOut or SMSIn and inserts the information into a database.

We are not processing PreAuth, only SMSSend,SMSOut and SMSIN. That probably why PreAuth are so fast compared to the rest.

I did a test where i disabled SMSOut and SMSIn Callbacks and the messages processed faster. Our script is not complex, it just gets the posted variables, check what type then inserts into a mysql database. Nowsms shouldn't be bottleneck from simple inserts.

What i'll do is try and re-structure the script and see if we get any improvements. If nowsms can't handle multiple mysql inserts without affecting performance then will will have to process these inserts from outside of nowsms.

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3925
Registered: 08-2008
Posted on Tuesday, April 24, 2012 - 07:50 pm:   

Hi Blair,

I'd suggest creating some dummy data and putting your script through some load testing outside of NowSMS, in order to better understand its performance characteristics.

Typically, when you're working with a database, and inserting or updating a large number of records, you get better overall performance grouping multiple non-time-critical updates into a single transaction, as opposed to a single transaction per update.

That's why I suggest your script creating a transaction log to record the update, and then having a separate process that reads the transaction log to do a bulk update.

Depending on the database format that you are using, you may want to disable or limit commits and/or flushes. There are lots of tips to search through depending on database format.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 42
Registered: 07-2011
Posted on Tuesday, May 01, 2012 - 02:42 pm:   

Hi,

We are currently working on changing our accounting callbacks, instead of processing everyithing to insert directly into database from our php file, we are just sending the variables we received from the accounting callback to a textfile and have a 3rp party appliacation handle inserting those records into our database. So, the database proscessing takes place outside of NowSMS.

I just have 1 question. If there anyway to disable PreAuth callbacks and enable just SMSSend,SMSOut and SMSIN? I know you can disable SMSOut and SMSIN callbacks with AccountingExtended=No, but is there a configuration option to disable only PreAuth Callbacks?

Thanks,
BLair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3944
Registered: 08-2008
Posted on Wednesday, May 02, 2012 - 08:48 pm:   

Hi Blair,

No, there is no setting to disable just the PreAuth callbacks. From what I saw in your log files though, they should not be a problem ... we were seeing a response in under 1ms.

The setting to block the other callbacks exists primarily because those callbacks didn't exist in older releases. We needed a way to quickly turn them off if someone upgraded and their callback handlers were not expecting these new callbacks.

--
Des
NowSMS Support