Increase threading for SMPP client callbacks

Increase threading for SMPP client callbacks SearchSearch
Author Message
Hareeharan Kandavel
New member
Username: Haree

Post Number: 22
Registered: 12-2012
Posted on Wednesday, September 18, 2013 - 04:11 pm:   

Hi Support,

We would like to cap messages to one of our SMPP client connections to 100 Msgs/ sec. After applying the changes, 2 tests were done with different configuration settings:

Test-1 (with Accounting callbacks)

Window size - 50
Transceivers - 5
Throughput achieved : 11.4 Msg/sec
Throttling errors : Yes

Test-2 (without Accounting callbacks)

Window size - 50
Transceivers - 5
Throughput achieved : 50.5 Msg/sec
Throttling errors : No

The test results suggests that with the callback settings ON the throughput was reduced by 5 times.

We suspect that the callbacks are single threaded which is affecting the throughput. We also see that the call backs are processed in such a fashion where a Pre-Auth call is sent first followed by the Pre_auth call for the second call. This increases the overall wait time for all the calls.

During our first test, we checked our IIS logs to see how long each HTTP request takes for both types of callbacks. These are averaging out at more-or-less 21 milliseconds per request, which to us shows that the problem in the processing is lying within NowSMS’s Callback mechanism.

We would prefer this mechanism work in a multi-threaded environment that submits the requests in a “pre-Auth/SMS Send” coupling fashion. This should drastically increase the throughput from an SMPP client to SMPP Servers perspective and will also shift the “heavy lifting” of the work back to the receiving end of the HTTP requests.

Meanwhile we will continue to do some more tests with the configuration changes on our end to achieve the maximum send limit of 100 Msgs/sec without the Accounting callbacks.

Hoping to hear from you soon.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4661
Registered: 08-2008
Posted on Wednesday, September 18, 2013 - 09:37 pm:   

Hi Haree,

I'm not really sure where to start...so my first inclination is to take a step back.

Callbacks are already multithreaded to the extent that we have found to be practical.

That said, there are some critical paths ... and you need to ensure that your callbacks do not become significantly slower when they are called simultaneously by multiple threads.

On the subject of critical paths, you seem focused on the PreAuth. If you are processing HTTP submissions to a large list of recipients, then this can be a critical path if you are using one of the following configuration parameters: SMSAccountingMustSetRoute=Yes or SMSAccountingPreAuthPerRecip=Yes. The presence of either setting means that NowSMS will issue one PreAuth callback for each recipient ... all callbacks must complete before the first recipient is processed.

Another critical path is the SMSOut callback. There is a separate thread processing the callback compared to the one that is filling the transmit queue (and there are separate instances of both threads for each connection). But entries are only removed successfully from the transmit queue as each callback is processed ... so speed is eventually limited to the speed at which the SMSOut callbacks for that connection are processed.

There is a way to allocate more threads for the SMSOut callback processing. To allocate additional threads, edit SMSGW.INI, and under the [SMSGW] header add AsyncCallbackThreads=##, where ## is the number of additional threads to allocate per async SMPP connection.

But if you do this, you need to make sure that this multithreading will not slow down your callback processing even further. If most of the time in the callback is in a database lock, you may end up with callbacks even taking longer as more are processed simultaneously.

I am also concerned that you indicate throttling errors in your test with the accounting callbacks ... does this mean you were receiving throttling errors from your upstream provider?

I'd suggest as a next step that you run some tests with simple callbacks that do no processing other than returning an OK response. This will give an idea of how much the processing time of your callback logic is an issue.

I'd be happy to look at some SMSDEBUG.LOG files to better understand the timing issues in the callbacks. The SMSDEBUG.LOG does slow things down considerably but with reasonably fast disk, it'll at least give us an idea of what type of response time NowSMS is seeing with the callbacks...and provide a better idea of where the timing issue is.

One other question ... what is your ping time to the SMPP server?

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 23
Registered: 12-2012
Posted on Friday, September 20, 2013 - 02:21 pm:   

Hi Des,

Thank you for your response.

As you suggested, I did a few tests with callbacks pointing to a blank page which instantly returns OK response. However, I did not see any difference in the Throughput as it was still restricted to 11-12 Msgs/sec. So, this test suggests that we don't have any issues with our own callback logic. I have also emailed you the SMSDEBUG.LOG as requested where the URL's were pointing to a blank page.

We were also able to achieve the required 100 Msg/sec throughput without any callbacks by optimising our binds.

Since you have mentioned that the Callbacks by default are multi-threaded, I would like to know the default thread value for each SMPP connection as I did a few tests with values of 3,5 and 10 which did not make any difference.

Regarding throttling errors, it occured when we were submitting the messages to the SMPP client users using 5 Transceivers binds. However, when the binds were reduced to 3 Transmitters they did not appear again. So, I think we should be fine.

Also, please find the round trip ping times below to our SMPP Server:

min-0ms
max-1ms
Avg-0ms

Could you please investigate the sent debug logs in the email?

Hoping to hear from you soon.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4663
Registered: 08-2008
Posted on Friday, September 20, 2013 - 03:56 pm:   

Hi Haree,

We're analysing this. There are some odd timings and differences compared to when we create similar tests ... and we are trying to understand what would cause these differences.

One question I do have ... what version of Windows is running on this system?

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 24
Registered: 12-2012
Posted on Friday, September 20, 2013 - 04:09 pm:   

Hi Des,

Our Servers are running on:

MS Windows Server 2003
Version 5.2 R2
Standard Edition
Service Pack 2


Let me know if you need more information.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4664
Registered: 08-2008
Posted on Friday, September 20, 2013 - 06:20 pm:   

Hi Haree,

I've been trying to recreate your accounting callback performance problem (without success) ... and it just hit me ... there are no SMSOut callbacks in the debug log that you sent me. In fact, there do not seem to be any messages being sent out at all ... probably because SMSAccountingMustSetRoute=Yes is set in the configuration ... so with the empty callback response all messages are failing.

So I'm not sure what throughput is actually being measured?

Is it inbound SMPP? If it is inbound SMPP, then if the submitting client is not using async mode itself, accounting callbacks will definitely slow down how fast messages are accepted by the NowSMS SMPP server. NowSMS does not acknowledge that it has received the message until the callbacks are complete ... so if the client is synchronous and waits for that response before sending the next message, you will definitely see a slow down.

If that is the scenario, there may be ways to tweak the performance ... but I need to make sure I fully understand the scenario so that I know how and where performance is being measured.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 25
Registered: 12-2012
Posted on Monday, September 23, 2013 - 09:28 am:   

Hi Des,

Apologies if I was not clear. It is actually inbound SMPP. As mentioned in the first post,we would like to achieve the speed of receiving messages for a SMPP client user(SMS Users). So, we have set the message receiving speed to 100 Msgs/Sec for the SMPP client user in the first server.

In the second server, I can confirm that we have set up an asynchronous bind of window size set to 50 and the SMSC Send Limit set to 100 Messages/Sec to send messages to the SMPP client in the first server.

When the tests were carried out with the above set up, we got a message submission speed of 11 messages/sec in our second server with our own call back logic in the first server. As requested, we also did a few tests with the callbacks pointing to a blank page which did not make any difference to the messaging speed. However, we were able to achieve a message submission speed of 100 msgs/sec without any callbacks.

So, we would like you to help us achieve a message submission speed of 100 Messages/sec to our SMPP client user in the first server.

Hope the above explanation makes sense.

Let me know if you need more information.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4667
Registered: 08-2008
Posted on Monday, September 23, 2013 - 10:16 pm:   

Hi Haree,

It's still not clear to me what entity is submitting the messages. It is an SMPP client of some sort, but does it understand and implement async SMPP?

My feeling is that the bottleneck is on that side of the connection, not the outbound connection. In other words, 11 msg/sec cout because that is all that is coming in.

In my tests I am running:

NowSMS1 -> NowSMS2 -> simulator

All are on the same network and I am testing with an accounting callback on NowSMS2 that is a simple text file on an IIS server ... so that I can factor out any callback delays.

Connections between the servers are configured as TRX 3 sessions, async window size 50.

I have no problem pushing hundreds of messages per second end to end.

(I submit to a distribution list with 100,000 entries via the web on NowSMS1 to fill its queue.)

I even removed the async mode on NowSMS1 and still could pump several hundred messages per second through the system with accounting callbacks on NowSMS2. (Async mode is more necessary when the systems have more of a transport delay between them.)

So I'm trying to figure out why it doesn't work on your system.

The only thing that stands out to me is that your debug log is showing an odd delay between the PreAuth and SMSSend callbacks. That is something that does not occur in any of my tests.

I notice that your IIS server is not using keep-alive sockets (as evidenced by Connection: close header in response). So I disabled keep-alive on my system, but still no drastic performance drop ... immediately ... after disabling keep-alives I started experiencing slow down because of connection failures. Messages would stop transmitting for periods of time until the system caught up. So I believe keep-alive connections are important ... but they should be enabled by default in IIS. (If you don't see Connection: close in the HTTP response, keep-alive is enabled and working.)

I don't think keep-alives are the issue on your system, but they are a potential concern if you can get past the current issue.

I've since dropped TRX to 1 and no drastic performance limitation.

You mention an SMSC Send Limit ... where exactly have you applied limits? And what if you remove the limits?

In other words, are the limit settings causing the performance problem ... are the callbacks ok without any limit settings?

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4669
Registered: 08-2008
Posted on Tuesday, September 24, 2013 - 11:47 am:   

Hi Haree,

We've been looking real closely at forced throttling settings, both the SMSCSendLimit and the receiving limit for an account defined under "SMS Users".

In particular, the only explanation for such a pronounced delay between the PreAuth and SMSSend callbacks would be a receiving speed limit being enforced.

Running tests on different systems, we found that limits were enforced differently on different systems, where throttle timing was less consistent on some systems.

Apparently the Windows timers that we were using for throttling were not guaranteed to be accurate enough for higher speed throttling. On some systems they were accurate enough, others not, depending on other environmental factors.

We modified the throttling logic to use more accurate timers, and this resolved the issues where we were able to see throttling speeds that were lower than configured. An update with more accurate throttling timers is now posted at http://www.nowsms.com/download/nowsms20130923.zip.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 26
Registered: 12-2012
Posted on Thursday, September 26, 2013 - 10:14 am:   

Hi Des,

Apologies for the delayed response.

To make things clear, we have NowSMS installed on 2 different servers, NowSMS1 & NowSMS2. In the first server, NowSMS1 is used to submit messages to SMS user on NowSMS2.

The NowSMS1 which submits the messages has SMSC limit set to 100Msg/sec in the .INI file whereas the NowSMS2 on another server has the client SMPP user (SMS users) which has been set to receive 100Msg/sec("Limit speed of receiving messages for this account").

Hope the above explanation gives you an idea of the setup.

Regarding SMSC limit, as mentioned earlier, I have applied it to NowSMS1 on the first server which submits the messages. I haven't tested taking off the limits. However, I could see the message submission speed dropping down when Nowsms accounting callbacks are included in Nowsms2.

Could you please confirm if the new release will resolve the issue based on my set up?

Looking forward to your response.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4672
Registered: 08-2008
Posted on Thursday, September 26, 2013 - 05:08 pm:   

Hi Haree,

Yes, the updated release should resolve the issue. At least with the quick callbacks in your most recent tests.

The timers that were being used for speed throttling applied via configuration settings were not accurate enough. So when NowSMS wanted to force a throttling delay of less than 15ms, the timer might wait up to 15ms, slowing things down more than intended.

The timers have been changed to improve accuracy. So if you can get the intended performance level without throttling limits, applying limits should not reduce performance below the configured limit.

The above is a confusing statement, so let me give an example ...

If you can achieve 100 mps without limits, applying a throttling limit of 100 mps or higher should not reduce performance below 100 mps. However, the inaccuracy of the timers before this update could reduce performance well below the configured throttle limit.

I guess what I'm saying is that you want to test your callbacks without speed throttling limits applied and make sure they can keep up.

Then test again with the throttling limits.

Before the update, the throttling limits may limit performance more than intended.

When you tested with the static callbacks, throttling limit problems caused the performance issues.

When you go back to your regular callbacks, try them first without performance limits. If running a single connection, 21ms response will be a bottleneck limiting traffic on the connection to under 50mps.


--
Des
NowSMS Support