SMS accounting - performance bug?

SMS accounting - performance bug? SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 65
Registered: 07-2006
Posted on Thursday, December 24, 2009 - 02:29 pm:   

Hi!

Seems that callbacks seriously decreasing NowSMS’s performance.
We have 2 test environments, WIN2003 servers with equal SMSGW.INI and bound to same SMSC.

1st server has SMS accounting feature enabled and pointed to another server 2XCPU Xeon Quad Core, callback handler run on IIS 6 (40 threads). Servers connected over 1gb LAN and IIS avg response time 170-200ms. We’ve own routing system, analyzing destination address and set up “SMSCRoute” param in Pre-Auth request, we can submit messages at high speed 200SMS/sec. But NowSMS can’t push these messages at full speed and queue is rising (avg outbound speed can vary 30-90 sms/sec. Seems our problem somewhere in SMSOut request, but that is our fastest request (we just pust http url to memory cache). We’ve tried different “AsyncCallbackThreads” parameter (10-100) but it hadn’t any serious impact on performance.

When we runs same tests on 2 nd server (without sms accouting) outbound speed is excellent 200sms/sec and stable.

Do you have any ideas how to solve that or where to dig?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 66
Registered: 07-2006
Posted on Sunday, December 27, 2009 - 08:36 pm:   

In addition,

We tested another (3 rd) NowSMS environment, similar to 1 st but our callback handler didn’t issue any response parameter like “RouteCharge” and “SMSCRoute”, just HTTP 200. In that scenario each SMS cost 1 credit and traffic was routed by NowSMS means (Route1=XXXX & AllowedUser1=XXXX). Outbound speed was very high and similar to 2 nd scenario.

Now situation looks a little bit different to me, in 1st scenario all routing and load-balancing adjustments were made in "PreAuth" callback. Just a simple example: 3 routes(RouteA, RouteB, RouteC), users sending 10 SMS.
SMS1 – SMSRoute=RouteA
SMS2 – SMSRoute=RouteB
SMS3 – SMSRoute=RouteC
SMS4 – SMSRoute=RouteA
SMS5 – SMSRoute=RouteB
SMS6 – SMSRoute=RouteC
SMS7 – SMSRoute=RouteA
SMS8 – SMSRoute=RouteB
SMS9 – SMSRoute=RouteC
SMS10 – SMSRoute=RouteA
Therefore NowSMS in each user’s queue always contains messages with different “SMSCRoute=” parameter, what potentially isn’t happening when we use simple “AllowedUser1=” parameter.
We suspect that NowSMS isn’t not speed-optimized with “SMSCRoute” parameter or doesn’t handle well messages with random “SMSCRoute” parameter in the same user’s queue folder.

Please let us know, what you think about that issue. Your help will be appreciated!

Wish you Merry Christmas and Happy New 2010 Year!
Luck and prosperity for all members of NowSMS team!
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1627
Registered: 08-2008
Posted on Monday, January 04, 2010 - 09:31 pm:   

Hi Alex,

I ran some quick tests before you posted your second message, and I suspected that there was more to it ... some combination of factors.

I'm trying to think of what we can look at to narrow this down.

Is the CPU load significantly higher in the problem scenario?

If the SMSOUT callback does no processing at all ... just returns back HTTP 200 OK immediately, is the delay still there?

--
Des
NowSMS Support
Adding the SMSCRoute parameter does have a performance impact, but it typically performs better than using "AllowedUser1=" types of settings.

I think what we need to do here is figure out if the bottleneck is in the "SMSOut" callback processing, or somewhere else.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 67
Registered: 07-2006
Posted on Tuesday, January 05, 2010 - 01:20 pm:   

Hi Des,

Thanks for your response!

With enabled callbacks (with issuing SMSCRoute paramater) CPU load higher (85-95%) than without it (30-50%) on the same traffic. In our scenario SMSOut callback is 10 times faster than PreAuth callback because there is no CPU calcutation operations, just putting data in memory cache but queue at NowSMS is rising(after PreAuth). Also we believe that all callbacks except PreAuth run in async mode, is it true?

AllowedUser1 isn't suitable in our scenario, because we want to use automated user management system (adding, removing, blocking accounts).

To honestly, i don't know where to dig further. What kind of information i can provide for better situation understanding?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1632
Registered: 08-2008
Posted on Tuesday, January 05, 2010 - 10:27 pm:   

A follow-up Alex ... I'm working on trying to create a scenario to duplicate what you're seeing.

I do agree that it doesn't seem that the SMSOut accounting callbacks would be the cause of the performance hit. But maybe we could verify this just to be certain. If you could add AccountingExtended=No under the [SMSGW] header of SMSGW.INI, then this will disable SMSOut and SMSIn callbacks.

And if the queue seems to be rising quickly, then this does not seem to point to the PreAuth callback as the performance hit.

Bryce and I were discussing this earlier, and while he still wants me to come up with a recreatable scenario, he thinks a simple tweak could be made to greatly speed up processing whenever explicit routing is used. We're not 100% convinced that it will fix what you're seeing, but it's our best guess.

Even though it's supposedly "simple", it's still probably 48 to 72 hours away.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 68
Registered: 07-2006
Posted on Friday, January 08, 2010 - 12:32 pm:   

In a meantime.

We noticed that problem also connected with SMSIn callback, it’s seriously slowing down receiving messages (inbound SMS and DRs). If we have large inbound queue at external SMSC, speed of receiving varies 50-100 SMS/sec and never goes higher. All connections tested in TM + RX mode.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1642
Registered: 08-2008
Posted on Friday, January 08, 2010 - 11:32 pm:   

Hi Alex,

I'm sorry that it has taken longer than expected for the update that I promised.

We've almost completed testing it, and it does greatly reduce the CPU overhead when an accounting callback explicitly sets the route for a message.

I'm still concerned with your CPU load, as I don't get %'s that high in a dual quad core Xeon environment. But with this update I am seeing a drop from around 50% to around 20% during my test ... so that's very significant, and I expect it will make a difference in your environment.

Regarding receiving speed, SMSIn accounting callbacks are definitely in a critical path. How long do your callbacks take in this case?

We experimented with a configuration setting to use additional threads for inbound message processing and related accounting callbacks.

We didn't implement it because performance was slightly slower in this mode when there were no accounting callbacks. But we didn't really look at its impact with accounting callbacks ... but realistically, 100 messages per second requires that the callback average a response time of 10ms, which may be difficult to achieve if you are doing any real processing in the callback.

I've asked our guys to revisit that configuration setting so that I can test it with accounting callbacks.

In the meantime, I still need a little more time to test this other update.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 69
Registered: 07-2006
Posted on Sunday, January 10, 2010 - 02:19 pm:   

Hi Des!

That’s excellent news!

We get such CPU load because we add about 100-120 SMPP binds and submit messages by both methods (SMPP + HTTP). I’m pretty sure that our test environments different. Looking forward to test your update ASAP.

Current handling scheme that we don’t do any realtime callbacks processing (only PreAuth analyzing it in order to set correct RouteCharge and SMSCRoute parameters) all callbacks are being put in memory cache and after that special process flushes data. Also our software developers made significant performance updated with IIS, in logs we see that page response tine now between 10-25ms instead of those 150-170ms. If we count that sending of 1 sms generates 4 callbacks, 1 thread can handle 10 messages per second and if we're using 40 threads – 400 sms/sec.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1651
Registered: 08-2008
Posted on Tuesday, January 12, 2010 - 02:02 am:   

Hi Alex,

Ok ... 100 to 120 SMPP binds ... that would aggravate this issue. I was testing with 20 binds.

The update that I've been testing has a new configuration option to enable additional route specific queues, which help things out a lot when you are explicitly setting a route.

For now, this setting needs to be explicitly enabled with the setting UseRouteQueues=Yes under the [SMSGW] header in SMSGW.INI.

We've also done some closer study of the performance impact of accounting callbacks on receiving messages. And indeed, the way it is currently implemented, these callbacks are in the critical path ... and can have a significant performance impact on receiving messages. Awhile back, we partially implemented an async message receiver queue so that the receiving and callback processing would not be in the critical path for SMPP connection processing.

Based upon this performance impact, we decided to revisit this implementation.

In my tests, it makes a big difference. In the updated version, to enable this setting, it is necessary to edit SMSGW.INI and add AsyncReceiveMessages=Yes under the [SMSGW] header. The way it is implemented, this setting applies for all SMPP connections that have async windowing enabled.

The promised update is available for download at http://www.nowsms.com/download/nowsms20100111.zip.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 71
Registered: 07-2006
Posted on Friday, January 15, 2010 - 05:48 am:   

Hi Des,

Unfortunately we couldn’t test your updated. After adding "UseRouteQueues=Yes" parameter and after restart, NowSMS stops any outbound traffic. NowSMS is accepting new messages and put them into RT-<SMSCRoute>-<User login> folder but messages just stick and don’t go away. Could you please check that issue, seems your update has a bug.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1660
Registered: 08-2008
Posted on Tuesday, January 19, 2010 - 04:14 pm:   

Hi Alex,

Were you able to test the version that I updated at that link?

There was a problem using "raw" route names (like "SMPP - server:port"), and the setting was only working properly with route names that were set explicitly by the "RouteName=" setting in SMSGW.INI.

This has been fixed.

The link above has been updated with this fix, but to avoid confusion, I've also placed a new download at http://www.nowsms.com/download/nowsms20100119.zip.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 72
Registered: 07-2006
Posted on Wednesday, January 20, 2010 - 08:36 am:   

Hi Des!

Thanks for your response!

We tested previous patch and also found a bug, that messages got stuck in queue when explicit routing is set. New patch will be tested as soon as we can solve problem described below.

But all tests we decided to step back and revise our NowSMS configuration sets because new patches didn’t provide any serious results. And we have discovered that amount of outbound SMSCs connections have direct impact on the performance.
1-7 – 100% (full speed)
7-12 – 70%
12-22 – 50%
Over 22 – approx 20%

That bug not connected with explicit routing and appear even if you just “AllowedUser” parameter. Please confirm that your faced the same problem. Looking forward to hear from you soon!

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1672
Registered: 08-2008
Posted on Wednesday, January 20, 2010 - 06:29 pm:   

Hi Alex,

With a large number of routes, explicit routing is the way to go.

The update with the UseRouteQueues=Yes should have minimal performance degradation if you use explicit routing.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 73
Registered: 07-2006
Posted on Thursday, January 21, 2010 - 10:31 am:   

Hi Des,

Great update, messages with explicit routing sending at full speed! But we notice some other issue, some messages(whole folder) got stuck in the queue and never sent to SMSC. Could you please check that.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1685
Registered: 08-2008
Posted on Thursday, January 21, 2010 - 03:37 pm:   

Hi Alex,

Anything unusual about the folder name in question?

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 74
Registered: 07-2006
Posted on Thursday, January 21, 2010 - 06:19 pm:   

Hi Des,

examples
RT_-_SMPP - SMARTS-ASTRAHAN-20240_-_8907_SINICH
RT_-_SMPP - SMARTS-ASTRAHAN2-2775_-_kryukovm

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 75
Registered: 07-2006
Posted on Monday, January 25, 2010 - 02:56 pm:   

Hi Des,

Any updates on that issue?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1714
Registered: 08-2008
Posted on Monday, January 25, 2010 - 04:43 pm:   

Hi Alex,

We haven't had any luck recreating this.

However, upon further review, I think we may be on to something.

The current logic appears to work only if the "SMSCRoute" is set in the "pre-auth" callback. If one route is returned in the pre-auth, but a different one is returned in the regular "SmsSend" callback, then it gets confused.

Does that match what you are experiencing?

What I would expect to see in this scearnio is that if you looked at the stray files in one of those directories, you would find an "SMSCRoute=" setting that does not match the route name that is embedded in the directory name.

It's an oversight in how we implemented these route queues ... we didn't expect this scenario.

We'll have to see if there is some way to overcome this without major changes to the underlying logic.

But before we go too far down this path ... does this scenario sound like what you are encountering?

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 77
Registered: 07-2006
Posted on Monday, January 25, 2010 - 06:26 pm:   

Hi Des,

Thank you for the respone!

Yes, i've found an error, SMSCRoute parameter doesn't match directory name. Any ideas how such situation can occur? We don't provide any responses in SMSSend callback. Seems another bug in NowSMS?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1717
Registered: 08-2008
Posted on Monday, January 25, 2010 - 07:56 pm:   

Hi Alex,

We can't see any bug in this area. The only way that the route can change in between creating the file in the directory and writing the message file to disk is via the SMSSend (no PreAuth flag) accounting callback.

I'm not sure that we have any good solution for this, other than to ignore the route specified in the SmsSend callback when the route queues are enabled.

We're still thinking about it, so maybe there will be a better solution.

But we're fairly confident that if you investigate the callbacks, this is what is occurring.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 78
Registered: 07-2006
Posted on Tuesday, January 26, 2010 - 01:45 pm:   

Hi Des,

We made some more tests and seems that something wrong is still happening with NowSMS, for example we set in PreAuth “SMSCRoute=RouteA” then NowSMS puts message in folder “RT_RouteB_-_<UserLogin>” but SMSCRoute parameter in message “SMSCRoute=RouteA”, so I guess that’s why we got stuck messages, as soon as we move file to the root QDir path, message are being sent. Another thing, as I explained you before, SMSSend doesn’t provide any routing information, like it was before, all routing logic located in PreAuth, SMSSend just writes data to DB and all stuck messages have “SMSCRoute=RouteA” – there is no chance for SMSCRoute override. Any ideas? We can try your version which ignoring “SMSCRoute” parameter in SMSSend callback.

Regards
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1726
Registered: 08-2008
Posted on Tuesday, January 26, 2010 - 07:25 pm:   

Hi Alex,

I'm stumped, and a little concerned about the "SMSCRoute" inconsistencies.

We are implementing a solution where if the route was set by the pre-auth response, it is not allowed to be changed in the "SmsSend" accounting callback.

It is still being tested, but should be ready in a day or two.

I'm just concerned that if you're not returning it in the response, how are we seeing a different value? I hope that it is just something that you don't realise is happening, and not a larger problem.

--
Des
NowSMS Support
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7875
Registered: 10-2002
Posted on Wednesday, January 27, 2010 - 08:47 pm:   

Hi Alex,

A follow-up ... an updated build of NowSMS has been posted which ignores the "SMSCRoute" in the "SMSSend" callback in this scenario. (This scenario = route queues enabled + route set by pre-auth accounting response.)

Download link: http://www.nowsms.com/download/nowsms20100127.zip

-bn
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 79
Registered: 07-2006
Posted on Sunday, January 31, 2010 - 12:40 pm:   

Hi Des and Bryce,

Just made tests to an updated build of NowSMS. Unfortunately bug is still exist, messages with explicit routing get stuck in the queue. Seems that problem isn’t at our side, you ought to revise NowSMS’s routing (file generation) logic. We found more bugs in your latest releases but I’ll update you about them when performance issue will be finally resolved.

Regards,
Alex K.
uday
New member
Username: Udayitsme

Post Number: 1
Registered: 02-2010
Posted on Monday, February 01, 2010 - 11:02 am:   

sssss
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1738
Registered: 08-2008
Posted on Monday, February 01, 2010 - 09:44 pm:   

Hi Alex,

I'm going to ask Bryce to follow-up with you tomorrow.

--
Des
NowSMS Support
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7880
Registered: 10-2002
Posted on Tuesday, February 02, 2010 - 02:24 pm:   

Hi Alex,

We're investigating this more, and I believe we've spotted a problem. However, I'm not sure the problem that we've found applies to your configuration ... so I have to ask.

Are any of these messages submitted via HTTP?

We've identified a problem, but thus far it seems to be specific to messages submitted via HTTP.

Talking with Des, he believes that your message traffic is SMPP only ... is that correct?

-bn
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 80
Registered: 07-2006
Posted on Tuesday, February 02, 2010 - 03:48 pm:   

Hi Bryce and Des,

We're using both methods. Yes, seems accounts submitting via HTTP only affected by this bug. Hope that you'll be able to find solution soon.

Regards,
Alex K.
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7881
Registered: 10-2002
Posted on Tuesday, February 02, 2010 - 10:19 pm:   

Hi Alex,

Ok. We're still not clear on how to recreate the problem, but in a code review we noticed some inconsistencies in how the variables that hold the routing information are handled.

The inconsistencies are related to how we added support for "SMSCRoute=" in the accounting pre-auth.

While we still couldn't recreate any problems, we decided to rework this area.

So unfortunately, I still can't guarantee it is fixed.

If this doesn't fix it, then I'm going to have to write a script to randomly return different routing information, and see if I can recreate it under load.

But before I do that ... we'd like you to try this update with the UseRouteQueues=Yes setting, as there's a good chance it will fix the problem.

http://www.nowsms.com/download/nowsms20100202.zip

-bn
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 81
Registered: 07-2006
Posted on Wednesday, February 03, 2010 - 06:48 pm:   

Hi Bryce and Des,

Unfortunately, message still sticking in the queue. Waiting for further updates!

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1745
Registered: 08-2008
Posted on Thursday, February 04, 2010 - 02:00 am:   

Ok Alex,

We finally setup a random routing script, and the problem is clear, you just need a single HTTP submission to multiple recipients on different routes. All messages are queued in the directory for the route of the first recipient.

This update fixes that.

http://www.nowsms.com/download/nowsms20100203.zip
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 82
Registered: 07-2006
Posted on Thursday, February 04, 2010 - 08:43 am:   

Archive is corrupted :-)
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7882
Registered: 10-2002
Posted on Thursday, February 04, 2010 - 05:36 pm:   

Something tells me that this "UseRouteQueues" feature was never meant to be implemented.

The file has been reuploaded.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 83
Registered: 07-2006
Posted on Thursday, February 04, 2010 - 05:52 pm:   

Hi Bryce,

Why do you think so? Performance is still low with
explicit routing?

Regards,
Alex K.
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7883
Registered: 10-2002
Posted on Thursday, February 04, 2010 - 06:13 pm:   

Hi Alex,

I'm just saying so because this has dragged on far longer than it should have ... :-(

-bn
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1748
Registered: 08-2008
Posted on Thursday, February 04, 2010 - 08:54 pm:   

Hi Alex,

Bryce may be discouraged that an update that seemed so simple has consumed so much resources. But I do agree that the performance enhancement that this can offer is extremely significant.

Cheers,

Des
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 84
Registered: 07-2006
Posted on Friday, February 05, 2010 - 05:39 pm:   

Hello Bryce and Des,

Yes, i agree, 2 months of my life wasted away :-) But when i started my investigation, i couldn't even guess that problem was covered in NowSMS's (legacy) routing system. We don't use "AllowedUserOnly" for many months because there is no flexibility and scale at all :-( Please excuse me, if I mixed you up with some previous posts, it won’t happen again :-)