Speed tests

Speed tests SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 124
Registered: 07-2006
Posted on Wednesday, September 01, 2010 - 05:53 pm:   

We have spent some tests for speed of sending of messages through NowSMS http request and through SMPP.
Sent about 200 000 test messages (without SMSC).

Http request (Account callback it is switched on): speed of 150 messages in a second.
Http request (Account callback it is switched off): speed of 500 messages in a second.
SMPP sending (Account callback it is switched on): speed of 570 messages in a second.

Question: why such difference at sending through HTTP request and SMPP with switched on Account callbacks?
All options Account callback for smpp and http requests are identical.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 125
Registered: 07-2006
Posted on Thursday, September 02, 2010 - 04:09 pm:   

I apologize, I meant reсeive of messages instead of sending.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2479
Registered: 08-2008
Posted on Thursday, September 02, 2010 - 04:20 pm:   

Do you mean the speed at which NowSMS accepts messages from clients?
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 126
Registered: 07-2006
Posted on Thursday, September 02, 2010 - 04:22 pm:   

yeap. via web and smpp interfaces
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 127
Registered: 07-2006
Posted on Monday, September 13, 2010 - 04:07 pm:   

Hi Des!

Do you have any updates for that issue? Or you need any additional information?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2497
Registered: 08-2008
Posted on Monday, September 13, 2010 - 10:13 pm:   

Hi Alex,

I suppose we do need more information about the HTTP submitter.

It is not surprising that HTTP is less efficient. Keep in mind that HTTP must reauthenticate with ever submission, whereas SMPP authenticates once, and then establishes a session.

If the HTTP submission is not multi-threaded, it may be slowed down by the fact that the NowSMS web interface does not support HTTP keep-alive sockets.

Is the HTTP submission single threaded?

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 128
Registered: 07-2006
Posted on Tuesday, September 14, 2010 - 12:29 pm:   

Hi Des!

HTTP submission is multi-threaded. We have tried various configurations of HTTP submission:

1 thread, speed 100 per second;
4 threads, speed 150 per second;
8 threads, speed 150 per second;
20 threads, speed 150 per second.

I think that there is no problem with HTTP reauthentication. Is there any limit for number of web threads?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 129
Registered: 07-2006
Posted on Wednesday, September 15, 2010 - 06:06 pm:   

Hi Des,

We made some more test in order to confirm our suspicions about HTTP interface.
Single submission(1 recip per request)
dynamic callbaks: 130 SMS/sec
callbacks disabled: 250 SMS/sec

Bulk submission (250 recip per request)
dynamic callbaks: 230 SMS/sec
static callbacks: 800 SMS/sec

It's impossible to use bulk submission with our custom parameters which is being forwared to PreAuth callback. And performance looks very poor in compare to SMPP. Will it be a field of any improvements?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2503
Registered: 08-2008
Posted on Wednesday, September 15, 2010 - 07:03 pm:   

Hi Alex,

I agree that bulk HTTP submission (submission to a large number of recipients) is not practical when you are using the SMSAccountingPreAuthPerRecip=Yes setting.

This was one of the major concerns that we had about implementing this setting (for you), and in many ways I regret that we did this, because a lot of your support issues seem to come back to this.

That said, single recipient submissions should not be that much slower using HTTP compared to SMPP. It will be slower because authentication, user account lookup, needs to be performed for each HTTP submission whereas this is performed only once per session with SMPP (and not with each message submission). It just should not be that much slower.

We are investigating this further.

Does the CPU max out during your tests, or do things seem to be disk i/o bound?

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 130
Registered: 07-2006
Posted on Wednesday, September 15, 2010 - 07:27 pm:   

Hi Des,

We running tests on 4 CPU and fast SSD disks, performance monitor looks like usual. But if disable authentication for web interface will it helps?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2504
Registered: 08-2008
Posted on Wednesday, September 15, 2010 - 08:30 pm:   

Hi Alex,

If you disable authentication for the web interface, it disables the accounting callbacks.

A good solution for performance <g>, but not practical.

So the CPU did not appear maxed out during the tests?

Can you also tell me the exact version? (We did optimise the user authentication earlier this year, but I need to double check when that was.)

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 131
Registered: 07-2006
Posted on Wednesday, September 15, 2010 - 08:40 pm:   

hehe, 2 sides coin :-)

2010.08.23 version. CPU seems pretty fine, load near 40%. Submission speed is very stable, doesn't look like a bottleneck. Do you think, that issue could be solved soon?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2507
Registered: 08-2008
Posted on Friday, September 17, 2010 - 10:40 pm:   

Hi Alex,

The accounting callback is the "wildcard".

We can't test with your accounting callback, so we generally test with static callbacks.

As long as the submission is multi-threaded/multi-connection (HTTP or SMPP), it should be able to overcome most of the performance drag of an accounting callback.

In my testing on one of our test systems, with a static callback, 8 HTTP threads submitting, I saw about a 10% reduction in message submission speed with accounting callbacks enabled.

We studied our HTTP server implementation and the re-authentication issue, and were able to almost double the submit performance.

On my test system, we could average around 400 mps submission speed before the change. After the change, it was around 775 mps.

This is with a static accounting callback defined (simply requests a text file from a local server).

What I can't explain is why a dynamic accounting callback would have such a severe impact on your performance (but doesn't have as much impact when used with SMPP submissions).

I think you need to do some more tests to compare things with static callback vs. dynamic callback, and different numbers of submitting HTTP threads. Or maybe your dynamic callback is blocking on a database update, and that is causing the delay?

When we can upload this performance update, it should offer some performance improvement, but I am still concerned that throughput drops so dramatically in your tests. I'd encourage you to do some comparisons with static callbacks to try to narrow this down further.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 133
Registered: 07-2006
Posted on Monday, September 20, 2010 - 07:15 pm:   

Hi Des,

Do you submit single message per request or multiple? I'm curious about your results but with multiple recipients we getting similar speed.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2516
Registered: 08-2008
Posted on Tuesday, September 21, 2010 - 06:49 pm:   

Hi Alex,

The tests we've been running are focused on single recipient submissions. (The SMSAccountingPreAuthPerRecip=Yes setting that you need to be able to provide custom route charging to SMS User accounts complicates multiple recipient submissions because accounting callbacks are processed serially for each recipient.)

I wish I could release the code, but it is based upon the code in NowSMS that performs the HTTP request for an accounting callbacks.

It took me awhile to test this, because I had to ask our engineers to give me a tool to test HTTP performance.

The test program simply loops repeatedly sending an HTTP request to submit a message.

In multithread mode, it spawns multiple threads that perform this looping logic.

Maybe I can convince them to clean it up a little more and release a version that lets you customise the URL and specify the number of messages and number of threads in an INI file.

For the moment, we're still working to verify the update doesn't introduce any new problems. So far, so good.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 134
Registered: 07-2006
Posted on Wednesday, September 22, 2010 - 02:29 pm:   

Hi Des,
I see, seems that we have similar test conditions. Let's wait for that great update. Or maybe it's better to create COM-API instead of HTTP callbacks and replace funny folders with real SQL engine for storing, sending and tracking messages. I believe it would be 10x speed increase.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2523
Registered: 08-2008
Posted on Wednesday, September 22, 2010 - 09:25 pm:   

Update posted at http://www.nowsms.com/download/nowsms20100921.zip.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 135
Registered: 07-2006
Posted on Wednesday, September 22, 2010 - 09:26 pm:   

Hi Des,

Seems that we found a clue to the situation, CPU isn't normal, it's near a peak 90-97%. I don't know why we hadn't found that before. Vs. SMPP submission - 20-30% CPU load. Now CPU is a bottleneck with HTTP request, similar to shared user balances, finally we removed it. Please try to issue updated release ASAP or we have to do stupid job, re-making our www message-pusher from HTTP to SMPP API. It could be a waste of efforts.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 136
Registered: 07-2006
Posted on Wednesday, September 22, 2010 - 09:28 pm:   

Great support! Just pushed the button!

We're on the same wave :-))))))))))))))))
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 137
Registered: 07-2006
Posted on Thursday, September 23, 2010 - 07:56 pm:   

Hi Des,

Just fresh test.

Dynamic callbacks: 183 SMS/sec. Not so big improvement. But CPU dropped to 15% :-) Maybe there is any important parameters to add into config? Tomorrow we will check it with static callbacks and re-make HTTP message pusher, but it isn't practical, getting dynamic route and cost our key for success.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 138
Registered: 07-2006
Posted on Thursday, September 23, 2010 - 08:42 pm:   

Could you please make tests with 30 random responses and move HTTP handler to remote host(in LAN). I guess, you'll see the difference...

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2529
Registered: 08-2008
Posted on Friday, September 24, 2010 - 05:40 pm:   

Hi Alex,

Indeed there is something to this. I changed the callback on my test server to call a PHP script instead, and I see similar results.

We don't have any explanation for it at this time. As you've noted, CPU utilisation is reasonable, so NowSMS is not CPU bound.

We are going to implement keep-alive sockets to see if this makes a difference.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 139
Registered: 07-2006
Posted on Friday, September 24, 2010 - 09:32 pm:   

Hi Des,

Seems new release has a bug, it's raised CPU to 100% without any reasons. We rolled back. Please revise the code. Crazy friday...

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 141
Registered: 07-2006
Posted on Wednesday, September 29, 2010 - 05:34 pm:   

Hi Des,

Any updates regarding that issue? Btw, http acception speed with shared user balances around 70mps but it was tested with 1gb channel.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2536
Registered: 08-2008
Posted on Thursday, September 30, 2010 - 01:39 am:   

Hi Alex,

We are continuing to investigate and experiment.

Several days ago, HTTP keep-alive support was implemented for the HTTP request handler. This replaces the experimental logic from the last update. (We may revisit the other logic in the future, as we believe we found one race condition flaw in it. But keep-alive support ultimately provides even better performance.)

But keep-alive support didn't noticeably improve performance for dynamic callbacks.

So we've been running a variety of tests and experiments, including benchmarking the static and dynamic callbacks that we were using directly, so that we understand their performance limitations.

What we found was that the multithread test submitting to NowSMS never performed any faster than a single thread test against the callback directly, which suggested that something in NowSMS had a critical section that was serializing the processing of the callbacks.

We ran a lot of internal timing tests, and never found such a problem, directly.

However, the analysis did show us some internal synchronisation issues that we could improve.

It's preliminary, but I'm now seeing 400mps with a dynamic callback, compared to around 150mps before.

At present, this does require that the client implement HTTP keep-alive sockets. (If we re-enable the other code, which we may leave in an INI setting to do so, then similar performance could be achieved without keep-alive sockets.)

More time is needed for testing and verification.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2537
Registered: 08-2008
Posted on Friday, October 01, 2010 - 09:00 pm:   

Update with the keep-alive socket support and improved performance for dynamic callbacks: http://www.nowsms.com/download/nowsms20100930.zip

If your client doesn't handle keep-alive sockets, the logic added in the previous release has been cleaned up and can be enabled with the setting EnableThreadPool=Yes under [SMSGW] in SMSGW.INI.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 142
Registered: 07-2006
Posted on Monday, October 04, 2010 - 07:17 pm:   

Hi Des,

Found the bug in new release. You have broken administrative API. We can send messages over HTTP but actions like http://192.168.0.55:8800/provision?Type=SMS&AdminAction=CreditCheck&Name=user77 fall by timeout. Please advise.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 143
Registered: 07-2006
Posted on Monday, October 04, 2010 - 07:56 pm:   

One more bug, messages don't appear in nowsms's logs.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2543
Registered: 08-2008
Posted on Monday, October 04, 2010 - 09:31 pm:   

Hi Alex,


quote:

One more bug, messages don't appear in nowsms's logs.




Are you sure this is not because you have used configuration parameters to turn these logs off? I see no indication of this problem.


quote:

You have broken administrative API




There is indeed a problem with "/provision" requests unless EnableKeepAlive=No is set in the [SMSGW] section of SMSGW.INI.

Of course, setting that parameter negates most of the performance updates.

We will fix that.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 144
Registered: 07-2006
Posted on Monday, October 04, 2010 - 10:04 pm:   

Log’s always enabled for sure but we'll double-check. And path's SMSGW.INI defined. Performance’s still looks the same, nothing revolutionary.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 145
Registered: 07-2006
Posted on Tuesday, October 05, 2010 - 07:59 am:   

Hi Des,

Logs are fine, we didn’t' see them because we tested just inbound messages. But there is no perfromance improvments, speed is always around 200mps. We tried all those parameters: EnableKeepAlive=No, EnableThreadPool=Yes and our messages pusher has keep-alive sockets enabled. Please advise.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2550
Registered: 08-2008
Posted on Tuesday, October 05, 2010 - 06:09 pm:   

EnableKeepAlive=No will have some performance impact (however, it was necessary to work-around the problem with the "/provision" interface).

A version that fixes this problem (so EnableKeepAlive=No can be removed from your configuration) is at http://www.nowsms.com/download/nowsms20101004.zip

Have you tried running HTTP load tests against your accounting callback using parameters similar to what NowSMS uses to confirm that this is not the limiting factor?

Is it possible to use different user accounts on the submissions? (In my tests, I have not done this, and all submissions are from a single user account. However, I am suspicious that deadlocking over the updating of the user statistic counters and quota is a timing factor.)

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 146
Registered: 07-2006
Posted on Tuesday, October 05, 2010 - 08:19 pm:   

Hi Des,

we tried new release but got similar speed results. Could you please provide your http testing tool? Maybe we have problem with our http pusher. We also submit messages over one account.

Regards,
Alex K
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2554
Registered: 08-2008
Posted on Thursday, October 07, 2010 - 06:44 pm:   

Hi Alex,

The engineer who built the tool for me is out of the office this week. There are some small changes necessary before I can send it out. He thinks he can get to it early next week.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 147
Registered: 07-2006
Posted on Tuesday, October 12, 2010 - 01:39 pm:   

Hi Des,

Any updates?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2571
Registered: 08-2008
Posted on Thursday, October 14, 2010 - 09:26 pm:   

Simple HTTP tester available at http://www.nowsms.com/download/httptest.zip.

Create HTTPTest.INI in the same directory with the following format to control the tester:

[HttpTest]
Threads=##
Requests=########
URL=http://xxxxxxxxxxxxx/xxxxxxx/xxxxx



The tester will create the number of "Threads" specified, each repeatedly issuing HTTP requests for "URL" until the number of "Requests" have been completed.

Note that the tester stops counting when the number of completed requests is complete. Several additional requests may be completed before the tester is shutdown.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 152
Registered: 07-2006
Posted on Monday, October 25, 2010 - 09:05 pm:   

Hi Des,

New test results:
0921 vs. 1004 (408 vs. 160 mps)
I'm little bit confused, we have same configuration file and test environment, we simply re-installed 1004 over 0921 and restarted the server. I don't know what can decrease performance in such scale.


here is the smsgw.ini:
[SMSGW]
MessageIDPrefix=GW01
WebAuth=Yes
WebMenu=Yes
WebPort=8800
SMPPPort=2775
SMPPPortSSL=3550
ReceiveSMS=Yes
ReceiveMMS=No
ReceiveSMSCharset=utf-8
DisableUserLog=Yes
DisableWebLog=Yes
QDir=D:\NowSMS.Queue
LogDirectory=D:\NowSMS.Logs
UsersDir=D:\Users
SMSInDir=D:\NowSMS.IN
MessageIDTrackingDir=D:\NowSMS.DR
SeparateUserQueues=Yes
UserQueueThreshold=10
SMPPRejectErrorCodes=8
SMPPThrottleErrorDelay=30
UseRouteQueues=Yes
TrackSMPPReceipts=Yes
ReceiptRequested=Yes
MaxSMPPShortCodeLen=3
RetryMaxAttempts=10
SMSAccountingPreAuthPerRecip=Yes
SMSAccountingURL=http://192.168.0.75/QueryManager/ProcessSMS.aspx

Please advise what decrease performance and many thanks in advance.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 153
Registered: 07-2006
Posted on Friday, October 29, 2010 - 09:05 pm:   

Hi Des,

Any new updates?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 154
Registered: 07-2006
Posted on Monday, November 01, 2010 - 04:46 pm:   

Hi Des,

Any updates?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2611
Registered: 08-2008
Posted on Tuesday, November 02, 2010 - 08:30 pm:   

Hi Alex,

I'm sorry, but my product manager is concerned that you do not have a license for a product that operates at that speed.

I am continuing to investigate (my question to you would be if you go back to 0921 does the speed go back up), but I think you may also want to follow up with Keith.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 155
Registered: 07-2006
Posted on Tuesday, November 02, 2010 - 10:32 pm:   

Hi Des,

We already discussed that issue with Keith several months ago. We had those VDS with trial licenses and found a lot of bugs in NowSMS with them. But we’re not using them anymore. But I don’t how to prove Keith that we don’t use them anymore? Can we ask for another test key and show you our environment? We also make similar tests with Kannel.

If we're talking about inbound speed, 0921 definitely has 2 times higher inbound speed.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2625
Registered: 08-2008
Posted on Thursday, November 04, 2010 - 10:17 pm:   

Hi Alex,

FYI. The 0921 to 1004 change that brought back down performance has been undone in the update at http://www.nowsms.com/download/nowsms20101104.zip.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 156
Registered: 07-2006
Posted on Sunday, November 07, 2010 - 03:52 pm:   

Hi Des,

Something is wrong with a path, getting an error and can't download from provided link.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2632
Registered: 08-2008
Posted on Sunday, November 07, 2010 - 05:43 pm:   

Looks like a problem with the web site update. It should be fixed now.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 157
Registered: 07-2006
Posted on Monday, November 08, 2010 - 06:34 pm:   

Hi Des,

Seems that update's performance still broken.
0921 350 sms/sec
1104 140-169 sms/sec
On the same test server.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 158
Registered: 07-2006
Posted on Monday, November 08, 2010 - 08:38 pm:   

let me double-check everything