Slow Modem Delivery

Slow Modem Delivery SearchSearch
Author Message
Blair
New member
Username: Bcotnam

Post Number: 1
Registered: 07-2011
Posted on Wednesday, August 24, 2011 - 04:44 pm:   

Hello Support,

We are currently using the trial version of nowsms with the 10 message per second license.

We have 48 modems connected to the SMSC. We are experiencing very slow message delivery thorugh the nowsms gateway/modems.

Are there any limitation on the trial license on nowsms? We are sending 2 messages per minute and we are using the Multi-Tech GSM modems.

The speed is extremely slow and we are looking at improving the speeds, any help would be appreciated.

Regards,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3429
Registered: 08-2008
Posted on Wednesday, August 24, 2011 - 08:30 pm:   

Hi Blair,

Do you experience the same timing issue if you only have one modem active (or just a couple)?

I'm wondering if it is an i/o issue related to the large number of modems, or a license issue.

I'm suspecting a license issue, but I'd like to rule out the other first.

I'm also assuming that you asked for and received a 10 message per second trial license from us. Slowing down the speed to 1 or 2 messages per minute is usually a sign of an invalid license.

Could you please e-mail this license file to me at nowsms@nowsms.com, and put Attention: Des in the subject line. Also, please indicate what version of NowSMS you are running, and I would like to see at least a few lines from the current SMSOUT-yyyymmdd.LOG file so that I can understand the outbound message timing.

Also, make sure you are using the license with a current version of NowSMS. (I believe you are an existing customer and maybe you are trying to add this temporary license to an existing installation.) These large trial licenses are frequently not compatible with older versions of NowSMS.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 2
Registered: 07-2011
Posted on Wednesday, August 24, 2011 - 08:53 pm:   

Hello Des,

Thanks for your reply.

First off, we are using V2011.07.05 of nowsms. We requested a trial license for the 10 message/second key as we are looking at high volume.

When we firt connected a few modems to nowsms we notice the speed was fast. We have been using 48 modems for a while now and notice the consistency of speed wasn't that great especially if we send bulk messages.

We will email you our license file we received from nowsms sales department (Luke).

I have attached our current SMSOUT log file for your reference. Its not the entire log, but partial. I marked all ip addresses with x.x.x.x

We are using the current version of nowsms. We are a new customer and wanted to try with a faster license before we purchase to see if it will suit our needs.

Thanks,
Blair

application/octet-streamSMSOUT.LOG
SMSOUT-20110824.LOG (9.3 k)
Blair
New member
Username: Bcotnam

Post Number: 3
Registered: 07-2011
Posted on Wednesday, August 24, 2011 - 09:02 pm:   

Hello Des,

I emailed nowsms with the licesne key and i put attenstion to you in the subject. Let me know if you need anything further.

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3431
Registered: 08-2008
Posted on Wednesday, August 24, 2011 - 09:12 pm:   

Hi Blair,

Ok, it doesn't look like a license problem. When you said 2 messages per minute, that was my first suspicion.

In the log, I'm seeing 10 messages one second, 7 the next, 5 the next.

I probably need a longer log ... but also some more detailed explanation about the problem you are seeing from a performance perspective.

In that short log, I do see a lot of delivery receipts, and those do have a tendency to really slow down some modems ... but when that happens there are usually a lot of retry attempts shown in the logs.

So can you explain more about the performance observations?

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 4
Registered: 07-2011
Posted on Thursday, August 25, 2011 - 01:08 pm:   

Hello Des,

I have attached our complete SMS log from yesterday for you to review.

We do receive a lot of delivery receipts and we are also experiencing a lot of retry attempts in the outbound message queue. We have also added the below settings in SMSGW.ini file to try and help the retry attempts as by default was set to 20 attempts which is high, but we also don't want a high failure rate either.

RetryDelay=1800
RetryDelayMultiplier=0
RetryMaxAttempts=8

What we notice is if submitting the messages are gradual then there doesn't seem to be any probelms for the most part, but if the messges comes in bulk, then most of thos bulk messages get put into the outgoing message queue and we notice extremely slow message delivery and a lot of retry attempts. In the log i supplied there are a few large bulk message submissions.

So, in general, we really notice slow message delivery where there are bulk submissions.

Let us know if there is anything we can do.

Thanks,
Blair

application/octet-streamSMS.LOG 20110824
SMSOUT-20110824.LOG (9208.1 k)
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3432
Registered: 08-2008
Posted on Thursday, August 25, 2011 - 10:37 pm:   

Hi Blair,

Unfortunately, retries are a fact of life for modems that handle simultaneous send and receive (the delivery notifications count as receiving from the network).

We haven't done an extensive analysis of different modems, but what we have observed is that if a modem is attempting to send a message when a message is received, there will usually be a delay before the modem reports that the message sending failed ... requiring NowSMS to schedule a retry.

(There are some settings that may make a difference. Try setting "SMS Access" to "GPRS" ... if the operator network supports this, we have observed that receiving messages during a send attempt is usually less of a problem. Also try "SMS Message Storage" = "Direct to Modem - Phase 2+", which can make a difference if the modem properly handles this mode.)

For now, I'd suggest ignoring the above ... retries can be a factor of life with modems and delivery receipts.

RetryDelay=1800 would be an excessive setting. It means to wait 1800 seconds (30 minutes) to retry sending a particular message that encounters an error. (This delay kicks in after RetryDelayAfterAttempts failures.)

I'm guessing that while the log was running, you had this set for 600 (10 minutes), as I see several messages (many with common destination recipients) that are failing every 10 minutes.

I'm wondering if the problem is just that there are so many messages in the queue because there are so many destined to invalid numbers, or to numbers that are somehow blocked.

Unfortunately, it's hard to get a good reading of how many messages are in a retry state (and may eventually fail because they are invalid) and how many are in the queue but have yet to see a delivery attempt.

That said, I do see that while some seconds are good for sending, other times it's only one to five per second.

I'm not an expert in configuring large banks of modems, but I know in the past that customers who have done this have concentrated on things like directional antennas, out of concern that the signaling traffic could be too much if all being routed through a single cell tower.

I'm going to delete the log here because it may contain some personal information. Can you e-mail me SMSWEB-20110824.LOG. This should record submission times, and I just want to get a feel for transmission delays for messages that are not stuck in a retry state. (Send to nowsms@nowsms.com with Attention: Des in the subject line.)

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 5
Registered: 07-2011
Posted on Friday, August 26, 2011 - 04:43 pm:   

Hi Des,

I have sent an email with the SMSWEB-20110824.LOG

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3433
Registered: 08-2008
Posted on Friday, August 26, 2011 - 07:02 pm:   

Thanks Blair,

I'm looking over the log ... and I guess I probably should have asked you to enable the SMSDEBUG.LOG and send that as well.

The SMSDEBUG.LOG would tell me how long we are having to wait for the modem to send a message.

If we hit a time period where many of the modems all try to send at the same time, it looks like any individual modem takes 10 to 30+ seconds to send the message it is working on.

The activity at 07:58:12 is a pretty good example of the start of a rather large batch of messages. The SMSWEB log records when NowSMS receives a message to be sent out.

The SMSOUT log around that time seems to suggest that the system is relatively idle.

It is not until 07:58:48 that the first 20 of these messages are recorded as sent. An SMSDEBUG.LOG would tell me more, but generally speaking, if the system is idle, it shouldn't take more than 5 seconds before NowSMS starts sending a message out via an available connection. So I am a little suspicious that it takes 36 seconds in this case.

At 07:59:05, there is the start of CMS ERROR 512 reports from several modems. It is important to note that these modems are not modems that reported a successful sending at 07:58:48 ... it is likely that they began processing these send attempts just after 07:58:12.

The next successful send attempt from a modem that successfully sent at 07:58:48 isn't until 07:59:23.

While the messages are being sent, the traffic is very bursty, and during delay periods, at least one modem reports a CMS ERROR 512 error sending.

This suggests to me that all of these modems are simply overwhelming the signaling channel of the cell tower to which they are connected.

You'd probably get better throughput with fewer modems.

If "SMS Access" = GPRS is supported on the network, I'd also highly recommend trying that, as I believe it has less signaling overhead.

To use more modems, you'd need directional antennas ... or to geographically separate them (which is why some people use IP modems). That gets complicated to setup ... as a starting point, I'd suggest cutting back to 24 modems and comparing throughput. It may also be worth trying 18 and 12.

I don't believe it to be a NowSMS limitation, but more of a signaling issue with all of those modems connecting to the same tower.

If you want me to look at an SMSDEBUG.LOG, let me know. Just give me an idea of the time period to be analysed, as digging through a debug log on a busy system can take a lot of effort.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 6
Registered: 07-2011
Posted on Tuesday, August 30, 2011 - 01:05 pm:   

Thanks for the update. We have tried your suggestions above with no improvements. The cell phone towers doesn't seem to be the problem as we were getting good speeds with another appliation with the modems. Looks isolated to the nowsms application.

We have done some testing with the amount of modems and this is what we found.

First Test:
20 modems
Sent 100 msg
Result: 3 min total

Second Test:
5 modems
100 msg
Results: 2 min

Third Test:
5 modems
200 msg
Results: 6 min

Fourth Test:
5 modems
400 msg
13min

The fourth test we did again with 10 modems compared to 5 and received the exact same time/results. Looks like we are aveaging 3min per 100msg.

Looks like the nowsms application can't handle bulk modems. Is there anything that can be done to help connect more modems to nowsms?

Seems like as soon as we add more than 5 modems, we either get the same results or much worse results. The more modems we add the worst the performance.

Thanks,
Blair
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7993
Registered: 10-2002
Posted on Wednesday, August 31, 2011 - 12:25 pm:   

Hi Blair,

We will do some more investigation. What puzzles me, however is how the application can be at fault when there is such a large number of errors being returned by the modem.

Let me ask this...in your other application test, is that application also processing delivery reports? I believe delivery reports are a very big part of the performance problem.

I believe you sent a debug log to my colleague Keith. Please send him an SMSDEBUG.LOG if you have not already.

The only idea I have at this point is that there is no synchronization of activity across the modems. What I mean by this, is that if a large batch of messages is received. Each modem will grab a message and try to send out at the exact same time. Performance might improve if a delay could be inserted, for example 100ms. I'm not sure this would make a difference, but I am going to ask engineering to implement such a setting.

-bn
Blair
New member
Username: Bcotnam

Post Number: 7
Registered: 07-2011
Posted on Wednesday, August 31, 2011 - 01:11 pm:   

Hello Bryce,

Thanks for your reply. I have sent the SMSDEBUG.LOG to Keith for investigation.

In our other application we were processing and receiving delivery reports.

Anything you can do to help speed and reliability would be much appreciated.

Regards,
Blair
Blair
New member
Username: Bcotnam

Post Number: 8
Registered: 07-2011
Posted on Thursday, September 01, 2011 - 02:29 am:   

We have done a few more tests and here is what we found.

1. We notice same performance when receiving DR or not.
2.Some modems are randomly locking up when sending bulk messages. They still send messages some of the time, but at a lower rate.

We send a bulk test of 400 messages to our test phones and noticed sometimes we don't even receive the messages from all the modems. The load balance across the modems is up and down.

From the tests we have done it seems that nowsms is not meant to have many modems. Does Nowsms support multiple modems? We get the same results with 3 modems compared to 5 and so on.

Test 1:
400msg sent to 3 phones
DR not requested
5 modems
14min total from time of sent first message to last message received

Test 2:
400msg sent to 3 phones
DR not requested
3 modems
14min total

Test 3:
400msg sent to 3 phones
DR requested
3 modems
14min total

Thanks,
Blair
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3453
Registered: 08-2008
Posted on Thursday, September 08, 2011 - 12:33 pm:   

Hi Blair,

Apologies for the delay in response ... I know you've also been in communication with Keith.

We've made two adjustments, which I believe may make a difference in your results.

The first is what Bryce mentioned earlier in this thread. We've added a configurable throttling delay to delay processing of messages to prevent the situation where multiple modems are trying to send at the exact same instance.

To configure this, with the updated version, edit SMSGW.INI, and under the [SMSGW] header, add MessageThrottleDelay=100

The value is in milliseconds, and I would suggest 100 as a good starting point.

Second, we noticed several instances of unusual timing when studying your SMSDEBUG.LOG. It is difficult to explain ... but essentially part of our modem initialisation was telling the modem to keep the SMS channel open after sending to allow faster sending of the next message. We suspect that in a configuration with many modems on the same tower, this could lead to errors and delays.

Between these two changes, I am optimistic that we could see some improvement.

The update is available for download at:

http://www.nowsms.com/download/nowsms20110907.zip


--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 9
Registered: 07-2011
Posted on Thursday, September 08, 2011 - 06:26 pm:   

Thanks for the update.

We will try this new version with the features you mention.

Is it better to do a fresh install or can i install over our current version? If we install over current version will it keep our current configuration or will it be overwritten?

Thanks..
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3457
Registered: 08-2008
Posted on Thursday, September 08, 2011 - 06:43 pm:   

It is best to install over the current version as this retains all existing configuration information.
Blair
New member
Username: Bcotnam

Post Number: 10
Registered: 07-2011
Posted on Monday, September 12, 2011 - 06:02 pm:   

Hello,

we are getting a much higher fail rate in messages with the new version. I sent 100 messges and only received 43. We did a few more test with the same results.

This wasn't happening wtih the previous version we were using.

First thing we want to ask, is it possible to downgrade to the version we were previously using (V2011.07.05)?

Or do we have to do a fresh install. Some software doesn't allow to to downgrade, thats why i'm asking right now if we can or not.

thanks.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3469
Registered: 08-2008
Posted on Monday, September 12, 2011 - 09:58 pm:   

Hi Blair,

That is surprising. The only thing we did to the modem handler, other than add support for MessageThrottleDelay, is to remove AT+CMMS=2 from the initialisation sequence.

If supported, this tells the modem to keep the SMS relay protocol link active after sending a message, so that subsequent messages can be sent more quickly.

We had theorised that maybe doing this across a large number of modems was causing congestion.

I'm posting an update that adds AT+CMMS=2 back ... and before you downgrade, I'd be curious if the MessageThrottleDelay setting helps.

http://www.nowsms.com/download/nowsms20110912.zip


There is no problem downgrading an installation by running an older install. That is perfectly ok.

--
Des
NowSMS Support
Blair
New member
Username: Bcotnam

Post Number: 11
Registered: 07-2011
Posted on Tuesday, September 13, 2011 - 07:17 pm:   

Thanks you for the information. We noticed with the first build you gave us it was much faster receiving the message, but we wouldn't receive all message consistently, so i think it does help, but will need to conduct more testing.

We will try the new build and report back.

Thanks again.
Blair