| NowSMS polling the modem slows it down |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through September 26, 2009 ⬆ |
◄ ► |
| Author | Message | ||||||||
| ashot shahbazian New member Username: Animatele Post Number: 12 Registered: 06-2004 |
Bryce, Des, We've noticed two issues each severely limiting the speed of receipt of messages via modems. The system is as follows: - a commercial server, dual 2nd generation 1-core Xeon CPUs, 8GB RAM running W3k - NowSMS v.2009.05.21 - 30 modems Siemens MC35i in a telco-grade channel bank, excellent reception - comports connected via a Digi serial concentrator, in the same LAN. Tried 38400 and 115200 port speeds, no noticeable difference, left at 115200 - host network supports CSD only, no SMS via GPRS for sending - the system is used for receiving only. When a batch of 30 messages is sent to one of the modems in sequence (via an extremely fast SS7 gateway) they appear in the NowSMS SMSIN log as being delivered in 108 seconds or 0.28 SMS/sec., see the 1st log:
The modem polling mode was set to "direct Phase 2+". If set to "Default" it takes even longer, by about 1.5 minutes. If the same batch is sent to the same modem with the modem disconnected from NowSMS and connected to a terminal program, 30 messages seem to deliver and store in the modem's memory within 24 seconds or at 1.25 SMS/sec., or 4.5 times faster! Please see the second log:
We have an idea why such drastic slowdown. The solution could be a relatively simple one, but some experimenting with your involvement would be needed. If you recreate the scenario we can send messages to any UK number, except O2, very fast. Our thoughts are based on that when the messages are received and stored in a modem disconnected from MowSMS and then the terminal is disconnected and the service started - the messages stored in memory poll about as fast as they get stored in a disconnected modem - in 28 seconds! See the 3d log:
So most of the slowdown could be the modem poorly handling receiving and polling at the same time. If that is the case, perhaps the polling logic can be redesigned. For example: If the "Receive" is enabled, the modem is not polled once a second, but once in X (say 10) seconds. But when it is polled, it is done not once per second but 10/sec., thus only bothering it with polling for 1/10th of the time and leaving the 9/10th for receiving and storing the messages. To make sure that just engaging the modem via MAPI isn't slowing it down, I've disabled the "receive" but kept it connected to NowSMS. Now the applicaqtion won't poll the modem for received messages, but should initialise it and keep connected. Sending another batch: 24 seconds!
Now enabling "receive" and making NowSMS poll the modem: Same result, 29 sec.
Must be due to NowSMS polling frequency hardcoded at 1/sec. Provided that the modems may indeed be polled at 10/sec., let's estimate if we can squeeze 10 seconds worth of received data in a 1-second polling interval. 10 seconds X 200 bytes X 1.25 messages X 8 bits = 20,000 bits. 115,200 bps port speed supported by most modems should be more than sufficient. Implementing this for a modem that's sending at the same time is a lot tricker, but initially the patch could be designed for modems not configured for sending. NowSMS is also to blame for some of the slowdown though, which is the second problem. The receiving in NowSMS slows down even more if many modems are connected and being polled at the same time, even if the messages are sent to one out of 30 modems while 29 are idle but being polled. If the same batch is sent to one modem that's connected to NowSMS while the rest 29 are removed, it takes 90 seconds as per the SMSIN log. With 30 modems being polled and just one receiving it's 108 sec. - slower by 20%. Now we're trying to engage as many modems in receiving in parallel as possible. We have two independent gateways connected directly to SS7 carriers' transit switches supporting our destination network. As we want every number to receive 25-30 texts as in our previous tests, we would use 8 receiving modems out of 30 to blast off 190-200 messages to this modem gateway simultaneously. The result as per the NowSMS SMSIN log was 1.06 SMS/sec on 8 lines, or 0.13 SMS/sec. per line in average. This is 2.5 TIMES slower than with NowSMS polling just one modem! With 30 lines receiving simultaneously, the result sould be so bad the system would hardly be useful. Ok, let's check if that's because our messages are congesting the control channels. Stopping NowSMS and repeating the test: Line1 37 sec. per 12 texts or 0.32 SMS/sec Line2 same Line3 37 sec. per 13 texts Other lines were similar. So it looks like our tests did create a congestion. But that wasn't the most limiting factor: When restarted NowSMS, the speed of receiving polled messages stored in 8 modems was 0.139 SMS/sec., not much different from when receiving+polling simultaneously, and about 1/8th of the speed per line when compared with a single modem. So it looks like there are two separate issues, each contributing to the slowdown. One might be related to the modem performance, while the second is some kind of a bug that makes for inefficient use of redundant devices. Note that when sending via modems, the overall throughput scales out almost linearly with the increase in the number of modems. The trouble is with receiving only. In our tests, the CPU load wouldn't peak to more than 4-5% - and that's only when starting-stopping the service. So the older server cannot be blamed, it seems. Kind regards, Ashot | ||||||||
| Bryce Norwood - NowSMS Support Board Administrator Username: Bryce Post Number: 7816 Registered: 10-2002 |
Hi Ashot, The first part can be dealt with relatively easy. We've been considering configurable polling delay intervals for quite some time, and I'm going to go ahead and see if we can get this implemented this week. I'm also going to take the suggestion that for the "Direct" settings, the polling interval should be increased by default. We'll set it to 10 seconds as a default in that case. Regardless, the delay will be configurable. I'm at a loss to explain the second part. Well, maybe there's some explanation. There are some critical sections during modem initialisation. Once they've been initialised, nothing happening on any one modem should effect any other modem. That *might* be enough to throw off your test. It might be interesting to see an SMSDEBUG.LOG to see if it shows us something else. But changing the polling delay might be enough to start with. -bn | ||||||||
| ashot shahbazian New member Username: Animatele Post Number: 13 Registered: 06-2004 |
Hi Bryce, On the 2nd part: We are starting with the tests once all modems are initialised. That is clearly visible by the network load on the interface we have the serial concentrator connected to: it increases for 1.5-2 minutes while initialising 30 modems then drops and stabilises. So this may not have been the cause. As for the 1st part, questions: 1. Which AT commands do you use when polling in "direct" mode as opposed to others? In this mode, where would the messages store during intervals between polling? 2. Do you set the modem explicitly not to "report" new messages when they're received and just wait to be polled? I think you should, as that alone might slow it down. 3. I recall seeing you changing the message storage location, polling, changing it and polling again. How does the application know what kind of modem is used? xx in at+cpms="xx" in text mode are different for different modems (e.g. Wavecom and Siemens.) Or are they standardised for the PDU mode? 4. In recent tests, while polling the Siemens (disconnected from NowSMS and in text mode) the screen output would often stall for a second or two between messages on the 1st attempt ("REC UNREAD") but when tried for the second time with "REC READ" or "ALL" 30 messages would always retrieve instantly. Have you seen such behaviour? Perhaps instead of polling for all messages, apart from making the polling interval variable it's also worth trying to: - set the storage type and get the total message count - then poll very quickly for each message individually? It seems that when you set the "index" to a specific memory location the messages are retriving instantly, unlike that when you let the modem figure which ones are new. 5. Interestingly, modern Simcards often provide for "quicker" storage than the old modems' internal memory. But different simcards have different memory capacity. It might worth trying the variable polling interval in both scenarios. In Siemenses though if you set the storage to "SM" and it gets full the messages seem to spill over to internal memory, so this can be tricky. 6. As there might be some issue with resource allocation by the application, is it worth trying some precise timing? In other words, the 1st modem is polled on the 10th second, the 2nd on the 11th, the 3d on the 12th etc? We can't re-test before the weekend as that system is already in production. However, if you have a system with several modems either in the U.S. or the U.K. I can batch-send for you to have a look. Email me the numbers and when you'd be ready, if that's an option. The most troubling of course was that the overall receiving speed (unlike sending) would seem to be limited and not scalable by adding more modems. At any rate, if you can implement the variable polling time this week I'll make sure to run more tests over the weekend first without the patch then with a patch applied and send you the results and the debug logs. Kind regards, Ashot | ||||||||
| Bryce Norwood - NowSMS Support Board Administrator Username: Bryce Post Number: 7818 Registered: 10-2002 |
Hi Ashot, You raise some good points and questions. I'm still mostly troubled by the 2nd part, because I can't see any way that polling one modem would effect others. Overall, I would say that our polling is very much brute force. Unfortunately, it caters for the "least common denominator", that being phone modems, which tend to have the most unreliable implementations. As a result, on modems that have both "ME" and "SM" storage, there is polling of both locations in most cases. We tried disabling that extra logic yesterday, and we had a phone modem that has otherwise been extremely reliable become very unstable. For optimum performance, it is almost like there should be more settings. But I think we can improve it ... mostly by leaving the "Default" alone, but modifying the logic behind the other options. And instead of sending the AT+CMGL command to poll, we can use AT+CPMS? which will report non-zero results if there are any messages. That might be a little better for performance. And if a memory location is specified for the receive settings in NowSMS, then we can turn off the notifications and just rely on polling. (Currently, the notifications may be on if they are on by default, or if they have been turned on by a previous command.) And I think we can be more intelligent and avoid the polling of both "SM" and "ME" for most devices. These changes may help some with performance. But I don't have any explanation of why polling one modem would slow down polling of other modems. We're going to load up a few modems with a bunch of stored messages and then retrieve them to see if we can find anything. But maybe the second part is just related to the first part ... too much polling. -bn | ||||||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 974 Registered: 08-2008 |
Ashot, We haven't really gotten anywhere on finding problems with simultaneous polling of multiple modems. But granted, we've only been testing with 2 or 3 modems, looking for any signs of performance degradation. We've also been testing primarily with the update that Bryce is referring to above. The polling frequency has been scaled back to default to 10 seconds when "direct" mode is configured, or 2 seconds for other modes. And the actual commands involved in the polling activity have been reduced so that they will hopefully not put as much processing strain on the modem. That may be enough to make a significant difference. The update has been uploaded to replace the previous version at http://www.nowsms.com/download/nowsms2009rc.zip. -- Des NowSMS Support | ||||||||
| ashot shahbazian New member Username: Animatele Post Number: 33 Registered: 06-2004 |
Hi Bryce, Des Need to revisit this unfortunately. With the latest change in modem polling logic, which I assume got copied in subsequent releases, it got troublesome: When using a relatively large number of modems to send and receive at the same time, messages from some modems would stop being fetched by the application. If the service is stopped/started they would start coming at a rate of about 1 per second. Secondly, in cases where the modem was disconnected from NowSMS but powered on, received messages would sometimes completely fill its SIM card (although you seem to use "ME" as default storage with "phase 2+" setting) and when that modem is again reconnected to NowSMS it would ignore the messages in SIM which would make that modem respond to network with an error (sending SMSC gets a "memory full" error.) Despite that NowSMS would change the location to "ME". So stopping to poll the modem's SIM was not a good idea. Is there a setting to change the behaviour back to "brute force," as is was before? That seemed to slow down receiving but at least it was reliable. Or perhaps there is some flexibility, such as setting the polling interval, enabling/disabling SIM polling and perhaps setting the SIM polling interval separately from that for "direct to modem", using/not using the new message alert etc? We had to revert to the previous release, which took care of the receiving. However, in the previous releases the general logic of sending is by no means "brute force" and is unsuitable for SAT sending whereas each message has a pre-defined Source Address and can be sent from one modem only. In the older releases, it was not possible to do away with those \q\ subfolders, and once the message .req file gets there the application is using some opaque retry logic which is unsuitable for our sending scenario (message needs to be sent the very instance the modem's finished with the previous sending or receiving operation). As both inbound and outbound streams are quite dense this results many received messages (especially fragments of concat messages) stuck in the outbound queue for a few minutes - even though by looking at the log we can see that the modem was clearly idle from either sending or receiving for 10-20-20 seconds at a time during that 2-3 minutes the outbound message for that line was hung. It'd be great if you could enable necessary flexibility in the latest software release with an aim of absolute prioritisation of sending of messages with pre-defined source addresses matching individual modem numbers. At the same time, the receiving should be possible in the old more stable format with a reasonable ability of tweaking parameters. In other words, the prioritisation of sending should be in the form of the ratio of sent messages to a polling operation. If the modem is not sending, the application is polling it for messages at (configurable) intervals, say every 2 seconds for "direct" and every 10 seconds the SIM card. If there are messages in the outbound queue for that modem it lets the configurable maximum of 5 (might be more if there were segmented ones as that must take precedence) queued messages (FIFO, not delaying the segmented ones and (received) segments of one long message all go through in sequence uninterrupted by the polling operation) and pauses the polling while the batch is sending. Once it's through it performs polling and proceeds to sending once it's become possible, etc. As you may recall, we don't use the separateuserqueue setting, use relatively powerful commercial servers and rarely have many messages queued up due to "streaming" nature of our traffic. Thanks in advance for your help! We’ve not forgotten the “RerouteReceived and DLR” issue, and are in fact in the process of specifying that for a simple and graceful dynamic routing logic solution, will post on that later. Also the issue reported earlier about messages hung or bounced in the outbound queue around midnight – I think we know what’s causing that (and other weird SMPP problems,) will post on that too. Kind regards, Ashot | ||||||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1127 Registered: 08-2008 |
Hi Ashot, I have to admit that I'm a little confused by a few things. So let me start with the points that I'm less confused about. I wish that the "SMS Message Storage" setting was independent of the "Direct to Modem" setting. We were trying to simplify the configuration options, but the two options are not directly related. Historically, NowSMS has always preferred using "ME" if it is present. So the "Default" and "Direct to Modem" options always try to request "ME" storage, if it is available. We had a long internal discussion of this issue ... and we've noted your issues along with some issues that others have had upgrading to this new release. Our conclusion was that while long term it would be a good idea to separate these settings, a good temporary change for us would be to prefer "SM" storage for the "Default" and "Direct to Modem" settings. This may help you out with some of your issues. With regard to the new polling logic, when it comes time to poll, NowSMS 2009 is using the AT+CPMS? command. If any of the memory locations returned by this command return a non-zero value, then we will check for messages in that memory location. For any of the "Direct" settings, NowSMS will try to init the modem to receive to "ME" ... so NowSMS will not see any messages in "SM". It is strange that the modem would have problems receiving if "SM" is full when it is supposed to be receiving to "ME". Anyway, we have made the change to prefer "SM" memory for "Default" and "Direct to" settings, instead of "ME". So the init will direct the modem to use "SM". And the AT+CPMS? results will only reference "SM". More notes ... The polling interval is configurable on a modem by modem basis. It defaults to 10 seconds when "Direct to" is selected, or 2 seconds for other settings. To change the default add ReceivePoll=##, where ## is the number of seconds, under the appropriate [Modem - driver name] section header of SMSGW.INI. There is also an OldPollingLogic=Yes setting that can be put in the [Modem - driver name] section which specifies that the old polling logic should be used. (The old logic is still subject to the polling interval.) One other thought ... I thought that you were planning to disable the direct receipt of messages and rely completely on polling. We made a change where when a specific memory location was specified in the "SMS Message Storage", we would send an init command to the modem to make sure that it was not delivering messages to the application as they were received ... so that they could only be received by polling. Did you try that? We thought that you were thinking that would be faster. The version with the change to prefer "SM" for "Default" and "Direct to" settings has been uploaded to http://www.nowsms.com/download/nowsmsupdate.zip. -- Des NowSMS Support | ||||||||
| ashot shahbazian New member Username: Animatele Post Number: 34 Registered: 06-2004 |
Hi Des, Let me comment: Historically, NowSMS has always preferred using "ME" if it is present. So the "Default" and "Direct to Modem" options always try to request "ME" storage, if it is available. In our case, we used a 2009.07.09 release. A Wavecom modem, which was configured for Direct Phase 2+, had for some reason disconnected from NowSMS stayed like that for some time, then connected to NowSMS again. The monitoring picked it up by that the modem was sedning but not receiving anything. I've stopped the service, connected a terminal program and queried it with AT+CPMS? Location was set to ME, but ME was empty while SM had a maximum of 30 messages. I tried texting to it and got a Memory Full error from the MNOs switch, even though the storage was set to an empty ME. Once I've deleted the messages in the SM manually and reconnected NowSMS the modem worked. So, two conclusions here: 1. If the SIM card is full the switch would report (or some switches would) a Mem Full condition - and new messages would keep retrying until the timeout for that error is over. This is clearly the case on several modems and different networks, as we've noticed the same problem in the past. 2. Why has NowSMS not polled the SM, if it goes by what's returned by the CPMS command, I don't know. But the condition persisted for at least a day and that modem was actively sending while its SIM was full. In view of the above, to prefer "SM" storage for the "Default" and "Direct to Modem" settings. Woold be wrong. A SIM card has lower capacity, and if it gets full for one or another reason (they do) the receiving gets impossible. It is also often the case that when a SIM gets full the modem becomes less responsive or hardly responsive. For example, we've observed cases in the past of messages in a modem's full SIM not being polled even though we'd stop/start NowSMS. If we cleared just one of 30 messages in SM though, the next stop-start would poll all of them quickly. So a full SIM should be considered a critical condition for more than one reason, and perhaps adding an automatic logic to delete the last message (and polling the SIM immediately after) when that happens would not be a bad idea. I can only guess why that modem's SM storage got full. Perhaps it was in the middle of SM polling when a lightning struck and then, by the time the application hooked it back again the CMGL command could no longer execute because of the filled SIM. What you should do is I think to keep the polling logic and storage location separate and independent. There are 2 settings for Direct, one Default one Device Memory and one SIM. It is confusing even for a person like me dealing with modems for 7-8 years. What was, before the latest patch, the difference between 2 Directs, the Default and the Device Memory. I couldn't tell before I connected a sniffer and even that was not very conclusive. On the other hand, I know for sure that once you've introduced the "direct" option, things got a lot more smooth than they used to. Apart from a slowdown on many modems, we've not noticed serious issues for a long time. Also, the "direct" setting, as I recall from our discussions with Bryce a long while ago made it possible to retrieve the delivery receipts, and in some of the later ones to even convert them to SMPP DLR for SMPP users (which we tested and it didn't work well.) Now you might want to check if any of the recently introduced changes affected the DLR functionality, as it could be a useful one for some customers. I thought that you were planning to disable the direct receipt of messages and rely completely on polling. To understand this better, I need to know what exacly happens in both scenarios, preferrably if you could describe the AT commands to init then perform the receipt and then what happens next (you could email it to ashotanimatele.com) We made a change where when a specific memory location was specified in the "SMS Message Storage", we would send an init command to the modem to make sure that it was not delivering messages to the application as they were received ... so that they could only be received by polling. Did you try that? We thought that you were thinking that would be faster. This we have missed. What we did was we kept the modems with Direct Phase 2+ setting. From what I understand now, in this mode the modem is still polled but once in 10 secs instead of 2 in the previous releases. Was this the only change for the Direct 2+? That had a clearly negative impact. Many modems would randomly stop sending mmessages to the application and only stopping/starting the service would make the modem dump them, at about 1/sec. This was observed separately from the issue with the modem having a full SIM. Okay, let's go one-by-one. I'll try measuring the receiving speed now setting the storage to "direct to modem" with default settings with the 2009.07.09 release (which makes the modems randomly stop delivering the messages to the app,) then I'll change the storage type to SIM, try it with default 2s. 5s and 10s polling intervals, then would repeat the same with Equipment Memory. Will post the results later today. Kind regards, Ashot | ||||||||
| narvesh kumar kirodiwal New member Username: Kumarnarvesh Post Number: 1 Registered: 08-2009 |
Ptet councelling llnd list not avilable,}} | ||||||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1136 Registered: 08-2008 |
Hi Ashot, I'm going to defer any real detailed response until after Bryce is back from holiday. But here is some explanation of what has changed. Polling interval ... in previous versions it was about once a second if the modem was idle. While polling, previous versions would use AT+CMGL to poll, switching between "SM" and "ME" (i.e., AT+CPMS="ME") before issuing the AT+CMGL command, so that both locations were checked. The OldPollingLogic=Yes setting under the modem header restores that logic. The newer version uses AT+CPMS? and then polls any memory locations referenced in the response that have a non-zero result. AT+CPMS="xx" commands are NOT sent if the location being polled matches MEM1 from AT+CPMS?. As a result, there are a lot fewer commands being sent in the newer version, especially because we're not sending AT+CPMS="xx" so frequently when polling. The other difference is that if the modem is set to "SIM" or "Device Memory", we send AT+CNMI=2,0 to disable unsolicited notifications, so that the modem is only polled. (We do still enable unsolicited notifications for delivery receipts.) Previously, if the modem was set to "SIM" or "device memory", we did not send any AT+CNMI command to enable or disable unsolicited notifications for new messages ... it was just left at whatever the modem defaulted to. I'm going to run some tests, and go over this all in more detail with Bryce when he returns from holiday. -- Des NowSMS Support | ||||||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1173 Registered: 08-2008 |
Hi Ashot, I finally have some follow-up. As you have noted, the changes to the polling logic did not work real well in the real world. Here are a couple of things that we have observed: If you set the modem to receive to "SM" or "ME", it is not safe to assume that all received messages will go to that location. It is still necessary to poll both locations to find all waiting messages. When using "Direct to Modem" (or the "Phase 2+" setting), one of the AT commands that NowSMS sends will inadvertently disable this on some modems, forcing messages to only be received by polling. We've made some changes to address these issues. We've also adjusted the polling interval to default to every 3 seconds for the "Direct" settings and 2 seconds for other settings. (ReceivePoll= can still be used to override.) We've kept some of the optimisations to the polling logic. In particular, AT+CMGL will only be used if AT+CPMS tells us that there are waiting messages. This results in less commands being sent to the modem each time we poll. Going back to your original message ... what I don't understand is the slow speed at which NowSMS retrieves the polled messages from your modem. It's basically only able to process 1 per second. On my system, using an Option ICON 322, I'm able to pull in 5 messages per second when polling already received messages. What slows it down is that after we decode each message, we send a command to delete the message from the memory storage on the modem. My guess is that this command is taking longer on your modems. I haven't compared different modems to see how fast they process the AT+CMGD command, but that might be interesting. It is also very possible that this delete operation is what is the resource intensive operation for the modem ... also perhaps the modem locks memory storage while deleting so that new messages cannot be received. Does it make a difference between "ME" and "SM" settings? -- Des NowSMS Support | ||||||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1174 Registered: 08-2008 |
Forgot to post the update link. It is the same as before ... http://www.nowsms.com/download/nowsmsupdate.zip. | ||||||||
NowSMS log
