| NowSMS Performance Issues |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through July 28, 2010 ⬆ |
◄ ► |
| Author | Message | |||
| Chris New member Username: Chrisc Post Number: 27 Registered: 12-2008 |
Hi guys We've had a slight issue today with a very amount of queued messages on NowSMS. It started at around 10:00 this morning and got resolved at around 14:00. We noticed quite a few things while this was happening, which could be related to things going on at our end, but might also be at NowSMS's end. The queued message count was hovering around the 25k mark for most of this period. During this time we noticed the following: Memory usage during this time was sitting at around 90MB. We took a chance and restarted the service, where memory usage was around 25MB. The CPW was at around 50% - 60% during the high load and after the restart it was 5% - 10% despite load conditions being the same. During the high load our message throughput was approximately 10 msg/second yet we're licensed for 50 msg/second. After the restart the throughput increased back to this level. We think that a part for the drop in performance could be due to quite a few .req and .err files still inside the Q folder, which is due to us deliberately not processing them properly. This is particularly the case for multi-segmented messages. We only accept the first receipt that comes in and disregard anything else afterwards related to that message. This has led us to wonder if the drop in performance and high memory usage could be due to an overly high count of invalid messages in memory. Is it possible for you guys to have a look at how the counts of the various types of messages are managed by NowSMS and how it would affect memory in terms of very high volumes? Moreover, if it is something we're doing wrong, could you point us in the right direction to avoid this from happenning again? Thanks Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1785 Registered: 08-2008 |
Hi Chris, I'm not overly concerned about the memory usage. But I do find it very odd that a restart would make this big of a difference in performance. Did you delete these .REQ and .ERR files that were deliberately not being processed when restarting? An overabundance of these files can cause a slow down, but this same slow down would still be present at restart (although it may come later, as NowSMS may process other messages first before it gets around to trying to process these). The .ERR files themselves don't cause any increase in memory usage ... they only slow things down indirectly. Windows doesn't like a large number of files in a folder, and a large number of .ERR files will end up slowing things down. If you have operational reasons for a large number of messages to be rejected like this, there is an ErrorQRetainDays=0 setting that can be added under the [SMSGW] header of SMSGW.INI. This will stop NowSMS from generating these files at all. (Note: The "0" value for this setting is only present in versions 2009.07.23 and later.) A large number of .REQ files that can't be processed is a bigger concern. I'd be curious to hear more about this to see if there's a better way this can be handled for your environment. What directory are they in? It would also help me to know: What version of NowSMS are you running? How many outbound SMSC connections? What is the primary way that messages are submitted into NowSMS (e.g., do they arrive via HTTP or SMPP)? Do you have separate outbound message queues enabled on the "SMS Users" page? (We've migrated toward enabling this setting by default, but older versions disabled it by default, and even if you've updated, it may still be disabled.) Tell me more, and then hopefully we can make some good suggestions. -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 30 Registered: 12-2008 |
Hi Des To answer the questions you've given above, during high load periods we do have quite a high number of .REQ files but not so much for .ERR files. The number .REQ files we find normally amounts to the same number of messages that are 'Queued'. We are looking at the ErrorQRetainDays setting and we'll probably implement that quite soon. All of our .REQ files are always contained within the 'Q' folder, sometimes though they are kept in seperate folders under the 'Q' folder. If the questions we raised above aren't related to sluggish performance, is there anything else that would cause NowSMS to spike from time to time in terms of CPU and memory usage? Thanks Chris | |||
| Chris New member Username: Chrisc Post Number: 32 Registered: 12-2008 |
Hi Des Another query related to the .ERR files, is it possible to send a notification similar to a receipt, along with a reason code back to 2-Way SMS URL? Thanks Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1863 Registered: 08-2008 |
Hi Chris, If you're not seeing a large number of .ERR files then they are likely not contributing to a problem. If there are a large number of ".REQ" files directly in the "Q" folder (not in multiple subfolders), then this can cause performance problems for the Windows operating system ... which of course then causes performance problems for NowSMS. The use of subfolders provides a dramatic performance improvement. We need to analyze why the subfolders are not being used. Do you have the "separate outbound message queues" setting enabled on the "SMS Users" page? (As I mentioned above, we've migrated toward enabling this setting by default, but older versions disabled it by default, and even if you've updated, it may still be disabled.) It would also help to know what version of NowSMS you are running. We've made a lot of optimisations to deal with larger queues, so the version can make a big difference. One other tip for extremely large queues, there can be a performance degradation having the NowSMS configuration program window open. (There is a configuration setting under [SMSGW] in SMSGW.INI, CacheQSize=Yes, which addresses this specific performance issue.) So tell me about the NowSMS version number. Apply the CacheQSize=Yes setting if you frequently leave the NowSMS configuration window up for statistical display. And also, look at the setting that pertains to subfolder usage.
Regarding the .ERR files, a non-delivery notification (receipt) is generated in this instance ... only if a delivery receipt was requested when the message was submitted. If you're submitting messages via SMPP, you also have the option of only being notified for failure delivery receipts by setting the registered_delivery flag to 2 in your SMPP client. NowSMS will also respect this setting for generating non-delivery receipts. Unfortunately, when submitting via HTTP, there is no option to request only non-delivery receipts. You either request delivery receipts, which includes non-delivery receipts ... or you don't get either. -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 34 Registered: 12-2008 |
Hi Des I don't think I was completely clear on the last post. The "separate outbound message queues" setting is enabled on our version of NowSMS and it set to 400. Does the number make any difference?? The ".REQ" files are put into seperate folders and Windows does seem to process the files faster in this respect. This however, has always been the same case with us since we try to keep up with the latest version of NowSMS whenever we find something needs amending. Our current version on Live and test is 2010.01.27. We've been doing quite a lot of testing these last couple of weeks, applying the different settings as this discussion goes on. Our overall messsage throughput still remains at around 24 msg/sec though. We've done some testing on the 2010.02.09 version as well, with the results being the same. There is one part worth mentioning though. We are using the SMPPSim software from Selenium for our testing. But based on the Queue size and the time the Simulator holds the messages in the queue this doesn't seem to be holding us back in terms of performance. Can you provide us with any more thoughts? Regards Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1882 Registered: 08-2008 |
Hi Chris, Thanks ... that helps me understand where you're at with your current setup. In which case, I may be going down the wrong path for diagnosis. I was working off of an assumption that the speed was at an acceptable rate until the message queues get large. Is that assumption correct, or is speed a problem regardless of queue size? Do you have async mode enabled for your SMPP connections (with a window size set to your target message rate for between 1 and 2 seconds)? -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 35 Registered: 12-2008 |
Hi Des No we do not have this setting enabled for our connections. Should we enable this? Regards Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1883 Registered: 08-2008 |
Hi Chris, Yes ... absolutely. Without this setting enabled, speed is seriously limited. Sorry I did not suspect or mention this previously. Most connections will top out somewhere between 3 and 10 mps without this configuration setting. Set your window size to somewhere around your target message rate for 1 second. -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 36 Registered: 12-2008 |
Hi Des Thanks for providing us with this useful info. We'll definitely start testing this right away. However, looking at the documentation and the pages describing async mode, I'm confused by what is meant by the Window Packet size. Could you explain a bit further? Regards Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1886 Registered: 08-2008 |
Hi Chris, Yes, I should explain a little bit more, as there is a reason why this setting is not enabled by default. The SMPP protocol can be implemented either with synchronous and asynchronous communication. Synchronous means that when the client issues a command, it waits for a response to that command before issuing the next command. In this mode, network latency can be a major limiting factor on performance. If you're on the same local network with your SMSC connection, you can achieve far greater throughput than with an SMSC connection that occurs over the internet. In asynchronous mode, the client does not stop to wait for a response before sending the next request. This allows more efficient use of available network bandwidth and removes network latency from being an issue. The "window size" is the maximum number of requests that can be outstanding before the client waits before transmitting more. In most cases, you don't want to set this to more than your target number of messages per second. All this said, I should caution you that many operator SMSCs can be very picky about messaging speeds. If you exceed your allotted speed, the SMSC may take unexpected actions to slow you down and reduce throughput. When you first enable async mode, you should keep a close eye on whether or not this increases your retries. There is an additional advanced configuration setting that can specify a sending speed limit per connection. It is described in the following link: http://blog.nowsms.com/2008/06/smsc-speed-limits.html -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 37 Registered: 12-2008 |
Hi Des Thanks for providing us with the async info and setting the speed limits, it definitely helped us! However, would it be possible to have the .ERR files move to a different folder if the overall size starts becoming problematic for Windows? This can only help in the overall performance of the Q folder if these files are moved into a child folder "Q" and then gets deleted after a certain period of time. Is there any possible way of doing this? Regards Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 1922 Registered: 08-2008 |
Hi Chris, How many .ERR files are you seeing? If you are not actually doing any processing of these files with another program, you can edit SMSGW.INI and under the [SMSGW] header, add ErrorQRetainDays=0 This setting will stop the files from even being generated. The problem messages will still be logged, the .ERR files just won't be left behind. (By default, they are left behind for 72 hours/3 days, before they are automatically deleted.) -- Des NowSMS Support | |||
| Chris New member Username: Chrisc Post Number: 38 Registered: 12-2008 |
Hi Des Thought I'd rather post it here than send an email. Many thanks for providing all the support in getting our performance up to the spec it should have been. We have been monitoring continued large message sends for the last couple of weeks now and we've been able to process messages at our maximum rate without any drop in performance while the Queue size has grown to over 150k in a single send and yet NowSMS continues to send at its peak rate. In fact, we still have to discover what our new roof margin is for the overall Queue size before we can see any drop in performance. Again, many thanks! Regards Chris | |||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2059 Registered: 08-2008 |
Hi Chris, Thanks for the feedback. We have been doing additional work in this area, especially for configurations where UseRouteQueues=Yes is not a complete solution. With these additional changes, we believe that we can achieve similar large message queue performance results in all environments. -- Des NowSMS Support | |||
