| Messages appear stuck in Message Queue |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through July 28, 2010 ⬆ |
◄ ► |
| Author | Message | ||||
| Chris New member Username: Chrisc Post Number: 40 Registered: 12-2008 |
Hi Des We've installed the new version of NowSMS on our testing environment only to find a rather strange occurance when attempting to send messages. We can connect to our binds OK and the setup seems to be the usual procedure. However, if we attempt to send any messages it simply stays in the message Queue and does not go anywhere. No error has been recorded nor has the message been retried. After some digging, I was able to send a message through a certain route (all our binds uses the RouteName parameter) and the message went through fine. If I try the same through another bind (which is part of 4 sessions) the message remains stuck in the Queue. I've seen in the release notes that you can now define multiple transmitter / receiver connections. Would this change have anything to do with it? Also, would it help if I send you our SMSGW.ini file?? Thanks Chris | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2123 Registered: 08-2008 |
Hi Chris, To start, let me ask a couple of quick questions. Are you using the UseRouteQueues=Yes setting? What directory is the message physically located in? -- Des NowSMS Support | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2124 Registered: 08-2008 |
Follow-up... Never mind. I've just confirmed the problem. There was a last minute performance optimisation that was not implemented correctly. If you have the UseRouteQueues=Yes setting applied, then it works properly. If this setting is not applied, then the messages get stuck. It is a very simple fix. One line of errant code. An update should be available within the hour. -- Des NowSMS Support I'm already testing a fixed SMSGWS.EXE. We are no | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2126 Registered: 08-2008 |
Hi Chris, I've posted that update. The version identifier is 2010.05.17. The download link is the same as before http://www.nowsms.com/download/nowsmsupdate.zip. -- Des NowSMS Support | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 100 Registered: 07-2006 |
Hi Des, Are you sure that patch is correct, GUI version says it's still 20100310 | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 101 Registered: 07-2006 |
sorry, incorrect post, all seems to be fine | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 103 Registered: 07-2006 |
Hi Des, Release 2010.05.17. has a bug. NowSMS random rejects messages submitted over HTTP like account has 0 balance but it's not (UserDir folder at share). Maybe that bug also affects SMPP users, haven't tested yet. Please advise. Regards, Alex K. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2137 Registered: 08-2008 |
Hi Alex, There have been no changes in this area. What version were you running before? -- Des NowSMS Support | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 104 Registered: 07-2006 |
Before we used 2010.03.10 and conditions were the same. More explanations: User tries to submit message over HTTP, (balance > 0) he got 403 error, means that balance reached 0 BUT statistics got increased and balance deducted, also we receive SMSSend callback, other callbacks will never be issued and no records in SMSOut log. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2138 Registered: 08-2008 |
Something doesn't make sense Alex. The "SmsSend" callback (and the quota adjustment) occurs after the message is written to the queue. The "PreAuth" occurs before the message is written to the queue. There is no way that the user would receive a 403 error if the message was actually queued. The queued message should be recorded in SMSWEB-yyyymmdd.LOG. Is it? Can you see no sign of this message file anywhere? | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 105 Registered: 07-2006 |
Please excuse me for a mess, my developers mislead me. Let me start from a scratch. Seems we have 2 problems here: 1. Users with positive balance receive error 403, we've tracked that bug because we have special service pushing messages from web queue to NowSMS. As soon as service gets 403 - no more messages will be sent for particular user. 2. Messages disappear after SMSSend callback, we see messages in DB, statistics increased and balance deducted but no records in SMSOut log. Unfortunately we've disabled SMSWEB log for performance reason and it's useless for us. We rolled back 2010.03.10 and situation become normal. Please look check one more time. Regards, Alex K. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2144 Registered: 08-2008 |
Hi Alex, OK ... it sounds like you are having a similar problem to what Chris described. There was definitely a problem with the 2010.05.07 release, where if you submitted a message with explicit routing information, this message would get stuck in the message queue, unless UseRouteQueues=Yes was specified in the SMSGW.INI file. 2010.05.17 fixed this problem. I hate to ask you to update again, but I don't know how else to resolve the problem, because I can't recreate any problems. It sounds to me like messages are getting queued, but the routing information is inconsistent. If you update, there are two configuration settings that can be relevant. UseRouteQueues=Yes enables some logic that optimises message queueing for installations where messages are frequently sent to an explicitly named route. (Or where the route is set by an accounting callback.) By default, UseRouteQueues=No is assumed. UseRouteCache=No disables a new routing algorithm. If you encounter problems with the update, adding the setting UseRouteCache=No should make the routing algorithm behave as it did before. Restart the server, and the queued messages, will get picked up and routed. However, the performance benefit of UseRouteCache=Yes is big, so we need to figure out why it is not working for you. What can you tell me about your configuration? How many SMSC connections? I seem to recall that you use accounting callbacks to set the routing fo messages? Do you have RouteName values configured for the connections, or do you use the SMSC address as the route? Was it only messages for certain routes that were getting stuck? -- Des NowSMS Support | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 106 Registered: 07-2006 |
Hi Des, Thank you for the response! We use only explicit routing defined by callbacks and UseRouteQueues=Yes is always set. Here part of our SMSGW
Today we will try UseRouteCache=No and try to figure out more results. Our configuration: 2CPU (16 cores), 24GB DDR3 RAM, RAID5 ON SSD disks. We have about 20 SMSC connections and all messages are being routed by PreAuth callback with SMSC address (only explicit routing). Messages for all connections got stuck in the queue. Regards, Alex K. | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 107 Registered: 07-2006 |
Hi Des, We've made all test again and can confirm that: Problem 1 - exists in release 2010.05.17 Problem 2 - NowSMS issues PreAuth & SMSSend and occasionally doesn't issue SMSOut & SMSIN. Manipulations with UseRouteCache - no effect. Please take a look into the problem. Regards, Alex K. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2150 Registered: 08-2008 |
Hi Alex, I'm at a loss to explain any of this. We've spent most of today trying to recreate problems like you describe, but we can't find any problems. In fact, I'm still not sure exactly what problem we're looking for with regard to #2. It is likely that whatever problems are occuring are related to accounting callback processing, and it may also be specific to how your SMSC connections are defined. We've tried to implement accounting callbacks similar to what we think you are using, returning routing information using full SMSC addresses, but we have not seen any problems with this, with or without UseRouteQueues=Yes set. I can't see anyway of resolving this without debug information from your system. (If you want to send any debug logs to nowsms@nowsms.com with "Attention: Des" in the subject line, I will look at them.) Regarding #1. The way I understand the problem, is that you are performing HTTP message submissions, and receiving an unexpected error 403, indicating a credit balance problem, even though the user still has a positive credit balance. There have been no changes to how credit balances are handled for submitted messages since the version that you were running. We've looked for anything that could be affecting this and found nothing. It should be noted that a 403 quota error could be triggered by either an actual credit balance exceeded, or by a failure in the pre-auth accounting callback. It can be difficult to determine why without looking at debug logs. However, one thing that might tell us whether it is an accounting callback problem, or a credit balance problem ... is if you check the Windows Event Log. Do you see indications like this: "Account name: credit limit exceeded." If you do, then NowSMS does think the credit balance is used up. If you do not, then something else is happening, most likely something is going wrong in connecting to the accounting callback URL. Other than adding additional parameters to the accounting callbacks (SMPPOptions parameter values), there have been no changes in how they are handled. However, I think we need to determine whether it is a perceived negative credit balance that is incorrect, or some other issue. Regarding #2 ... Are the messages being sent or not? I was assuming that you were having a problem similar to Chris (who started this thread). However, re-reading your messages, that is not clear. Are you saying that the messages appear to be sent out (statistics increase to indicate messages sent) ... however there are some missing "SMSOut" callbacks. (And apparently also some missing "SMSOUT" log records. It would be helpful if you re-enabled "SMSWeb" so that we could see the submit details, as well as the dispatch/out details.) I need to better understand what you are saying is happening. Is the message actually sent? Is it stuck in the Q directory? I am wondering if both of these problems are accounting callback related. That is ... if NowSMS experience a failure in a "PreAuth" callback, it will return a 403 error to the client that is attempting to submit a message. I'm wondering if a similar fate isn't occurring for the missing "SMSOut" and "SMSIn" callbacks? Do you make use of the SMPPOptions setting? By any chance is there an EXCEPT.LOG file present in the NowSMS directory? With entries that have recent dates? -- Des NowSMS Support | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2156 Registered: 08-2008 |
Alex, We had another report with more detail which explains why this problem has been so hard to recreate. Are you using the ReRouteReceived= setting for one or more of your SMSC connections? This setting appears to be broken in the recent update (due to changes related to supporting the accounting callbacks to support routing information). An update is being prepared to address this issue. -- Des NowSMS Support | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 108 Registered: 07-2006 |
Hi Des, No, we don't use such parameter. Regards, Alex K. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2157 Registered: 08-2008 |
Hi Alex, We're still stumped then. This other problem was very specific to the use of the ReRouteReceived= setting. (Messages processed by this directive would get stuck in the queue.) There will be an update posted shortly to address that issue. I still have many questions about the problem(s) y you are experiencing. Regarding #1, I am curious if the Windows Event log has records of "Account name: credit limit exceeded". This event is recorded if a user account exceeds its credits. If you are getting a 403 error and this event is not recorded, then it suggests that a problem is occurring in the interface to the accounting callback. I don't know why accounting callbacks would be a problem in the new release. The only change is that SMPPOptions parameters are now passed to them. But it would be helpful to know whether it is a credit balance issue or a problem interfacing to the accounting callbacks. Regarding #2, it's not clear to me what statistics are updating. Are messages stuck in the queue? Or are all messages actually being sent, but no "SMSOut" callback and no "SMSOut" log file entry? Is there an EXCEPT.LOG file in the NowSMS directory with recent date entries? The only potential issue that I can see is that the "AsyncCallbackThreads=30" setting is very high. I'd be hesitant to set this value higher than 3. I can't see a reason why this setting would be any different in the newer version vs. the older one. However, this high of a value for this setting could be dangerous. I'll post a note when the new update is posted. However, I think we need more info before we can determine what is causing your problems. -- Des NowSMS Support | ||||
| Bryce Norwood - NowSMS Support Board Administrator Username: Bryce Post Number: 7912 Registered: 10-2002 |
Hi Alex, A follow-up. The update mentioned by Des has been posted to http://www.nowsms.com/download/nowsmsupdate.zip. It corrects a problem where messages processed by the "ReRouteReceived" directive were being added to the queue with invalid routing information, which caused them to get stuck in the queue. -bn | ||||
| Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 109 Registered: 07-2006 |
Dear Des and Bryce, We decreased AsyncCallbackThreads to "3" and situation became normal, seems that your new release has influence from that parameter. Thanks a lot for your brilliant support! Regards, Alex K. | ||||
SMSGW.INI
