| .rct files stuck in endless loop |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through April 11, 2011 ⬆ |
◄ ► |
| Author | Message | |||||
| Julio da Silva New member Username: Omniscout Post Number: 2 Registered: 04-2006 |
our server just recently started or more so would stop processing out bound SMS. by restarting the server it would start sending but usually after a day, sometimes less it would get stuck again. In looking in the SMSDEBUG.LOG I found that it was in a loop with OK 10:22:28:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:28:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:29:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:29:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:30:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:30:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:30:203 [16] ModemReceiveMessages: AT+CPMS? 10:22:30:203 [16] ModemReceiveMessages: +CPMS: "ME",24,25,"SM",1,50,"SM",1,50 OK 10:22:31:062 [18] ModemReceiveMessages: AT+CPMS? 10:22:31:062 [18] ModemReceiveMessages: +CPMS: "ME",22,25,"SM",1,30,"SM",1,30 OK 10:22:31:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:31:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:32:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:32:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:33:093 [9] ThreadProcessInboundReceipts: Processing 49D08B60.rct... 10:22:33:093 [9] ThreadProcessInboundReceipts: Processing 49D08B72.rct... 10:22:33:281 [18] ModemReceiveMessages: AT+CPMS? 10:22:33:281 [18] ModemReceiveMessages: +CPMS: "ME",22,25,"SM",1,30,"SM",1,30 I would then delete these from the SMSIN queue and it seems to get going again... This is our production server and we send critical SMS alerts to our customers and right now we are having to manually babysit the server... Help Please | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3040 Registered: 08-2008 |
Hi, Those are delivery receipts where the message ID in the receipt cannot be resolved. I would need to see SMSDEBUG.LOG entries showing the messages being sent and the receipts coming in to understand why those receipts are not being resolved. However, that situation would not block other activities from occurring. Messages can still be sent and received. There is no looping, just retries on the receipt processing occurring independently. So I don't really understand what the problem is you are experiencing. Is other activity stopping? -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 3 Registered: 04-2006 |
Hi Des Yes, we have 2 issues. 1. for some reason the server will stop sending SMS. This is what it did... a couple of weeks ago we noticed that SMS would get stuck being sent out. We would reboot the server and all would start working again. We had this problem once before and just as a precaution an upgrade that I was meaning to do was done. that seemed to resolve the problem. Well it started again like I said 2-3 weeks ago where it would just stop sending. So I saw there was an upgrade and I did it. the problem of SMS not going out did not go away. 2. The .rct files are just filling up the SMSIN folder. now I have a new problem of these .rct files which honestly I have never seen before. I have attached the SMSDEBUG and also a copy of the SMS ini files. Is there anything else you need?
| |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3041 Registered: 08-2008 |
Investigating why there are so many .rct files is the easiest to troubleshoot. So let's start there, please email SMSOUT-yyyymmdd.log and SMSIN-yyyymmdd.log for today to nowsms@nowsms.com. Put Attention: Des in the subject line. | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3042 Registered: 08-2008 |
I'm curious about one comment in your first message. Are you saying that when outbound activity seems to have stopped, if you delete these .RCT files, activity seems to resume? How many .RCT files are we talking about? (tens? Hundreds? Thousands?) Does activity stop on all connections, or does it seem to be just one connection? (Look at SMSOUT-yyyymmdd.log and this will have recorded which connections outbound messages were sent through.) | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3043 Registered: 08-2008 |
Hi Julio, I've looked over your logs, and I do have some more information on the .RCT issue. It appears very consistent that no delivery receipts that come from the server2.msgtoolbox.com connection get processed as expected. They all end up in this retry queue, and ultimately fail getting their message ID resolved. Usually the reason for this is because NowSMS does not track receipt message IDs in cases where the submitting client does not ask for one. A quick fix is to add a configuration parameter to tell NowSMS to always request a delivery receipt for messages submitted via that connection. To do this, manually edit SMSGW.INI and under the [SMPP - server2.msgtoolbox.com:2775] header, add DefaultDelReceipt=Yes From the point where this setting is applied, NowSMS will then start asking for delivery receipts from this connection, and because it asked, it will track the IDs, and delivery receipts won't get stuck in this RCT queue. For the other problem, I'd like to see if we can get some logs from a period of time during which activity seems to be stopped. Hopefully the two problems are related, and fixing one fixes the other. (Even though I do not understand how they could be related.) -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 4 Registered: 04-2006 |
Not sure if it is coincidence or not with the .RCT files being deleted that the SMS is being held. What I have noticed is that the SMS is being held only on the Modems, but the SMPP connections seem fine. | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3045 Registered: 08-2008 |
Are there errors posted to SMSOUT logs with regard to the modems? I was suspicious about the repeated AT+CPMS? entires in your log. The results suggest that there are received messages waiting to be processed, and that modem memory is close to full. (Sending can block with many modems if receive memory is full.) I've already put in a request for one of my engineering colleagues to explain to me why these log entries would be in SMSDEBUG, but no messages being received from the modem. A setting may be helpful in getting more information. In the [SMSGW] section of SMSGW.INI, add DebugModemPoll=Yes This should add to the verbosity of what is logged in SMSDEBUG. I'd like to see this, because something doesn't seem right to me. -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 5 Registered: 04-2006 |
Hi Des Thanks again for all the help... it too is confusing me why it behaves in such a manner. I have done a couple of the changes you requested... see attached SMSGW.INI file. Unfortunately the entry of: DefaultDelReceipt=Yes is not working. I am still getting .RCT files hanging around in that folder. I have also added the: DebugModemPoll=Yes to the INI file and do see a lot more information. how long do you want the information to accrue the log before sending to you?
| |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3046 Registered: 08-2008 |
Go ahead and send the log now. I am most likely going to have a colleague take a look at it. Actually at both issues. I do recall an incident where message IDs longer than 20 characters were not resolved properly. I notice these IDs are exactly 20 characters, so I am wondering if there might still be an issue with ID length. Along with the SMSDEBUG.LOG, look for the SMPPDATA directory which will have a folder for each SMSC. Send me the .db files, so I can check to make sure that the IDs are not being truncated when stored. -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 6 Registered: 04-2006 |
Ok... attached is the file SMSDEBUG... I do have a directory called SMPPData but it is empty.
| |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3047 Registered: 08-2008 |
I didn't notice it before, but the DisableDeliveryReceipt=Yes setting is taking precedent. Because that is set, NowSMS is not tracking delivery receipts and not asking for them on a global basis across all connections. But this provider is returning delivery receipts even though they are not asked for. If you remove that setting, then the other setting will take effect. -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 7 Registered: 04-2006 |
Sorry, I cleaned up the INI file so things would be clearer... do I keep the DebugModemPoll=Yes running until otherwise told? is there any other logs or information you need? I have noticed that since I removed the DisableDeliveryReceipt=Yes I am not seeing any .RCT files anymore. Any Ideas as to why I would not have the .db files in the SMPPData directory? | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3048 Registered: 08-2008 |
Hi Julio, We have made process. The DisableDeliveryReceipt=Yes setting is what was causing the .RCT files, and is why the SMPPDATA files I asked about did not exist. You should now see database tracking files under that directory. I should have noticed that setting earlier, but missed it. It is likely that what changed is that this particular provider started returning delivery receipts even though they are not asked for. I will be asking a colleague to check the modem issue to see if there is something more to the part I do not understand. It is ok to remove DebugModemPoll=Yes. We do want to understand, however, when one or more modems stop sending, are any modem errors recorded in the SMSOUT log files? -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 8 Registered: 04-2006 |
Hi Des I have removed the Modem Debug and restarted the server. I am not seeing any .RCT files anymore and I do have for the first time data in the SMPPData directory. The files however are .D2A and .D2I Are these the files you want? | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3049 Registered: 08-2008 |
Hi Julio, I dont need the SMPPDATA files any more. I was trying to figure out why receipts weren't working, which led me to these files. But the fact the files were not there, led me back to studying your configuration. The reason the receipts were not being processed, was because these files were not present, which is because of that setting. -- Des NowSMS Support | |||||
| Julio da Silva New member Username: Omniscout Post Number: 9 Registered: 04-2006 |
Hi Des The server now has messages stuck again. I sent an email attention you with a copy of in the SMSIN SMSOUT and SMSDebug logs along with a copy of the messages stuck (16). Te process I tried to get them going again was to stop the services and then restart them. Sometimes that works but not often. This time it did not so I did a reboot of the server. Once the server restated the messages were delivered. | |||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 3054 Registered: 08-2008 |
Hi Julio, I think the modem is dying. I found a very good record of what was happening in one of the previous files. I assume that this is not a new modem, but one that you have been using for quite some time, and it may be experiencing some sort of internal failure. What type of modem is this, is it a USB modem? I'm guessing that it has to be a USB modem, because it appears to be locking up inside of an API call. We detect that the modem is not responding, and ask to cancel the operation, but instead of canceling, the modem driver appears to be locked up. In all likelihood, it was locked up before we asked to cancel. Unfortunately, when this happens with a USB device, there is no option other than to reboot. It is also possible to remove the device and re-insert. However, unfortunately there is no way to do this via software. I am asking our engineers to implement a watchdog process to monitor for this situation to trigger an automatic reboot. However, that said, a frequently rebooting server is obviously not a good solution at all. The best solution is to replace the modem, as it seems to be failing with regular frequency. On Thursday, after you added the additional debug parameters, that modem died within 5 minutes of the server being restarted. For a temporary resolution, it may help to remove the modem and re-install it. Also, check the manufacturer web site to see if there are any updated drivers before re-installing. It also may be worth adding the following parameter to the [SMSGW] section of SMSGW.INI: ModemMaxBytesPerSend=1 This tells NowSMS to slow down the interfacing with modem devices. I have seen situations where poorly written USB modem drivers could get overloaded and lockup. I've rambled a bit as I thought through this, so let me summarise my recommendation: 1.) Remove the modem from the system and re-install it. Check for new drivers from the manufacturer before re-installing. 2.) Edit SMSGW.INI, and under the [SMSGW] header add ModemMaxBytesPerSend=1 3.) Get a replacement modem. I'm hopeful that #1 and #2 will be enough to buy some time to acquire a replacement. -- Des NowSMS Support | |||||
