| Duplicate Messages |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through October 31, 2007 ⬆ |
◄ ► |
| Author | Message | |||
| Mark Ramirez New member Username: Ramirez Post Number: 3 Registered: 07-2006 |
Hi, Recently, we've been getting issues with duplicate messages. The program that processes all incoming messages is receiving and processing duplicate messages. We asked the sender and they said they only sent it once. These duplicate messages are almost ALWAYS the 2nd part of a long SMS (in excess of 160 characters). The time gap between these dupes are sometimes hours apart. And we get this duplicate message 5-10x before it stops. Normally, if the duplicate messages are a result of the carriers messing up, the gap is less than 5 minutes and include even short messages (less than 160 chars). We know to filter dupes with time gap of less than 5 minutes. But this time, the gap is longer and almost always the second part of a long message. We don't want to filter any dupes greater than 5 minutes because sometimes users send the same exact message several hours later legitimately. It seems like when we receive long SMS, they all gather in the SMS-IN directory as multipart message and the first part is processed correctly but the 2nd part stays there for several hours causing it to be processed multiple times. I've seen files in this directory that were received several days ago and are still waiting for the first or second part of the message. When it doesn't arrive or the first part was processed prematurely, these files remain there and maybe get processed multiple times. Can you explain to me the process of how files in SMS-IN are processed? Correct me if I'm wrong. The data goes thru these stages: 1. received in the modem 2. created in SMS-IN directory 3. written in SMS-IN.log 4. written in SMSDEBUG.log 5. passed to our processing program Thank for your support. | |||
| Mark Ramirez New member Username: Ramirez Post Number: 4 Registered: 07-2006 |
I forgot to add. I've also noticed that the time the duplicate messages were logged in SMSIN.log and the time our program processed it are not the same (sometimes half an hour later. For example, let's say the duplicate message is received at 1PM, our program processed this data (and logged in our database) half an hour later. This is what leads me to the conclusion that there is a glitch in hour files in SMS-IN directory is processed. The duplicate messages create a file in the SMS-IN directory but the NOWSMS daemon program doesn't pick it up until later and pass it to our processing program. Am I wrong in this assumption? Is there we can do to prevent this duplication? Thank you again. | |||
| Malcolm - Now Support Moderator Username: Malcolm Post Number: 254 Registered: 12-2006 |
Hi Mark, I'm not sure exactly how to answer your question ... but perhaps it would help to explain how NowSMS handles long messages when routing them to a 2-way application. First the received SMS messages are placed into the SMS-IN directory with a file extension of ".IN". Another daemon scans these message files to see if they are parts of a long message. If they are, they are renamed to files with a ".SAR" extension. When all of the parts of the message are received, NowSMS then recombines the message into a single long message, and routes it to the appropriate 2-way command (or if it is an MMS notification, retrieves the MMS). If all of the parts aren't received within 30 minutes, we go ahead and route it to the 2-way command with missing parts. I'm not aware of any glitches in this logic. The only problem that I recall was that if a "-" character appeared in a sender address, this confused the recombining process. ("-" would appear in the sender address only for a message with an alpha sender. In this case, a particular operator was sending out MMS notification messages with an alpha sender that included a "-" character ... so this would not be your situation.) What do you see in the SMSIN log compared to what is routed to your application? Are there missing pieces of the messages? Are the duplicate messages appearing as duplicates in the SMSIN log? Unfortunately, the 30 minute timeout is not configurable at this time. -malcolm | |||
| Mark Ramirez New member Username: Ramirez Post Number: 5 Registered: 07-2006 |
Hi Malcolm, Thanks for the thorough explanation. I went over the SMSIN log and compared it with the list of messages that my program processed and there were 2 groups that I can classify the messages in: 1.) the messages that were not duplicated and timestamp matched 2.) the ones that were duplicated and timestamp did not match. This second group contains all Binary so I can't see what the messages are but when I group them by the sender phone number, they match in quantity with the duplicate messages in my list (that my program processed). But the interesting thing about this is that the timestamps do not match. There is about 30 minutes gap between the timestamp in the SMSIN log and when my program processed them so my program didn't process the messages until 30 minutes after it arrived. Also, they all look like the 2nd part of a long msg (I checked and we did receive the complete msgs previously), so the 2nd part of this msg keeps getting passed to my program again and again (avg 4-5 times). I can't read the binary but they are duplicates when I group them by the sender. The 30 minute gap between when my program processes the messages and when they are logged in SMSIN log AND the 30 minutes you allot to process the messages regardless can't be a coincedence. What do you suppose is going on? Seems like the second part of a msg is waiting in SMSIN directory but it doesn't find the first part so gets sent anyway. And if my program processes the same message over and over again, it means that that .IN file doesn't get erased from the SMSIN directory after it has been processed or maybe the carrier keeps sending that 2nd part over and over again. I appreciate your help. I'm just trying to understand what's going on just in case we can do something to get around this problem. Thank you. | |||
