Unable to match Deliver Receipts

Unable to match Deliver Receipts SearchSearch
Author Message
Hareeharan Kandavel
New member
Username: Haree

Post Number: 16
Registered: 12-2012
Posted on Monday, July 29, 2013 - 01:52 pm:   

Hi,

We have recently discovered that some message receipts returned from NowSMS contains the message id from the SMPP provider rather than the message id that NowSMS provided us when the message was submitted via HTTP. After investigating those unmatched receipts, we realised that the parameters, SMSCReceiptMsgId and SMSCReceiptMsgIdOrig have the same values.

As a result, when the Two-Way mechanism is called and posts to our URL, the message Id that is provided is different to the Id originally provided to us when the message was sent.

This has lead us to believe that the NowSMS gateway is sometimes unable to match back a receipt message Id to the original MT message.

Please find the sample receipt below:

2013-07-29 06:10:43,+10000000000,SMPP - smpp.abcd.com:xxxx,Text="id:UfX2738AAAEAAD¡TSsk sub:000 dlvrd:000 submit date:130729050031 done date:130729050031 stat:UNDELIV err:515 text:xxxxx ";SMSCRouteName=SMPP-ABCD;Recip=****;SMSCReceiptMsgId=UfX2738AAAEAAD¡TSsk;SMSCR eceiptMsgIdOrig=UfX2738AAAEAAD¡TSsk

Could you please look into it and ensure that we have the smscid?


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4581
Registered: 08-2008
Posted on Monday, July 29, 2013 - 07:00 pm:   

Hi Haree,

Can you find that same message ID reported as the SMSCMsgId in an SMSOUT log record? Or otherwise manually locate the corresponding SMSOUT record so we can manually match message ID?

I am suspicious because there is an odd character in the message ID that is being reported ... ¡ is the upside down exclamation point ... and I find it very unusual that this character would be present in a message ID.

I am guessing that the original message ID had an @ character in this place.

If the SMSC is using the GSM character set, and the delivery receipt does not contain the receipted_message_id TLV, which means we have to parse the message ID from the text, but the SMSC uses iso-8859-1 encoding for the message ID, NowSMS is going to interpret iso-8859-1 "@" as GSM "¡".

If the SMSC used the receipted_message_id TLV or was consistent with character set usage, this would be be OK.

If my theory is correct, we could implement a work-around. But we need to confirm this is the issue. Find the corresponding SMSOUT log entry and check the SMSCMsgId value reported.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 17
Registered: 12-2012
Posted on Tuesday, July 30, 2013 - 10:21 am:   

Hi Des,

Thank you for your response.

Your guess is right actually. Please find the SMSCMsgId from ther SMSOUT log below:

SMSCMsgId=UfX2738AAAEAAD@TSsk


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4582
Registered: 08-2008
Posted on Tuesday, July 30, 2013 - 01:15 pm:   

Hi Haree,

We're working on a solution to address this.

In the meantime, can I ask...are you able to send messages that contain @ without problems?

The reason I ask is because if you change your SMSC character set to iso-8859-1, this would resolve the delivery receipt issue. However, you would not want to do that if it causes outbound character set problems.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 18
Registered: 12-2012
Posted on Tuesday, July 30, 2013 - 01:59 pm:   

Hi Des,

I could confirm that the SMSC character set is already set to iso-8859-1 and we could confirm that we don't have any issues when delivering @ in the message content.

Hope this information helps.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4583
Registered: 08-2008
Posted on Tuesday, July 30, 2013 - 04:43 pm:   

Hi Haree,

That is unexpected...and might affect how we work-around this issue.

I was assuming we were receiving a delivery receipt where the text read id:UfX2738AAAEAAD@TSsk encoded with iso-8859-1 but the SMSC character set was set to GSM. That would cause the @ to be interpreted as ¡.

If the SMSC character set is set to iso-8859-1, then it is unclear where the character set confusion is being introduced.

Do a large number of messages get assigned a message ID that includes this @ character?

Do only some receipts have the problem while other receipts with @ message IDs get handled correctly?

Is it possible that receipts that have a problem are only for messages that were binary or Unicode format? (Testing another theory...if it is only receipts for binary or Unicode messages, this may be an easy work-around.)

Sorry to drag this out further, but I want to make sure we understand why there is character set confusion. If this happens frequently or can easily be recreated, could you capture a debug log or WireShark trace? For the debug log, enable SMSDEBUG.LOG ... this will enable SMPPDEBUG.LOG which is the file we are interested in seeing. We'd like to see the raw SMPP data for one of these receipts that is not getting handled properly.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4584
Registered: 08-2008
Posted on Wednesday, July 31, 2013 - 09:58 pm:   

Hi Haree,

I wanted to follow-up with you on this.

Without seeing the raw SMPP data, we believe one of two possibilities is occurring:

1.) Received delivery reports experiencing the problem are arriving with the wrong data_coding value. We have seen some SMSCs before where delivery reports for binary and/or Unicode messages are received back with the wrong data_coding value (same as original message). In this case, if we have to parse the message ID from the text of the receipt (because the receipted_message_id TLV is not present), we assume the text is encoded with GSM.

If the delivery report has a standard text data_coding setting, then we use the configured SMSC character set. It is only for the incorrect data_coding value that we assume GSM text encoding. We will extend this handling of an incorrect data_coding value in the receipt to assume the configured SMSC character set.

2.) It is also possible that the upstream SMSC is getting confused and actually returning an incorrect value. We deal with quite a few quirks in SMSC receipt formatting, so we will add handling for "¡" in a receipt ID to also look for a receipt match as if it was "@".

One or both of these changes should resolve your issue.

In either case, I don't think NowSMS is doing anything wrong ... it sounds like a problem with how the SMSC is formatting the receipt message. But I would need to see the raw SMPP to confirm.

In the meantime, please try the update at http://www.nowsms.com/download/nowsms20130731.zip, which includes the two changes mentioned above.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 19
Registered: 12-2012
Posted on Thursday, August 08, 2013 - 10:30 am:   

Hi Des,

Apologies for the delayed response as the issue is occurring sporadically which delayed the packet capture.

Thank you for the update. However, we couldn't test the update on our test environment as we are unable to manipulate the message ID. So, the only option to test the release would be to move them on to Live.

Before we do that, we would like to have a confirmation from you whether the release will definitely fix the issue. To help you with this, I have managed to capture the raw SMPP data and have sent an email with the subject "Unable to match Deliver Receipts" to the email id: nowsms@nowsms.com.

Hope the information provided helps you with further investigation. Please let me know if you need more information. Also, could you please ensure that the packet is not posted on to the forum?

Hoping to hear from you soon.

Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4594
Registered: 08-2008
Posted on Thursday, August 08, 2013 - 03:13 pm:   

Hi Haree,

OK ... I see what is going on.

The good news is that I don't think an update is required, just a configuration setting applied.

Here is what is happening.

The delivery receipt does not have a receipted_message_id TLV parameter present. This means we have to parse the message ID from the text of the message (which is very common).

The data_coding value in the message is 1, which means the message text is encoded using the GSM character set. (Which is all very odd, considering that all of your other communications with the SMSC are using iso-8859-1.)

However, the @ in the message ID is encoded with an iso-8859-1 value of 0x40, which is interpreted as ¡ in the GSM character set.

So the problem is that the SMSC is telling us explicitly that the message is encoded using the GSM character set, when it is not.

For situations like this, there is a configuration setting that tells NowSMS to use the character set configured for the SMSC instead of the character set specified in the text message.

Under the server specific section header (e.g., [SMPP - server:port] of SMSGW.INI, add:

SMSCCharsetReceiveTextOverride=Yes

This won't affect Unicode or binary messages, just text messages, so I would not expect any other side effects.


--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 20
Registered: 12-2012
Posted on Thursday, August 08, 2013 - 04:53 pm:   

Hi Des,

Many thanks for the update.

Thanks,

Haree
Hareeharan Kandavel
New member
Username: Haree

Post Number: 21
Registered: 12-2012
Posted on Friday, August 30, 2013 - 03:17 pm:   

Hi Des,

Could you please confirm the oldest version of Nowsms this configuration setting can be applied?


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4619
Registered: 08-2008
Posted on Friday, August 30, 2013 - 03:38 pm:   

This is the reference that I see in our changes history:

2009-11-25:

* SMPP: Add a configuration option to override the character set for received SMS messages. A customer encountered a situation where the SMSC was encoding text using the iso-8859-1 character set. However, the SMSC was setting the character set flag in the message to indicate that text was using the GSM character set. NowSMS was configured to use the iso-8859-1 character set with this SMSC, however, since the character set was explicitly set as GSM in the received message, NowSMS could not decode the text properly. To address this issue, it is possible to add SMSCCharsetReceiveTextOverride=Yes under the [SMPP - server:port] section of SMSGW.INI. When this parameter is present, NowSMS will always interpret received text messages as being encoded in the character set configured for the connection.