Incorrect GSM encoding for Alphanumeric sender IDs

Incorrect GSM encoding for Alphanumeric sender IDs SearchSearch
Author Message
Hareeharan Kandavel
New member
Username: Haree

Post Number: 5
Registered: 12-2012
Posted on Wednesday, February 20, 2013 - 01:06 pm:   

Hi Support,

We have been doing some testing with GSM characters in Alphanumeric sender IDs. We realised that Nowsms did not encode a few characters as per the GSM character set. However, we did not have any issues with encoding those characters in the message content.

The incorrect encoding of GSM characters in Alphanumeric sender IDs include

Hex Character
0x00 @
0x01 £
0x02 $
0x03 ¥
0x04 è
0x05 é
0x06 ù
0x07 ì
0x08 ò
0x09 Ç
0x0A
0x0B Ø
0x0C ø
0x0D
0x0E Å
0x0F å
0x10 Δ
0x11 _
0x12 Φ
0x13 Γ
0x14 Λ
0x15 Ω
0x16 Π
0x17 Ψ
0x18 Σ
0x19 Θ
0x1A Ξ
0x1C Æ
0x1D æ
0x1E ß
0x1F É
0x3A :
0x40 ¡

Could you please help me with this so that all the above symbols could be used in Alphanumeric sender IDs?


Thanks

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4298
Registered: 08-2008
Posted on Thursday, February 21, 2013 - 12:05 am:   

Hi Haree,

I don't know what type of SMSC connection you are using.

I am guessing SMPP?

SMPP does not explicitly state what character set should be used for alphanumeric sender, so it may vary based upon provider.

I suppose we should use whatever character set NowSMS is configured to use for the message text (we default to GSM for the message text). However, we use whatever the URL character set is.

I will escalate this to engineering for further analysis, but I do have a concern that since the SMPP spec defines the field as a null terminated string, it will not be possible to support the @ character when the GSM character set is used.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 6
Registered: 12-2012
Posted on Wednesday, February 27, 2013 - 11:55 am:   

Hi Desosms,

Can I have an update on this please as one of our customers would like to use special characters in AlphaNumeric sender ID's?


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4300
Registered: 08-2008
Posted on Wednesday, February 27, 2013 - 03:39 pm:   

Hi Haree,

You have not answered my previous question. Let me restate and add additional questions.

I still don't know what type of SMSC connection you are using.

I am guessing SMPP?

What protocol is the customer using to connect to NowSMS?

What version of NowSMS are you running? I may be able to release an updated SMPP DLL that gives the option to use the GSM character set to encode alphanumeric senders. However, even with that, I cannot guarantee how your service provider will encode the resulting message. I am only guessing that they are expecting the GSM character set.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 7
Registered: 12-2012
Posted on Wednesday, February 27, 2013 - 04:03 pm:   

Hi Des,

Apologies. Since the issue was escalated, I assumed that you would get back to me.

I could confirm that I am using SMPP connection.

We are using HTTP protocol to submit the messages to NowSMS.

Currently, we are running version 2012.02.16.

An updated SMPP DLL release would be highly appreciated. Once this is done, I will take this to the service providers to encode appropriately.


Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4302
Registered: 08-2008
Posted on Thursday, February 28, 2013 - 07:08 pm:   

Hi Haree,

I've posted an update to only the smssmpp.dll file at http://www.nowsms.com/download/smppharee.zip

To install this DLL, you need to update to NowSMS 2012.06.28 or later.

2012.06.28, the official download is at http://www.nowsms.com/download-free-trial

And 2012.11.30, a stable interim update is at http://www.nowsms.com/download/nowsms20121130.zip

After running the install to update to either of those two versions, manually replace smssmpp.dll using the file in the zip.

This update will encode alphanumeric sender addresses using the GSM character set if the SMPP character set is default or GSM/IA5.

Note that it will not be possible to send the @ character in an alpha sender in this scenario. The reason is an SMPP limitation. The sender address parameter is defined as a null terminated ASCII string. In the GSM character set, @ is encoded as a null character. As null is being used as a terminator, it cannot be included in the string itself.

This is not an issue for message text because the message text field in SMPP has an explicit length parameter and does not use null termination.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 8
Registered: 12-2012
Posted on Thursday, March 07, 2013 - 11:24 am:   

Hi Des,

Update to the DLL is very much appreciated.

I have followed your instructions and did some testing which sorted out the issues with a few special characters.

However, I am afraid NowSMS still don't encode quite a few characters according to GSM encoding.

Please find the list of unsupported characters below:

LATIN SMALL LETTER I WITH GRAVE ì
LATIN SMALL LETTER O WITH GRAVE ò
LATIN CAPITAL LETTER C WITH CEDILLA Ç
LATIN CAPITAL LETTER O WITH STROKE Ø
LATIN SMALL LETTER O WITH STROKE ø
LATIN CAPITAL LETTER A WITH RING ABOVE Å
LATIN SMALL LETTER A WITH RING ABOVE å
LATIN CAPITAL LETTER AE Æ
LATIN SMALL LETTER AE æ
LATIN SMALL LETTER SHARP S (German) ß
LATIN CAPITAL LETTER E WITH ACUTE É
CURRENCY SIGN ¤
COLON :
LATIN CAPITAL LETTER A WITH DIAERESIS Ä
LATIN CAPITAL LETTER O WITH DIAERESIS Ö
LATIN CAPITAL LETTER N WITH TILDE Ñ
LATIN CAPITAL LETTER U WITH DIAERESIS Ü
LATIN SMALL LETTER A WITH DIAERESIS ä
LATIN SMALL LETTER O WITH DIAERESIS ö
LATIN SMALL LETTER N WITH TILDE ñ
LATIN SMALL LETTER U WITH DIAERESIS ü
LATIN SMALL LETTER A WITH GRAVE à


Could you please investigate this and suggest me a solution?

Looking forward to your response.

Thank you.


Kind Regards,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4305
Registered: 08-2008
Posted on Thursday, March 07, 2013 - 03:08 pm:   

Hi Haree,

As I mentioned previously ... I cannot guarantee how your service provider will encode the resulting message.

The change that we made was to encode the alphanumeric sender using the character set configured for the SMPP connection (default being GSM).

I tested with a few of the characters in your list (starting with the first one ... ì ... GSM character #7), and the resulting SMPP message showed the expected GSM character value.

I would strongly suggest that you use Wireshark for testing so that you can inspect the SMPP encoding for yourself in order to better determine where the "problem" exists.

That said, I do see two potential areas of confusion.

1.) The colon ":" character is used as a delimiter for port addressing by NowSMS. If it is present in a sender address, NowSMS will truncate the sender address at the ":" and remove all numeric characters. To disable this behaviour add AllowAlphaRecip=Yes to the [SMSGW] section of SMSGW.INI. (We will add automatic detection of this scenario in the future and only apply port address parsing when the address is otherwise numeric.)

2.) NowSMS expects HTTP submissions to use the UTF-8 character set for encoding. The "charset=" parameter can be used to specify a different input character set, however, currently this character set is only applied to the text of the message, not the sender address. The sender address must be encoded using UTF-8 when submitting via HTTP.

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 9
Registered: 12-2012
Posted on Friday, March 08, 2013 - 03:26 pm:   

Hi Des,

Thank you for that information.

I would like to confirm that all the characters in the alphanumeric sender ID's are now being encoded to the GSM character set. However, despite following your suggestions to encode "colon", I still don't see them being encoded correctly.

I am now facing a similar issue with Roman8 and this also needs to have the sender address change applied that you have done earlier for GSM/IA5. Would you be able to apply a change to NowSMS so that a flag could be set that changes the entire SMPP packet with all its contents to the character set that has been applied in the SMSC configuration?

Looking forward to your response.

Thanks,

Haree
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 8052
Registered: 10-2002
Posted on Friday, March 08, 2013 - 07:35 pm:   

Hi Haree,

OK ... that makes sense. Those characters that you have listed have different encodings in the Roman8 character set.

We are only performing the alpha sender character set conversions for ASCII/iso-8859-1 or GSM/IA5.

An updated SMSSMPP.DLL is at the same download link as before:

http://www.nowsms.com/download/smppharee.zip

(If the file timestamp of the DLL is before March 8, clear your cache and try the download again.)

The other issues that Des mentioned still apply.

-bn
Hareeharan Kandavel
New member
Username: Haree

Post Number: 10
Registered: 12-2012
Posted on Friday, March 22, 2013 - 10:49 am:   

Hi Bryce,

I downloaded the dll from the link and still, I am unable to see Roman 8 conversion in the sender Id.

Could you please confirm whether the attached download link is for the Roman 8 character set conversion in sender Id?

Thanks,

Haree
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4331
Registered: 08-2008
Posted on Friday, March 22, 2013 - 03:42 pm:   

Hi Haree,

If you look at an SMPP trace, do you see the expected encoding?

As a test, I focused on the first character in your list:

LATIN SMALL LETTER I WITH GRAVE ì

From the NowSMS web form, I submitted a message with a sender address of ìtestì

If the SMPP connection is configured to use the GSM character set, I see 0x07 test 0x07 in the sender address, which is the correct GSM encoding.

If the SMPP connection is configured to use the Roman-8 character set, I see 0xD9 test 0xD9 in the sender address, which I believe to be the correct Roman-8 encoding.

If you do not see that same encoding, verify that the file date stamp on smssmpp.dll is from March 8.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4352
Registered: 08-2008
Posted on Monday, April 01, 2013 - 01:37 pm:   

Hi Haree,

Just a follow-up. In case the problems you were experiencing are related to difficulties applying the single DLL update, we have posted a complete updated install at http://www.nowsms.com/download/nowsms20130401.zip.

This update does include the change to encode alphanumeric senders according to the character set configured for the connection (including support for Roman-8).

--
Des
NowSMS Support
Hareeharan Kandavel
New member
Username: Haree

Post Number: 11
Registered: 12-2012
Posted on Friday, April 05, 2013 - 09:04 am:   

Hi Des,

Apologies for the delay in responding back. It now encodes the characters appropriately. Thank you very much for your help.

Thanks,

Haree