Roman-8 inbound support

Roman-8 inbound support SearchSearch
Author Message
Rik Clews
New member
Username: Rikclews

Post Number: 14
Registered: 08-2006
Posted on Monday, March 07, 2011 - 07:40 pm:   

Hi,

We recently installed v2011.02.16 and have noticed that some of our inbound traffic isn't being encoded correctly.

We have 2 inbound sources one "standard" which uses shortcodes and has no issues but the other is the bulk of our inbound traffic (long numbers) and uses Roman-8 encoding. This one doesn't show the extended GSM character set, it appears to ignore the "escape" character and give that as a question mark then use the next character as the gsm character. For example 1B65 = € (Euro) symbol, we see ?e, where e = 65.

This SMSC is O2 UK so not a small company, I can't help but feel we may have an issue with our INI settings can you confirm these:

[SMPP - xxxx]
SMPPVersion=v3.4
UserName=xxxx
Password=xxxx
SenderAddressOverride=Yes
Receive=Yes
ReceiveMMS=No
UseSSL=No
LongSMSAlt=Yes
RouteName=SMPP-O2-RECEIVE
Charset=Roman-8
SourceTON=5
SourceNPI=0
DestTON=1
DestNPI=1
SMSCCharset=Roman8
RemapMessageClassDCS=No
SMSCSendLimit=60/1

We note the outbound traffic is showing correctly, just inbound is affected.

Cheers,

Rik
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2977
Registered: 08-2008
Posted on Monday, March 07, 2011 - 10:31 pm:   

Hi Rik,

Are you certain that this is related to the NowSMS update?

I can't see any change that would impact Roman8 character decoding. We did add iso-8859-15 support late last year, but we can't see any impact from that change.

I'm suspecting it may just be coincidental, and something changed on the provider end.

That said ... I did just notice something in your INI file.

It should be Charset=Roman8 instead of Charset=Roman-8.

Hopefully that is the issue!

If not, try adding SMSCCharsetReceiveTextOverride=Yes to that section of the INI file. If the data_coding value is non-zero, NowSMS ignores the "Charset" setting unless this parameter is present. (It may seem illogical, but it is a long story.)

If you're still stuck, we need to capture a raw SMPP packet, either from SMPPDEBUG.LOG or with Wireshark, so that we can see the raw encoding particulars.

--
Des
NowSMS Support
Rik Clews
New member
Username: Rikclews

Post Number: 15
Registered: 08-2006
Posted on Tuesday, March 08, 2011 - 10:15 am:   

I've changed the roman-8 and added the override but its still the same.

Here's the relevant section of the SMPP debug showing the inbound of a message I sent with content "123 €€}]" which is showing as "123 ?e?e?)?>"

09:54:57:898 (000003DC) 193.132.40.160 --: WaitForResponseSMPP: SMPP Debug: received a deliver message
09:54:57:898 (000003DC) 193.132.40.160 <-: 69 byte packet
09:54:57:898 (000003DC) 193.132.40.160 <-: 00 00 00 45 00 00 00 05 00 00 00 00 00 00 00 03 E
09:54:57:898 (000003DC) 193.132.40.160 <-: 00 01 01 34 34 37 35 35 32 35 39 38 31 35 37 00 447552598157
09:54:57:898 (000003DC) 193.132.40.160 <-: 01 01 34 34 37 38 36 30 30 30 33 30 30 32 00 00 447860003002
09:54:57:898 (000003DC) 193.132.40.160 <-: 00 00 00 00 00 00 00 00 0C 31 32 33 20 8D 65 8D 123 e
09:54:57:898 (000003DC) 193.132.40.160 <-: 65 8D 29 8D 3E e ) >
09:54:57:898 (000003DC) 193.132.40.160 ->: 17 byte packet
09:54:57:898 (000003DC) 193.132.40.160 ->: 00 00 00 11 80 00 00 05 00 00 00 00 00 00 00 03
09:54:57:898 (000003DC) 193.132.40.160 ->: 00
Hope it makes sense!

Cheers,

Rik
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2980
Registered: 08-2008
Posted on Tuesday, March 08, 2011 - 01:57 pm:   

Hi Rik,

If this ever worked, something has definitely changed on the operator side. Instead of prefixing with the extended GSM character with the escape (0x1B) character, they appear to be using 0x8D.

I take it you are able to send these characters ok, right?

It is a simple change for us to remap the 0x8D to 0x1B. We're just not sure why the escape character would be encoded like this.

Rather than a full update, I'm posting an update for just the SMSSMPP.DLL file to perform this conversion. You'll need to stop the NowSMS service and manually copy this file into the Program Files\NowSMS directory to apply the update. Here's the link: http://www.nowsms.com/download/smssmpp4rik.zip.

--
Des
NowSMS Support
Rik Clews
New member
Username: Rikclews

Post Number: 16
Registered: 08-2006
Posted on Tuesday, March 08, 2011 - 03:25 pm:   

Thanks for the prompt reply - it does strike me as being rooted at the SMSC, I have raised it with them. We don't have issues sending these characters through them so it is just inbound. Checking through our records it seems to have been like this for some time, including when running the previous version of NowSMS (v2010.11.04), so I think unrelated to the upgrade.

I assume this dll is a one off and if used, any subsequent version updates will revert back to expecting the correct encoding?

Still, very useful for a stop gap until I get to the bottom of the issue with the SMSC

Thanks,

Rik
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2983
Registered: 08-2008
Posted on Tuesday, March 08, 2011 - 03:43 pm:   

Hi Rik,

If you say it corrects the problem, we'll keep the change going forward.

Currently we have 0x8D defined in our Roman-8 table as an unknown/unused character, so it was getting converted to "?".

As Roman-8 is so infrequently used, we don't see a problem converting 0x8D to 0x1B for inbound SMS over a connection using this character set. As such, we're willing to carry this change forward to future updates.


--
Des
NowSMS Support
Rik Clews
New member
Username: Rikclews

Post Number: 17
Registered: 08-2006
Posted on Friday, March 25, 2011 - 05:04 pm:   

Hi Des,

Sorry it's been a while but (allegedly) we now have confirmation that the Roman-8 escape character can be either 0x8D or 0x1B. Quite how we're supposed to know which one and when remains a mystery though.

So if you could carry this change forward to future updates and confirm such to us we'd be most grateful.

Thanks,

Rik
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3053
Registered: 08-2008
Posted on Friday, March 25, 2011 - 11:36 pm:   

Hi Rik,

Thanks for the followup.

Although it is not mentioned in the change notes for an update that was just posted (http://www.nowsms.com/nowsms-update-2011-03-21), this change has been carried forward and will continue to be present in future updates.

--
Des
NowSMS Support