Confused about message encodings and "data_coding" in NowSMS Could ...

Confused about message encodings and "data_coding" in NowSMS Could ... SearchSearch
Author Message
Yury S
New member
Username: Yskanavy

Post Number: 3
Registered: 12-2018
Posted on Wednesday, January 22, 2020 - 01:49 am:   

Hi

My SMPP client is sending out messages with "data_coding" set to 8, which according to the specification is UCS2 (ISO/IEC-10646), which is similar to UTF-16. I send the message containing national chars. Android SMS app on my phone shows sent and received messages correctly and with no data loss. I also see the correct national chars displayed in NowSMS log files (SMSIN-....LOG and SMSIN-....LOG).

However, when my SMPP listener receives the message, "data_coding" is set to 0 (SMSC Default Alphabet) and the data has been converted to something like Ascii (GSM7 perhaps), the national chars were converted to question marks and all other chars are basically single-byte ascii.

If my SMPP client sends a message with "data_coding" 0 and UTF-8 encoded data, then the entire message is seen as garbage on my phone and and SmsNow log files.

So, my understanding that to ensure a specific message coding, one needs to use non-zero "data_coding" to indicate how exactly the text is coded. Seems to hold true except for when such message is sent from NowSMS SMPP to a client.

On the other hand leaving "data_coding" as zero does not provide a good result at all...Not sure I understand how NowSMS SMPP operates and to receive messages correctly.

Thanks.
Yury S
New member
Username: Yskanavy

Post Number: 4
Registered: 12-2018
Posted on Wednesday, January 22, 2020 - 01:52 am:   

"If my SMPP client sends a message with "data_coding" 0 and UTF-8 encoded data"

I meant to say "UTF-16 encoded data"}
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 6087
Registered: 08-2008
Posted on Thursday, January 23, 2020 - 11:13 pm:   

Hi,

I'm sorry, but I don't understand the overall logic flow through which the message is processed.

I understand that you can send SMS OK using data_coding=8, which is expected.

And I understand that when you receive SMS from NowSMS as an SMPP client, the encoding is not what is expected.

But I don't understand the overall path/flow. What is nowSMS receiving the message from? Does the SMSIN log show the correct characters and DCS=8? If the SMSIN log shows the correct encoding...I don't see why or how NowSMS would change the data_coding to 0. Unless maybe you're doing some processing with a 2-way command?

What you describe does not sound normal...but basically I need to see more details ... relevant log entries and an explanation of the path/flow of processing for the received message.

--
Des
NowSMS Support
Yury S
New member
Username: Yskanavy

Post Number: 5
Registered: 12-2018
Posted on Thursday, January 23, 2020 - 11:36 pm:   

I am sending a message to my own number using my program which acts an SMPP client. I set "data_coding" to 8, and encode my message as UTF-16.

My program also receives messages through SMPP. When I get deliver_sm with the message I have send to myself, "data_coding" is 0 and the "short_message" has been converted to a single byte ascii (i,e. non-ascii data I have sent is damaged).

Both SMSIN and SMSOUT show the correct data with all non-ascii chars present, it's just the deliver_sm I get looks incorrect.
Yury S
New member
Username: Yskanavy

Post Number: 6
Registered: 12-2018
Posted on Friday, January 24, 2020 - 01:45 am:   

For example, I am sending this message text:

русский english

This is the logging from my code:
Send message via "short_message", data coding => 8 byte len => 32 bytes => FE FF 04 40 04 43 04 41 04 41 04 3A 04 38 04 39 00 20 00 65 00 6E 00 67 00 6C 00 69 00 73 00 68

Receive "short_message" message: data coding => 0 byte len => 15 byte data => 3F 3F 3F 3F 3F 3F 3F 20 65 6E 67 6C 69 73 68

The byte sequence I have sent out is a correct UTF-16 representation of my text.
The received data is Ascii data for

??????? english

i.e. the russian word at the beginning is now question marks.

SMSOUT:

*** SubmitUser=**** ;SMSCRouteName=***;Sender=***;DCS=8;Text="русский english"
2020-01-23 18:21:08:790,5E27478D,,**** ,OK -- LocalUser:****,SubmitUser=NowSMSModem - ****;Sender=***;Text="русский english"

SMSIN:
2020-01-23 18:21:08:789,****,NowSMSModem - **** e,Text="русский english";SMSCRouteName=****;Recip=****

(I replaced sensitive info with *)

I see the same issue when sending the same text from my phone or through NowSMS Send Message page.

My Android SMS App also displays the sent and received messages properly.

To sum up, the messages are seen correctly in the logs and phone app, but the deliver_sm sent to my program has data re-coded to ascii and data_coding reset to 0
Yury S
New member
Username: Yskanavy

Post Number: 7
Registered: 12-2018
Posted on Friday, January 24, 2020 - 01:51 am:   

I also tried my program with SMPPSim with LOOPBACK=TRUE, so it would echo the message back.

Got the expected result:

Receive "short_message" message: data coding => 8 byte len => 32 byte data => FE FF 04 40 04 43 04 41 04 41 04 3A 04 38 04 39 00 20 00 65 00 6E 00 67 00 6C 00 69 00 73 00 68
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 6089
Registered: 08-2008
Posted on Friday, January 31, 2020 - 04:33 pm:   

Hi Yury,

Sorry for the delay in response. We were able to recreate this problem. Unicode text messages were not being encoded correctly when routed from the Android modem interface to an SMPP client.

An updated build of NowSMS fixes this problem: https://www.nowsms.com/download/nowsms20200131.zip

I notice that you are have posted in the support area for the cloud edition...we have updated the cloud servers to address this problem.


Des
NowSMS Support
Yury S
New member
Username: Yskanavy

Post Number: 8
Registered: 12-2018
Posted on Friday, January 31, 2020 - 05:23 pm:   

Thank you, I will try it out.

Add Your Message Here, or click here to start a new topic.
Post:
Bold text Italics Underline Create a hyperlink Insert a clipart image
Options: Automatically activate URLs in message
Action: