EMAIL to SMS with gsm 7bit encoding

EMAIL to SMS with gsm 7bit encoding SearchSearch
Author Message
dinos
New member
Username: Dinaras

Post Number: 23
Registered: 02-2006
Posted on Thursday, November 04, 2010 - 09:39 am:   

Hello, what evere message i send using email to sms is always converted to utf-8 even if there are only 7bit characters in the message.

I am trying to send email to sms with greek capital characters.

Is there any way nowsms will try to send message as 7bit before converting it to unicode

Thanks in advance

Best regards

Konstantinos Liadakis
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2619
Registered: 08-2008
Posted on Thursday, November 04, 2010 - 06:52 pm:   

Hi Konstantinos,

Can you give me an example message?

I think we have some logic in the SMS gateway which translates some Greek capital letters from their Unicode character to a 7-bit English alternative.

For example, Α (Greek Alpha) looks the same as A, so we translate it for SMS so that 7-bit encoding can be used.

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 25
Registered: 02-2006
Posted on Friday, November 05, 2010 - 10:07 am:   

Hello, i am attaching part of the log file of one message.

In this message all characters are greek capital letters which are included in gsm 7bit.

In the log you will also see:
Content-Type: text/plain; charset=utf-8

we did try to use charset iso-8859-7, no luck.

Thank you in advance

Konstantinos Liadakis


text/plainmmscdebug log
emailtosms_mmscdebug.txt (10.3 k)
dinos
New member
Username: Dinaras

Post Number: 26
Registered: 02-2006
Posted on Friday, November 05, 2010 - 10:10 am:   

> > > > Hello, i am attaching part of the log file of one message. > > In this message all characters are greek capital letters which are > included in gsm 7bit. > > In the log you will also see: > Content-Type: text/plain; charset=utf-8 > > we did try to use charset iso-8859-7, no luck. > > Thank you in advance > > Konstantinos Liadakis > > > > > >
dinos
New member
Username: Dinaras

Post Number: 27
Registered: 02-2006
Posted on Friday, November 05, 2010 - 10:45 am:   

Sorry for multiple posts, but had a problem loging in and tried to reply by email.

Anyway i also forgot to mention the most important, my nowsms version
Nowsms v2010.02.09

Thanks again

Dinos
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2628
Registered: 08-2008
Posted on Friday, November 05, 2010 - 02:05 pm:   

Hi Dinos,

The update at the following link will address the problem:

http://www.nowsms.com/download/nowsms20101104.zip

I just tested it with the content in your log file, and it appears to be encoded correctly when it goes out via SMS.

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 28
Registered: 02-2006
Posted on Wednesday, December 08, 2010 - 06:08 pm:   

I did try the above update but i am unable to send any greek at all now. I tried to encode sms as utf-8 iso8859-7 and all greek letters come as ???

Any ideas?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2706
Registered: 08-2008
Posted on Wednesday, December 08, 2010 - 06:33 pm:   

I just tested again with the example that you posted earlier (utf-8), and it seemed to work.

This is the text that was received:

someone@somewhere.com /ANAMONH AΠOΔOXHΣ KOΣTOYΣ /ANAMENOYME AΠOΔOXH KOΣTOYΣ ΓIA TO ΣYΣTHMA ME APIΘMO SIH-02158

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 29
Registered: 02-2006
Posted on Wednesday, December 08, 2010 - 09:43 pm:   

Hello Des and thank you for your quick reply

I am attaching part of a message sent and failed.

Message body is grrek alphabet and i get it in my mobile with ?????.

In fact i get ΑΒ??ΕΖΗ?...... all different chars from latin (ΔΨΩΘΛΞ e.t.c)

I am pasting a part from my log file


23:35:13:031 [12] ThreadProcessConnection: Thread started
23:35:13:031 [12] ThreadProcessConnectionSMTP: Processing SMTP connection from 85.196.25.2...
23:35:13:031 [12] SendCommand: 220 SMTP Ready
23:35:13:031 [12] ThreadProcessConnectionSMTP: 220 SMTP Ready
23:35:13:171 [12] ThreadProcessConnectionSMTP: EHLO mail.ans.gr
23:35:13:171 [12] SendCommand: 250-Ok
250-AUTH LOGIN
250 HELP
23:35:13:171 [12] ThreadProcessConnectionSMTP: 250-Ok
250-AUTH LOGIN
250 HELP
23:35:13:296 [12] ThreadProcessConnectionSMTP: MAIL FROM:<user@mydomain.com>
23:35:13:296 [12] SendCommand: 250 Ok
23:35:13:296 [12] ThreadProcessConnectionSMTP: 250 Ok
23:35:13:437 [12] ThreadProcessConnectionSMTP: RCPT TO:<lentis@email2sms.mydomain.com>
23:35:13:453 [12] GetMmsAccountingKeepAlive: Closing keep-alive socket
23:35:13:453 [12] GetMmsAccountingKeepAlive: Closing keep-alive socket
23:35:13:453 [12] RetrieveURL: Retrieving http://services.mydomain.com/nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMS EMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1
23:35:13:453 [12] RetrieveURL: Looking up services.mydomain.com
23:35:13:578 [12] RetrieveURL: Retrieving nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1
23:35:13:578 [12] RetrieveURL: GET /nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1 HTTP/1.1
User-Agent: Now SMS/MMS Gateway v2010.11.04
Accept: */*
Host: services.mydomain.com


23:35:13:781 [12] HttpResponseWait: Ok
23:35:13:781 [12] RetrieveURL: Saving keep-alive socket
23:35:13:781 [12] RetrieveURL: got success response
23:35:13:781 [12] RetrieveURL: text/html
23:35:13:781 [12] SendCommand: 250 Ok
23:35:13:781 [12] ThreadProcessConnectionSMTP: 250 Ok
23:35:13:921 [12] ThreadProcessConnectionSMTP: DATA
23:35:13:921 [12] SendCommand: 354 Ok, end with "." on a new line...
23:35:13:921 [12] ThreadProcessConnectionSMTP: 354 Ok, end with "." on a new line...
23:35:14:312 [12] ThreadProcessConnectionSMTP: Message received 1306 bytes ... processing ...
23:35:14:312 [12] SendCommand: 250 Message Accepted
23:35:14:312 [12] ThreadProcessConnectionSMTP: Received: (qmail 9859 invoked by uid 1012); 8 Dec 2010 23:35:14 +0200
Received: from 62.169.201.194 by mail (envelope-from <user@mydomain.com>, uid 1008) with qmail-scanner-2.06st
(clamdscan: 0.96.3/12077. spamassassin: 3.2.4. perlscan: 2.06st.
Clear:RC:1(62.169.201.194):.
Processed in 0.129817 secs); 08 Dec 2010 21:35:14 -0000
Received: from unknown (HELO ?10.0.0.192?) (62.169.201.194)
by mail.ans.gr with SMTP; 8 Dec 2010 23:35:13 +0200
Message-ID: <4CFFFA10.6030000@mydomain.com>
Date: Wed, 08 Dec 2010 23:35:12 +0200
From: Konstantinos Liadakis <user@mydomain.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: lentis@email2sms.mydomain.com
Subject: test
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ
ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ
ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ
ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ
23:35:14:312 [12] ThreadProcessConnectionSMTP: 250 Message Accepted
23:35:14:453 [12] ThreadProcessConnectionSMTP: QUIT
23:35:14:453 [12] SendCommand: 221 Goodbye
23:35:14:453 [12] ThreadProcessConnectionSMTP: 221 Goodbye
23:35:14:453 [12] ThreadProcessConnectionSMTP: Closing SMTP Connection
23:35:14:453 [12] ThreadProcessConnection: Thread ended
23:35:15:265 [9] ThreadRouteSMTPIn: E421D45B.RFC
23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-RCPT-TO: SMS:30XXXXXXXXXX
23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-Received-From: 85.196.25.2
23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-ENV-MAIL-FROM: <user@mydomain.com>
23:35:15:265 [9] ThreadRouteSMTPIn: From: Konstantinos Liadakis <user@mydomain.com>
23:35:15:265 [9] ThreadRouteSMTPIn: To: lentis@email2sms.mydomain.com
23:35:15:265 [9] ThreadRouteSMTPIn: Subject: test
23:35:15:265 [9] ThreadRouteSMTPIn: Content-Type: text/plain; charset=UTF-8; format=flowed
23:35:15:265 [9] ThreadRouteSMTPIn: charset=
23:35:15:265 [9] ThreadRouteSMTPIn: UTF-8
23:35:15:281 [9] GetMmsAccountingKeepAlive: Re-using keep-alive socket
23:35:15:281 [9] RetrieveURL: Retrieving http://services.mydomain.com/nowsms/accounting_callback.asp?Type=MMSEMail&From=u ser@mydomain.com&To=30XXXXXXXXXX&Size=228
23:35:15:281 [9] RetrieveURL: Using keep-alive socket
23:35:15:281 [9] RetrieveURL: Retrieving nowsms/accounting_callback.asp?Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&Size=228
23:35:15:281 [9] RetrieveURL: GET /nowsms/accounting_callback.asp?Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&Size=228 HTTP/1.1
User-Agent: Now SMS/MMS Gateway v2010.11.04
Accept: */*
Host: services.mydomain.com


23:35:16:031 [9] HttpResponseWait: Ok
23:35:16:031 [9] RetrieveURL: Saving keep-alive socket
23:35:16:031 [9] RetrieveURL: got success response
23:35:16:031 [9] RetrieveURL: text/html
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2708
Registered: 08-2008
Posted on Wednesday, December 08, 2010 - 10:58 pm:   

Hi Dinos,

Is that an actual message being sent by an e-mail client, or are you perhaps doing some cut & paste in a telnet client to test?

The reason I ask is because I initially tried simulating this with Telnet, but the characters got all scrambled, because Telnet does not accept paste in UTF-8 format. When I did a wireshark trace, I realized that this was the problem.

So I wrote a simple program to post the SMTP message and ensure that the message data was actually being sent with UTF-8 and it worked properly ... the Greek characters were sent and received as expected.

One other clue ... the MMSCDEBUG.LOG will show garbage characters in the message if the message is actually encoded in UTF-8. That is because the LOG file does not have a UTF-8 file signature. So in my working transmission, the resulting MMSCDEBUG.LOG shows garbage for the text, but the actual Greek characters do get encoded correctly.

If this doesn't fit what is happening to you, then I'd suggest doing a Wireshark trace on the MMSC so that we can see the raw binary data that is coming in, and verify that the text is actually encoded as UTF-8 like the mail headers say that it is.

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 30
Registered: 02-2006
Posted on Thursday, December 09, 2010 - 12:06 am:   

this is an actual copy/paste from nowsms mmsc debug log.

I used thunderbird to send email and manually set encoding to utf-8. I also tried iso-8859-7 but again i received same message. All 7bit characters that are different from latin are scrambled
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2710
Registered: 08-2008
Posted on Thursday, December 09, 2010 - 12:24 am:   

Ok. I do not understand why this is working without any problem for me.

Or why my MMSCDEBUG.LOG displays garbage for these characters when they are correctly encoded with UTF-8 But sends the message correctly, whereas you see the actual correct characters in MMSCDEBUG.LOG but not in the SMS.

I think we need to see a binary packet level trace, with WireShark or similar, in order to understand what encoding is actually being used.

The symptoms that you are experiencing suggest that the email client in question is encoding the text in iso-8859-7, but specifying UTF-8 in the header. Normally the MMSCDEBUG.LOG is formatted to display in your local character set. The UTF-8 text is not converted when written to this debug log. So I would expect the message to be corrupt in MMSCDEBUG.LOG, but it is not.

That's why I think we need a packet trace to see what is really going over the network.

There's probably some other factor that I am missing and this trace will tell me what is different so that we can recreate the problem.

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 31
Registered: 02-2006
Posted on Thursday, December 09, 2010 - 09:27 am:   

please give me your email address so that i can send you capture file.

Thank you

Dinos
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2713
Registered: 08-2008
Posted on Thursday, December 09, 2010 - 09:40 am:   

Send to nowsms@nowsms.com and put Attention: Des in the subject line.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2714
Registered: 08-2008
Posted on Thursday, December 09, 2010 - 11:05 am:   

Hi Dinos,

Thanks for sending the log, that clarifies things considerably.

Are you able to successfully send Greek text messages via this particular SMPP connection if the originator is not SMTP?

From what I can see in the log, the UTF-8 message content is being decoded correctly. The log file also shows NowSMS submitting the message via SMPP with the Greek characters encoded using the GSM character set.

I would expect that you would see similar problems with these characters, on this SMPP connection, if you submitted the message via an HTTP interface. I don't think this is an SMTP specific issue, even if it may have been originally. (Originally we were requiring a message containing these characters to go out in Unicode format.)

Try changing the SMSC Character Set to default instead of GSM.

If that doesn't resolve the problem, then you may have to contact your provider to ask how Greek characters are supposed to be encoded.

NowSMS is encoding them according to the GSM character set (http://www.nowsms.com/long-sms-text-messages-and-the-160-character-limit) ... example Δ is 0x10.

When you specify SMSC character set = GSM, NowSMS uses a data_coding value of 1. If you change this to default, NowSMS still uses the GSM character set, but uses a data_coding value of 0, which may be more to your provider's liking.

Unfortunately, these Greek characters are not part of iso-8859-1, so I would not expect that setting to work.

Try submitting messages from the web interface and troubleshoot the Greek characters from there, then come back to SMTP.

--
Des
NowSMS Support
dinos
New member
Username: Dinaras

Post Number: 32
Registered: 02-2006
Posted on Thursday, December 09, 2010 - 11:21 am:   

Hello Des and thank you for the detective work

Everything works right now.

I originally had this setting as default for this smsc link, but i was facing problems with @ (message was cut after @ character when sending messages through mblox and they suggested to turn it to gsm, so i did. the reason was because @ was not correctly encoded like they said. Before there was nothing affected from the change, or nothing that i realized off

Thankfully all works fine now

Thank you very much for your help, and for your fast responses to my problem

Regards

Dinos