EMAIL to SMS with gsm 7bit encoding | Search |
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through December 14, 2010 ⬆ |
◄ ► |
Author | Message | ||||
dinos New member Username: Dinaras Post Number: 23 Registered: 02-2006 |
Hello, what evere message i send using email to sms is always converted to utf-8 even if there are only 7bit characters in the message. I am trying to send email to sms with greek capital characters. Is there any way nowsms will try to send message as 7bit before converting it to unicode Thanks in advance Best regards Konstantinos Liadakis | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2619 Registered: 08-2008 |
Hi Konstantinos, Can you give me an example message? I think we have some logic in the SMS gateway which translates some Greek capital letters from their Unicode character to a 7-bit English alternative. For example, Α (Greek Alpha) looks the same as A, so we translate it for SMS so that 7-bit encoding can be used. -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 25 Registered: 02-2006 |
Hello, i am attaching part of the log file of one message. In this message all characters are greek capital letters which are included in gsm 7bit. In the log you will also see: Content-Type: text/plain; charset=utf-8 we did try to use charset iso-8859-7, no luck. Thank you in advance Konstantinos Liadakis
| ||||
dinos New member Username: Dinaras Post Number: 26 Registered: 02-2006 |
> > > > Hello, i am attaching part of the log file of one message. > > In this message all characters are greek capital letters which are > included in gsm 7bit. > > In the log you will also see: > Content-Type: text/plain; charset=utf-8 > > we did try to use charset iso-8859-7, no luck. > > Thank you in advance > > Konstantinos Liadakis > > > > > > | ||||
dinos New member Username: Dinaras Post Number: 27 Registered: 02-2006 |
Sorry for multiple posts, but had a problem loging in and tried to reply by email. Anyway i also forgot to mention the most important, my nowsms version Nowsms v2010.02.09 Thanks again Dinos | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2628 Registered: 08-2008 |
Hi Dinos, The update at the following link will address the problem: http://www.nowsms.com/download/nowsms20101104.zip I just tested it with the content in your log file, and it appears to be encoded correctly when it goes out via SMS. -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 28 Registered: 02-2006 |
I did try the above update but i am unable to send any greek at all now. I tried to encode sms as utf-8 iso8859-7 and all greek letters come as ??? Any ideas? | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2706 Registered: 08-2008 |
I just tested again with the example that you posted earlier (utf-8), and it seemed to work. This is the text that was received: someone@somewhere.com /ANAMONH AΠOΔOXHΣ KOΣTOYΣ /ANAMENOYME AΠOΔOXH KOΣTOYΣ ΓIA TO ΣYΣTHMA ME APIΘMO SIH-02158 -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 29 Registered: 02-2006 |
Hello Des and thank you for your quick reply I am attaching part of a message sent and failed. Message body is grrek alphabet and i get it in my mobile with ?????. In fact i get ΑΒ??ΕΖΗ?...... all different chars from latin (ΔΨΩΘΛΞ e.t.c) I am pasting a part from my log file 23:35:13:031 [12] ThreadProcessConnection: Thread started 23:35:13:031 [12] ThreadProcessConnectionSMTP: Processing SMTP connection from 85.196.25.2... 23:35:13:031 [12] SendCommand: 220 SMTP Ready 23:35:13:031 [12] ThreadProcessConnectionSMTP: 220 SMTP Ready 23:35:13:171 [12] ThreadProcessConnectionSMTP: EHLO mail.ans.gr 23:35:13:171 [12] SendCommand: 250-Ok 250-AUTH LOGIN 250 HELP 23:35:13:171 [12] ThreadProcessConnectionSMTP: 250-Ok 250-AUTH LOGIN 250 HELP 23:35:13:296 [12] ThreadProcessConnectionSMTP: MAIL FROM:<user@mydomain.com> 23:35:13:296 [12] SendCommand: 250 Ok 23:35:13:296 [12] ThreadProcessConnectionSMTP: 250 Ok 23:35:13:437 [12] ThreadProcessConnectionSMTP: RCPT TO:<lentis@email2sms.mydomain.com> 23:35:13:453 [12] GetMmsAccountingKeepAlive: Closing keep-alive socket 23:35:13:453 [12] GetMmsAccountingKeepAlive: Closing keep-alive socket 23:35:13:453 [12] RetrieveURL: Retrieving http://services.mydomain.com/nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMS EMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1 23:35:13:453 [12] RetrieveURL: Looking up services.mydomain.com 23:35:13:578 [12] RetrieveURL: Retrieving nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1 23:35:13:578 [12] RetrieveURL: GET /nowsms/accounting_callback.asp?PreAuth=Yes&Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&MsgCount=1 HTTP/1.1 User-Agent: Now SMS/MMS Gateway v2010.11.04 Accept: */* Host: services.mydomain.com 23:35:13:781 [12] HttpResponseWait: Ok 23:35:13:781 [12] RetrieveURL: Saving keep-alive socket 23:35:13:781 [12] RetrieveURL: got success response 23:35:13:781 [12] RetrieveURL: text/html 23:35:13:781 [12] SendCommand: 250 Ok 23:35:13:781 [12] ThreadProcessConnectionSMTP: 250 Ok 23:35:13:921 [12] ThreadProcessConnectionSMTP: DATA 23:35:13:921 [12] SendCommand: 354 Ok, end with "." on a new line... 23:35:13:921 [12] ThreadProcessConnectionSMTP: 354 Ok, end with "." on a new line... 23:35:14:312 [12] ThreadProcessConnectionSMTP: Message received 1306 bytes ... processing ... 23:35:14:312 [12] SendCommand: 250 Message Accepted 23:35:14:312 [12] ThreadProcessConnectionSMTP: Received: (qmail 9859 invoked by uid 1012); 8 Dec 2010 23:35:14 +0200 Received: from 62.169.201.194 by mail (envelope-from <user@mydomain.com>, uid 1008) with qmail-scanner-2.06st (clamdscan: 0.96.3/12077. spamassassin: 3.2.4. perlscan: 2.06st. Clear:RC:1(62.169.201.194):. Processed in 0.129817 secs); 08 Dec 2010 21:35:14 -0000 Received: from unknown (HELO ?10.0.0.192?) (62.169.201.194) by mail.ans.gr with SMTP; 8 Dec 2010 23:35:13 +0200 Message-ID: <4CFFFA10.6030000@mydomain.com> Date: Wed, 08 Dec 2010 23:35:12 +0200 From: Konstantinos Liadakis <user@mydomain.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: lentis@email2sms.mydomain.com Subject: test Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ 23:35:14:312 [12] ThreadProcessConnectionSMTP: 250 Message Accepted 23:35:14:453 [12] ThreadProcessConnectionSMTP: QUIT 23:35:14:453 [12] SendCommand: 221 Goodbye 23:35:14:453 [12] ThreadProcessConnectionSMTP: 221 Goodbye 23:35:14:453 [12] ThreadProcessConnectionSMTP: Closing SMTP Connection 23:35:14:453 [12] ThreadProcessConnection: Thread ended 23:35:15:265 [9] ThreadRouteSMTPIn: E421D45B.RFC 23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-RCPT-TO: SMS:30XXXXXXXXXX 23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-Received-From: 85.196.25.2 23:35:15:265 [9] ThreadRouteSMTPIn: X-MMSC-ENV-MAIL-FROM: <user@mydomain.com> 23:35:15:265 [9] ThreadRouteSMTPIn: From: Konstantinos Liadakis <user@mydomain.com> 23:35:15:265 [9] ThreadRouteSMTPIn: To: lentis@email2sms.mydomain.com 23:35:15:265 [9] ThreadRouteSMTPIn: Subject: test 23:35:15:265 [9] ThreadRouteSMTPIn: Content-Type: text/plain; charset=UTF-8; format=flowed 23:35:15:265 [9] ThreadRouteSMTPIn: charset= 23:35:15:265 [9] ThreadRouteSMTPIn: UTF-8 23:35:15:281 [9] GetMmsAccountingKeepAlive: Re-using keep-alive socket 23:35:15:281 [9] RetrieveURL: Retrieving http://services.mydomain.com/nowsms/accounting_callback.asp?Type=MMSEMail&From=u ser@mydomain.com&To=30XXXXXXXXXX&Size=228 23:35:15:281 [9] RetrieveURL: Using keep-alive socket 23:35:15:281 [9] RetrieveURL: Retrieving nowsms/accounting_callback.asp?Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&Size=228 23:35:15:281 [9] RetrieveURL: GET /nowsms/accounting_callback.asp?Type=MMSEMail&From=user@mydomain.com&To=30XXXXXXXXXX&Size=228 HTTP/1.1 User-Agent: Now SMS/MMS Gateway v2010.11.04 Accept: */* Host: services.mydomain.com 23:35:16:031 [9] HttpResponseWait: Ok 23:35:16:031 [9] RetrieveURL: Saving keep-alive socket 23:35:16:031 [9] RetrieveURL: got success response 23:35:16:031 [9] RetrieveURL: text/html | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2708 Registered: 08-2008 |
Hi Dinos, Is that an actual message being sent by an e-mail client, or are you perhaps doing some cut & paste in a telnet client to test? The reason I ask is because I initially tried simulating this with Telnet, but the characters got all scrambled, because Telnet does not accept paste in UTF-8 format. When I did a wireshark trace, I realized that this was the problem. So I wrote a simple program to post the SMTP message and ensure that the message data was actually being sent with UTF-8 and it worked properly ... the Greek characters were sent and received as expected. One other clue ... the MMSCDEBUG.LOG will show garbage characters in the message if the message is actually encoded in UTF-8. That is because the LOG file does not have a UTF-8 file signature. So in my working transmission, the resulting MMSCDEBUG.LOG shows garbage for the text, but the actual Greek characters do get encoded correctly. If this doesn't fit what is happening to you, then I'd suggest doing a Wireshark trace on the MMSC so that we can see the raw binary data that is coming in, and verify that the text is actually encoded as UTF-8 like the mail headers say that it is. -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 30 Registered: 02-2006 |
this is an actual copy/paste from nowsms mmsc debug log. I used thunderbird to send email and manually set encoding to utf-8. I also tried iso-8859-7 but again i received same message. All 7bit characters that are different from latin are scrambled | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2710 Registered: 08-2008 |
Ok. I do not understand why this is working without any problem for me. Or why my MMSCDEBUG.LOG displays garbage for these characters when they are correctly encoded with UTF-8 But sends the message correctly, whereas you see the actual correct characters in MMSCDEBUG.LOG but not in the SMS. I think we need to see a binary packet level trace, with WireShark or similar, in order to understand what encoding is actually being used. The symptoms that you are experiencing suggest that the email client in question is encoding the text in iso-8859-7, but specifying UTF-8 in the header. Normally the MMSCDEBUG.LOG is formatted to display in your local character set. The UTF-8 text is not converted when written to this debug log. So I would expect the message to be corrupt in MMSCDEBUG.LOG, but it is not. That's why I think we need a packet trace to see what is really going over the network. There's probably some other factor that I am missing and this trace will tell me what is different so that we can recreate the problem. -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 31 Registered: 02-2006 |
please give me your email address so that i can send you capture file. Thank you Dinos | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2713 Registered: 08-2008 |
Send to nowsms@nowsms.com and put Attention: Des in the subject line. | ||||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2714 Registered: 08-2008 |
Hi Dinos, Thanks for sending the log, that clarifies things considerably. Are you able to successfully send Greek text messages via this particular SMPP connection if the originator is not SMTP? From what I can see in the log, the UTF-8 message content is being decoded correctly. The log file also shows NowSMS submitting the message via SMPP with the Greek characters encoded using the GSM character set. I would expect that you would see similar problems with these characters, on this SMPP connection, if you submitted the message via an HTTP interface. I don't think this is an SMTP specific issue, even if it may have been originally. (Originally we were requiring a message containing these characters to go out in Unicode format.) Try changing the SMSC Character Set to default instead of GSM. If that doesn't resolve the problem, then you may have to contact your provider to ask how Greek characters are supposed to be encoded. NowSMS is encoding them according to the GSM character set (http://www.nowsms.com/long-sms-text-messages-and-the-160-character-limit) ... example Δ is 0x10. When you specify SMSC character set = GSM, NowSMS uses a data_coding value of 1. If you change this to default, NowSMS still uses the GSM character set, but uses a data_coding value of 0, which may be more to your provider's liking. Unfortunately, these Greek characters are not part of iso-8859-1, so I would not expect that setting to work. Try submitting messages from the web interface and troubleshoot the Greek characters from there, then come back to SMTP. -- Des NowSMS Support | ||||
dinos New member Username: Dinaras Post Number: 32 Registered: 02-2006 |
Hello Des and thank you for the detective work Everything works right now. I originally had this setting as default for this smsc link, but i was facing problems with @ (message was cut after @ character when sending messages through mblox and they suggested to turn it to gsm, so i did. the reason was because @ was not correctly encoded like they said. Before there was nothing affected from the change, or nothing that i realized off Thankfully all works fine now Thank you very much for your help, and for your fast responses to my problem Regards Dinos |