Wrong Phone number hang queue

Wrong Phone number hang queue SearchSearch
Author Message
Hala
New member
Username: Haref

Post Number: 4
Registered: 12-2015
Posted on Thursday, February 18, 2016 - 09:50 am:   

Hello,

We found suddenly queue is getting high, and SMS very slowly getting sent.

We found one of the SMS's sent has wrong number format,and when i move its req file from the queue folder , the sms's start to be flushed normally.


Sometimes we have sms's with wrong number and i find nowsms convert the req file to err, but in this case it didn't convert it to err and the sms stuck in the queue .
Kindly advice how can nowsms convert this request to err so that nowsms don't accumulate the messages.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5572
Registered: 08-2008
Posted on Thursday, February 18, 2016 - 04:15 pm:   

Hi Hala,

I do not understand how this could happen.

Unless the upstream provider is returning a throttling error (ESME_RTHROTTLED) instead of more appropriate error code. That could cause delays because NowSMS thinks the provider is telling us to slowdown.

Does the SMSOUT-yyyymmdd.LOG show this error? (If not, what error does it show for the problem messages?)

Assuming this is the issue, edit SMSGW.INI and under the [SMSGW] header add SMPPThrottleErrorDelay=0. This will disable any delays when this error is returned.

If you are running a version prior to 2015.06.24 (the version at http://www.nowsms.com/download-free-trial), I would suggest updating. The following changes related to ESME_RTHROTTLED were made in this version.

* SMPP: Change to ESME_RTHROTTLED error handling. Previous versions assumed that this error code indicated a system level issue at the upstream SMSC, not an error specific to a message recipient. Old default behavior was to delay submitting for 5 seconds when this error was encountered. SMPPThrottleErrorDelay=## in the [SMSGW] or [SMPP - server:port] section was used to adjust this delay. Previously, this delay was expressed in seconds, it is now expressed in milliseconds. SMPPThrottleErrorDelay=5000 would represent a 5 second delay, or a value of 0 indicates no delay. The default delay is now 5 milliseconds (thousandths of a second).

* SMPP: Change to ESME_RTHROTTLED error handling. Previous versions assumed that this error code indicated a system level issue at the upstream SMSC, not an error specific to a message recipient. Old default behavior was to attempt to resubmit the same message up to 5 times before moving on to the next message in the queue. This default has been changed to a single resubmit, with a configuration setting, SMPPThrottleQuickRetry=## in the [SMSGW] or [SMPP - server:port] section used to adjust the number of resubmit attempts before moving on to other queued messages.

* SMPP: Previous versions assumed that the ESME_RTHROTTLED error code indicated a system level issue at the upstream SMSC, not an error specific to a message recipient. Some SMSCs return this error if there are too many pending messages for the message recipient. NowSMS expects the ESME_RMSGQFUL error code to be used for this scenario. An existing setting has been enhanced to allow ESME_RTHROTTLED to be remapped to ESME_RMSGQFUL. To perform this remapping, edit SMSGW.INI, and under the [SMSGW] header (or [SMPP - server:port] header), add SMPPQFullErrorCodes=58 (0x58 is the numerical value for ESME_RTHROTTLED).


--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5573
Registered: 08-2008
Posted on Thursday, February 18, 2016 - 04:16 pm:   

P.S. - What version of NowSMS are you running?
Hala
New member
Username: Haref

Post Number: 5
Registered: 12-2015
Posted on Thursday, February 18, 2016 - 05:45 pm:   

We are running 2015.06.24

NO errors shown in the smsout log file.

We just noticed that the queue was very high, and we monitored the number of SMS sent on the nowsms console is not increasing in the normal pattern 2 messages/second.
Once I move the abnormal req file fro the queue folder , the other sms's start to be sent normally, and the queue flush, .
When I return this (.req) file again, the messages get stucked in queue and not sent.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5574
Registered: 08-2008
Posted on Thursday, February 18, 2016 - 09:58 pm:   

You renamed/deleted a .REQ file. Are you saying the message ID associated with that file appears nowhere in the SMSOUT-yyyymmdd.LOG?

I'm expecting it would appear somewhere with an error or retry pending. Otherwise, how did you know this message was a problem?

Do you have a sync mode enabled? What window size?

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 6
Registered: 12-2015
Posted on Friday, February 19, 2016 - 11:26 am:   

I moved the.req file from the Q folder.
The stucked message has no record in the log file or any retrials for it.
it was a trial and error troubleshooting, I move all req files from the queue folder and then started to put part by part , the messages are flushed normally from the queue, when it comes to this specific message, nothing is sent and all messages are held in the queue.

how can I check if the sync mode enabled or not?
What do you mean by windows size?
Windows server 2008 standard SP1
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5575
Registered: 08-2008
Posted on Friday, February 19, 2016 - 02:48 pm:   

Before I answer the questions:

What is your outbound SMSC connectivity? I assumed SMPP, but as I read back through these messages, I do not see anywhere that you have said this. So I may have made a wrong assumption.

Edit SMSGW.INI and under the [SMSGW] header add SMPPThrottleErrorDelay=0

I do not know that this will help any problem, but it will not hurt.

When the problem occurred, did SMSOUT-yyyymmdd.LOG show any messages being sent out? Any errors? What was the pattern of times for message sending before the problem occurred?

If you move the .req file back to the Q, does the problem return, or does the message get sent?


quote:

how can I check if the sync mode enabled or not?
What do you mean by windows size?




In NowSMS, when configuring an SMPP SMSC connection, this is specified under the “Advanced Settings” for the properties of the SMPP connection. Look for “Enable SMPP Async Mode (windowing)” and the “Window Size (packets)” parameter.

More details here: http://www.nowsms.com/smpp-async-mode

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 7
Registered: 12-2015
Posted on Monday, February 22, 2016 - 08:45 am:   

Yes, SMPP connection
I added SMPPThrottleErrorDelay=0 , but nothing changed.
NO errors in SMSOUT-yyyymmdd.LOG, also I enabled the debug but no erros shown too.

When I return back the .req file, the same behavior happened again.
The SMPP Async Mode is not enabled.
Should I enable it?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5576
Registered: 08-2008
Posted on Monday, February 22, 2016 - 02:35 pm:   


quote:

The SMPP Async Mode is not enabled.
Should I enable it?




I would try it, but wait until you generate some new debug logs for me first.

It sounds like the SMPP server is not responding when this message is submitted. In this scenario, NowSMS will wait up to 2 minutes for a response, and then would record an error in the SMSOUT-yyyymmdd.LOG.

If async mode is not enabled, no other messages will be transmitted during this time. If async mode is enabled, more messages will be sent 9if the server supports and allows.

Before you enable SMPP async mode...

I would like to see the REQ file, SMSDEBUG.LOG and SMPPDEBUG.LOG

To make it easier to see the problem in the logs, before you send these files, recreate the problem one more time...move the problem REQ file back to the Q again.

Wait 10 minutes.

Send the REQ file, SMSDEBUG.LOG and SMPPDEBUG.LOG to nowsms@nowsms.com with Attention: Des in the subject line. Also, please post a reply here, so that I know to look for the email.


--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5577
Registered: 08-2008
Posted on Monday, February 22, 2016 - 02:36 pm:   

P.S. - Are you generating this REQ file directly, or is NowSMS creating it?
Hala
New member
Username: Haref

Post Number: 8
Registered: 12-2015
Posted on Thursday, February 25, 2016 - 10:02 am:   

Hello,

I sent the mail.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5581
Registered: 08-2008
Posted on Thursday, February 25, 2016 - 03:42 pm:   

Thanks. I am able to recreate the problem and we are investigating.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5583
Registered: 08-2008
Posted on Friday, February 26, 2016 - 03:49 am:   

Hi Hala,

Actually I could not 100% recreate the problem, but introducing these message files did hang the outbound queue for one of my SMSC connections for about 10 minutes.

The problem appears to be caused by an SMPP client submitting a long message to a number that includes dash (-) characters. This causes some confusion as the system tries to find all of the message parts.

We have fixed this issue in the 2016.02.23 build at http://www.nowsms.com/download/nowsms20160223.zip

Note that this version has a redesigned web interface. If you have customized any web templates, they will be saved in an html.old folder. Those templates can be copied from html.old back to html if you wish to continue using the older templates.

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 9
Registered: 12-2015
Posted on Thursday, March 03, 2016 - 02:57 pm:   

Hi Des,

The new version is buggy.
The service is not installed and it corrupted the programdata folder.
When I installed it again without changing the folder path, still the service not opening the port 1804.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5592
Registered: 08-2008
Posted on Thursday, March 03, 2016 - 08:17 pm:   

Hi,

What do you mean by "it corrupted the program data folder"?

Enable the SMSDEBUG.LOG. Is the NowSMS service constantly restarting? (Task manager shows smsgws.exe constantly starting and stopping)

There is a newer update at http://www.nowsms.com/download/nowsms20160301.zip

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 10
Registered: 12-2015
Posted on Sunday, March 06, 2016 - 08:35 am:   

Dear Des,

"it corrupted the program data folder": after the installation , the folder suddenly changed from hidden to be shown normal folder.
Some of the windows files were missed, the administrative tools supposed to be under this path, but they vanished

I had to create another machine and installed the already published package on the site (2015.06.24)

I prefer to wait for this version you sent till it's published on the site.
Hala
New member
Username: Haref

Post Number: 12
Registered: 12-2015
Posted on Monday, March 07, 2016 - 03:49 pm:   

Hi,

I tried the new version again on a test server, but still the same issue.
the service is running, but the smpp port or the web interface are not opened.

Is there compatibility issue with windows server 2012 R2? noting that the version (2015.06.24) ran successfully on the 2012 R2 server
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5600
Registered: 08-2008
Posted on Thursday, March 10, 2016 - 11:36 pm:   

Hi Hala,

We are not aware of any compatibility issues, and I'm not understanding the ProgramData issue. (Note that earlier versions of NowSMS used the ProgramData directory, but newer versions, including 2015.06.24 do not unless they see ProgramData\NowSMS\SMSGW.INI. There are never any programs or admin tools in ProgramData.)

Does 2015.06.24 run on your test server?

Enable the SMSDEBUG.LOG on the test server. Restart the NowSMS server. Can I see that SMSDEBUG.LOG?
Hala
New member
Username: Haref

Post Number: 13
Registered: 12-2015
Posted on Wednesday, March 16, 2016 - 02:16 pm:   

Hi,

The version (2015.06.24) is running successfully.
You need the debug to run on the new version (nowsms20160301.zip)?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5606
Registered: 08-2008
Posted on Thursday, March 17, 2016 - 09:17 pm:   

If the newer version does not run, then yes, please enable the SMSDEBUG.LOG so that we can see what is happening. Collect both SMSDEBUG.BAK and SMSDEBUG.LOG.
Hala
New member
Username: Haref

Post Number: 14
Registered: 12-2015
Posted on Sunday, March 27, 2016 - 01:02 pm:   

Hi Des,

The wrong mobile number sms are getting more bad.
Not only number with dashes that stuck, we found now numbers written, but the number is not completed (ex: no country code, not listed in any connection), and also some letters.
Supposed the application to write in the log (No SMSC defined to route message), but the behavior is that the message stays in the queue and causes cpu performance issue.

Kindly advice
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5613
Registered: 08-2008
Posted on Sunday, March 27, 2016 - 11:43 pm:   

Hi Hala,

We really need you to run the newer version (nowsms20160301.zip).

It includes a fix for a problem that we were able to recreate, based upon your original problem description. There is a problem if an SMPP client sends a long message to a number where this destination number includes a - (dash) character.

If you are having a problem with that version, we need to understand what that problem is.


quote:

Not only number with dashes that stuck, we found now numbers written, but the number is not completed (ex: no country code, not listed in any connection), and also some letters.
Supposed the application to write in the log (No SMSC defined to route message), but the behavior is that the message stays in the queue and causes cpu performance issue.




This sounds like a different issue. A configuration issue.

But I need to understand your configuration better.

How many outbound SMSC connections are defined? What are their types? (SMPP? modem?)

What are the preferred routes defined for each SMSC connection?

Does any SMSC connection have "Support any outbound message traffic" checked? If yes, it will be used when there is no route match.

Look at the SMSOUT-yyyymmdd.LOG, which will report each time NowSMS tries to route one of these messages to an SMSC connection. Find an example...let's look at source and destination numbers...review the routes defined for the connection that was used, and that should tell us why.

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 15
Registered: 12-2015
Posted on Monday, March 28, 2016 - 02:46 pm:   

Dear Des,

We succeed in installing the new version on 2008 R2 server.

I will test the messages with wrong format , and check the behavior of the application and feedback.
Hala
New member
Username: Haref

Post Number: 16
Registered: 12-2015
Posted on Tuesday, March 29, 2016 - 11:01 am:   

Hi Des,

We tried the new version (20160301), but still the same issue, messages stuck in the queue.

I downgraded to the version (20150901, under Interim Development Releases) and it worked; the wrong sms are converted to err and dropped from the queue count .
But we noticed that if the option (support any outbound traffic) is unchecked but at the same time No preferable numbers or routes added the message is sent.

Or Should I use version20140630?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5614
Registered: 08-2008
Posted on Tuesday, March 29, 2016 - 01:42 pm:   

Hi Hala,


quote:

We tried the new version (20160301), but still the same issue, messages stuck in the queue.




Can you give me examples of "wrong" numbers?

What exactly is wrong with the numbers? Non-numeric characters? (If yes, provide example.) Or unexpected national vs. international format? (If yes, provide example, as there are settings that can reformat these numbers.)

When the messages are stuck...is any activity reported in SMSOUT-yyyymmdd.LOG for these messages? Or do the stuck messages not appear in this log at all?


quote:

But we noticed that if the option (support any outbound traffic) is unchecked but at the same time No preferable numbers or routes added the message is sent.




I looked again at your old logs. You are using settings to restrict users to particular routes. If a route is restricted to only some user(s), but no numbers are in the route list, then "support any outbound traffic" is enabled for the allowed users.

I am concerned that this is taking so long to resolve your problems, so I think I need to get more logs and info from you.

I have looked again at the logs you sent one month ago.

I see you have over 1000 number patterns listed in various preferred routes.

And 15-20 outbound SMPP connections, most restricted to one or more user accounts.

This is a complicated setup, so to understand what is going wrong, we will need to look at logs and configuration files. I'm guessing that NowSMS is having trouble evaluating all of the rules to determine there is no route for some messages.

Here is how I suggest we proceed:

Make sure SMSDEBUG.LOG is enabled.

Install 20160301 version.

Identify at least 1 (preferably 3 or 4) stuck message(s).

Send me the .REQ file(s) for the message(s), plus SMSDEBUG.LOG, SMSGW.INI, SMSOUT-yyyymmdd.LOG and SMSIN-yyyymmdd.LOG. Send these files to nowsms@nowsms.com with Attention: Des in the subject line. Also, please post a reply here, so that I know to look for the email.

--
Des
NowSMS Support
Hala
New member
Username: Haref

Post Number: 17
Registered: 12-2015
Posted on Tuesday, March 29, 2016 - 04:26 pm:   

I installed it again from scratch, the debug file is not created.
Also I can't open the web interface.
Hala
New member
Username: Haref

Post Number: 18
Registered: 12-2015
Posted on Tuesday, March 29, 2016 - 04:28 pm:   

am using now a test server, the production server still have the version 2015.06.24, as i can't test on it.
Hala
New member
Username: Haref

Post Number: 19
Registered: 12-2015
Posted on Tuesday, March 29, 2016 - 04:41 pm:   

i sent now, the ini file and the req files
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5615
Registered: 08-2008
Posted on Tuesday, March 29, 2016 - 05:44 pm:   

Hi Hala,

We think we understand the problem.

When routes are restricted to specific user accounts, combined with preferred route prefixes, and "Support any outbound message traffic" is not set for any route, messages get stuck.

We've reenable to recreate this, and are testing a fix.

I am going to look over your files and verify that the same problem is affecting you. Assuming it is the same, we will post a fix very soon.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 5616
Registered: 08-2008
Posted on Tuesday, March 29, 2016 - 06:37 pm:   

Hi Hala,

If the test server is having a stuck messages problem, then I believe it is a different problem. (Or the production server is seeing multiple problems.)

On the test server, do you see errors in SMSOUT-yyyymmdd.LOG where these messages are being attempted and retried?

I notice all of the messages you sent have strange sender addresses...for example instead of BANK, they show +BANK. This may be causing the upstream server to reject the message. NowSMS would add this + if the SMPP client submits with a source TON of 1, which indicates an international number. The correct TON value for alphanumeric is 5.

Looking at your old log, I am convinced that the primary stuck message issue is what I described above.

We've posted an update at:

http://www.nowsms.com/download/nowsms20160328.zip

Three changes related to these discussions are included:

* SMS Gateway: Fix for an issue where unroutable messages could get stuck in the outbound message queue when routes are restricted to specific user accounts, combined with preferred route prefixes, and "Support any outbound message traffic" is not set for any route.

* SMPP Server: Non-numeric characters will be stripped from destination/recipient addresses, unless AllowAlphaRecip=Yes is enabled in the [SMSGW] section of SMSGW.INI.

* SMPP Server: If an SMPP client submits a source/sender address that is alphanumeric, but has a TON value of 1 indicating international format, this update will no longer add + to the alphanumeric source address and will treat it as TON=5 (alphanumeric).

--
Des
NowSMS Support