Messages accumulate in Q folder while account is bound as transceiver

Messages accumulate in Q folder while account is bound as transceiver SearchSearch
Author Message
Cole
New member
Username: Cole

Post Number: 3
Registered: 02-2013
Posted on Friday, April 19, 2013 - 06:26 pm:   

Hello,

I am seeing a problem with NowSMS handling messages for a particular SMPP account we have configured.

The problem happens randomly and sporadically

The user is bound to NowSMS via SMPP transceiver bind. While bound and able to send SMS with no problems. The Q folder becomes populated with messages that should go down the SMPP bind to the user.

The user has to unbind and then establish a new fresh bind to pull down the messages from the Q folder. A complete restart of the NowSMS service also does this (basically same thing - it makes the user bind reset - and when the user bind reconnects all of the messages in the Q folder go to the user)

-Cole
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4420
Registered: 08-2008
Posted on Friday, April 19, 2013 - 07:22 pm:   

Hi Cole,

What version of NowSMS?

I recall an issue from several years ago where if a client fails to acknowledge a message (NowSMS sends deliver_sm, client fails to send deliver_sm_resp), but the client continues to periodically send enquire_link packets, NowSMS would stop delivering messages.

This particular issue was fixed several years ago, which is why I ask what version you are running.

In current versions, if a client fails to generate deliver_sm_resp, NowSMS will time out the message and move on to the next message.

This issue goes back to 2008 or 2009, so you'd have to be running quite an old version to be encountering this issue.

That said, it is possible that client is in an unstable state. That NowSMS is trying to deliver messages but the client is not processing any of them. In that situation, I don't think we will terminate the connection, but simply keep trying as long as the client sends enquire_link packets to indicate that it is still alive. (We should probably rethink this logic, and assume that if the client does not acknowledge a message, it should be considered unstable and the connection terminated. I will discuss this with my colleagues.)

The bad news is that this is a pretty tough scenario to figure out where the fault is.

My advice would be to use Wireshark to monitor the connection and wait for it to fail. Then we can reconstruct the activity that took place.

How many SMPP clients overall are connected simultaneously? If its a small number, we could also try enabling the SMSDEBUG.LOG/SMPPDEBUG.LOG and get a look at the activity there. But if the problem is on the client end, a Wireshark trace would be more convincing.

If you have any traces you would like me to look at, post in reply here or send the trace file to nowsms@nowsms.com with Attention: Des in the subject line. If you e-mail a file, please post a note in reply here also so that I know to look for the e-mail. Also give me an idea of what I'm looking at in the trace ... explain what devices or clients are on which IP addresses.

--
Des
NowSMS Support
Cole
New member
Username: Cole

Post Number: 4
Registered: 02-2013
Posted on Friday, April 19, 2013 - 07:52 pm:   

v2008.02.22

is this version the problem?

96 smpp clients overall (simultaneously)

I will send you the wireshark logs in an email

-Cole
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4421
Registered: 08-2008
Posted on Friday, April 19, 2013 - 08:13 pm:   

Hi Cole,

Yes, that version will stop transmitting messages to a client if this scenario occurs: Client fails to acknowledge a message (NowSMS sends deliver_sm, client fails to send deliver_sm_resp), and the client continues to periodically send enquire_link packets.

The server end of the connection is essentially stuck waiting for deliver_sm_resp before it will transmit the next message. Normally this should trigger a timeout error, but the timeout counter is getting reset each time the client sends an enquire_link.

In 2009 and later releases, the enquire_link no longer resets the timeout, so the server will move on to the next message.

Technically, it is a client error, in that the deliver_sm_resp is never sent. However, the server does need to robustly handle error conditions, so we did modify the server to better handle this situation.

--
Des
NowSMS Support
Cole
New member
Username: Cole

Post Number: 5
Registered: 02-2013
Posted on Friday, April 19, 2013 - 08:19 pm:   

Ok, is there a manual fix for this problem for the version I am currently using?

or is the only solution to upgrade?

-Cole
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4422
Registered: 08-2008
Posted on Friday, April 19, 2013 - 09:30 pm:   

Hi Cole,

I am not aware of any fix or work-around other than upgrading. (Server or client.)

--
Des
NowSMS Support
Cole
New member
Username: Cole

Post Number: 6
Registered: 02-2013
Posted on Friday, April 19, 2013 - 09:33 pm:   

Ok, can you send me the instructions to upgrade

-Cole
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4424
Registered: 08-2008
Posted on Friday, April 19, 2013 - 09:48 pm:   

As long as your maintenance license is current, it is a very easy process. To check that status, see the "Serial #" page. It will report the status at the top of the page.

Assuming it is not expired, download the latest trial version from our web site and run the install. (NEVER uninstall the current version, let the new install update the system.)

If the maintenance license is expired, there is no harm, the install will display an error and leave the system as is.

Otherwise, it will temporarily stop the running services and update the program files, preserving your existing configuration, then restart the services.

If after installing the update, unexpected issues are encountered, the install from an older version can be used to downgrade.


--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4427
Registered: 08-2008
Posted on Monday, April 22, 2013 - 09:33 pm:   

Hi Cole,

I wanted to provide an additional follow-up. I understand your maintenance/updates license may not be current, so you might not be able to update.

I found the discussion thread where this problem was first reported to us:

http://support.nowsms.com/discus/messages/1/24483.html

You'll see that there is a possible work-around discussed in that thread. Note that the client in the log files that you sent us is sending enquire_link messages at 20 second intervals. So you could try reducing the command timeout to 15 seconds. The problem is, that is a global SMPP setting, so if any of your SMPP connections gets overloaded and is taking a long time to respond, duplicate message deliveries may result. (That said, greater than 15 second delays in response are not something I would expect in anything other than extraordinary circumstances.)

There is also a link to a NowSMS 2008 series update in that thread, and that would be compatible as an update using your existing license code. That particular version was used as our stable release version for at least 6 months.


--
Des
NowSMS Support
Cole
New member
Username: Cole

Post Number: 7
Registered: 02-2013
Posted on Monday, April 22, 2013 - 09:46 pm:   

Would it resolve the problem if I got the user to configure their SMPP client to send enquire link messages at a less frequent rate?

If I got them to change their setting from 20 second intervals to 60 second intervals?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 4428
Registered: 08-2008
Posted on Monday, April 22, 2013 - 10:29 pm:   

Yes Cole, that would be a good idea. Have them set their enquire link interval to 60. (The default timeout interval required for enquire link in NowSMS is 120, set under the SMPP Server options on the web tab ... verify it hasn't been set lower.)

Then I'd suggest a 35 second command timeout using the setting mentioned in that other thread.

--
Des
NowSMS Support