Callback issue#2

Callback issue#2 SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 259
Registered: 07-2006
Posted on Wednesday, November 23, 2011 - 09:23 am:   

Further tests revealed that callbacks have negative performance effect.

- SMPP clients with session (TRX, I guess) older than 2 days occasionally can’t send faster than 20sms/sec, after NowSMS’s reboot problem disappear.
- SMPP clients in TRX mode on submission can’t receive DRs, if speed more 20sms/sec, we unable to fix that issue.

If callbacks removed – everything goes smoothly.

We tried parameters AccountingKeepAlive and AsyncCallbackThreads but no luck. I suppose that issue also has an influence on outbound speed, we saw many times that amount http requests to callback’s handler drastically decreased without no reason. I recommend to make separate handlers URLs for PreAuth (most resources consuming) and other callbacks. Please investigate.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3626
Registered: 08-2008
Posted on Monday, November 28, 2011 - 06:11 pm:   

Hi Alex,

It is unclear to me from your description whether both of these problems are related to long lived (> 2 days) TRX sessions. Or if it is only the first.

My first advice would be to try converting the TRX sessions to separate TX and RX sessions to see if this resolves the issue.

If callbacks are getting delayed, we don't see anything in NowSMS that would cause this to degrade over time.

However, if keep-alives are not being used (or possibly even if not), then maybe the TCP/IP stack itself is experiencing timed wait delay issues. I'm surprised that I can't find any other references to this in our forum, but as a rule for our testing servers, we always use the TcpTimedWaitDelay and MaxUserPort registry settings. We typically set them to the values suggested in the following Microsoft KB article: http://support.microsoft.com/kb/970406

Note that we do not queue accounting callbacks, even if keep-alive sessions are used. If an accounting callback is active for a request on one session, and another session needs to make an accounting callback, a new connection is allocated.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 260
Registered: 07-2006
Posted on Monday, November 28, 2011 - 07:52 pm:   

Hi Des,

Is that NowSMS or Callbacks server settings?

Regards,
Alex K
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3627
Registered: 08-2008
Posted on Monday, November 28, 2011 - 10:04 pm:   

Hi Alex,

I'd recommend both. These settings are good for any system that is initiating or accepting a large number of TCP connections.

The MaxUserPort setting is more important for the side that is initiating the connection (NowSMS), because that is the client end of the accounting callback connections, where the dynamic port numbers are allocated.

Lowering TcpTimedWaitDelay on both sides is a good idea.

Out of curiosity, do you know whether or not your accounting callback server is actually supporting keep-alive sockets for your callbacks? (Admittedly, it can be difficult to determine without using Wireshark.)

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 261
Registered: 07-2006
Posted on Tuesday, November 29, 2011 - 04:01 pm:   

Hi Des,

Let us make some more tests.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 262
Registered: 07-2006
Posted on Thursday, December 08, 2011 - 10:59 am:   

Hi Des,

Seems that we found the issue, SMSGW.ini file was opened with text editor which made a lock on it. Seems that NowSMS goes crazy if can't achive access to that file. Could you fix it somehow?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3669
Registered: 08-2008
Posted on Friday, December 09, 2011 - 06:55 pm:   

Hi Alex,

Do you have any ideas what kind of lock it places on that file?

NowSMS definitely needs to be able to access that file. It typically will scan for updates to the last modified file date/time, and then reloads it.

Are you making lots of edits to this file?

I'm not sure there is much we can do, except maybe try to be less aggressive about reloading the changed file if it is locked.

A text editor that doesn't lock the file so other processes can read it would be a good idea.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 266
Registered: 07-2006
Posted on Tuesday, December 20, 2011 - 12:53 pm:   

Hi Des,

Not a lot but we noticed that receiving speed can fall down occasionally after that operation.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 267
Registered: 07-2006
Posted on Tuesday, December 20, 2011 - 12:58 pm:   

We noticed that issue after adding/modifying/deleting users thru GUI. Then we caught that issue after manual edit of SMSGW.INI. I suppose that SMSUsers.D2A and SMSUsers.D2I files can be affected by that issue.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 268
Registered: 07-2006
Posted on Thursday, January 05, 2012 - 10:11 am:   

Hi Des,

Looks that problem occurs when you have huge SMSGW.ini file, more than 50 connections. I happen each time when we change it without NowSMS restart.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3704
Registered: 08-2008
Posted on Friday, January 06, 2012 - 12:54 pm:   

Ok Alex,

I will discuss this with our engineering team. Perhaps NowSMS needs to open the file for exclusive write access (even though it is only reading) and if this is not available, keep the cached copy in memory until it is available.

I'm sure one of the engineers will have a better understanding than me.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3774
Registered: 08-2008
Posted on Friday, February 10, 2012 - 07:20 pm:   

Hi Alex,

A follow-up.

We have made changes to how SMSGW.INI is reloaded. As you know, the content of this file is cached in memory while NowSMS is running.

If SMSGW.INI is updated, NowSMS will now only reload the file if no other process has the file open for write access.

Previously, if another process had the file open for write access, but allowed sharing, NowSMS might try to reload SMSGW.INI while the other process was midway through applying updates.

The new logic does prevent this in v2012.02.03 and later.

I am replying to you on a different issue shortly, so I will post a relevant link in that response.

--
Des
NowSMS Support