Shared UserDir

Shared UserDir SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 110
Registered: 07-2006
Posted on Friday, May 28, 2010 - 07:14 pm:   

Hello Des & Bryce!

We faced performance problem again, location of UserDir has incredible impact on accepting message. If you store that folder locally - no problem, inbound speed 300 SMS/sec but if you have it shared, in our scenario speed fell to 30-60 SMS/sec. I'm pretty sure if we’ll use optic fiber channel and other expensive stuff we will get more throughput but that is not a solution.

Many customers send huge bulk batches and getting throttling errors and what about redundancy and scalability policy scenarios. Sharing of users balances is essential and very useful feature for commercial systems. Please take a look how to improve that situation.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 111
Registered: 07-2006
Posted on Friday, May 28, 2010 - 08:13 pm:   

My thoughts,

SMB protocol is not so good for rapid file updates.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 112
Registered: 07-2006
Posted on Wednesday, June 02, 2010 - 12:43 pm:   

or maybe you know any perfromance tips?
Bryce Norwood - NowSMS Support
Board Administrator
Username: Bryce

Post Number: 7916
Registered: 10-2002
Posted on Thursday, June 03, 2010 - 02:56 am:   

Hi Alex,

In my opinion, with this type of scenario, performing the charging in the accounting callbacks is what makes the most sense. (That's why I was initially not so excited about the "RouteCharge" scenario.)

There are a few things that can be done to improve performance, but I would encourage you to rethink managing credits in the accounting callbacks.

Here are some settings that can impact performance:

DisableUserQuota=Yes

This setting disables any checking of daily/monthly quotas. It also disables the statistics counter for how many messages a user account submits per day. (There is a performance impact updating this file over a network connection, and if you are managing with credits, this is redundant.)

DisableUserLog=Yes
or
UsersLogDir=c:\path

Either disable the user log that is created in the USERS directory structure for each user account, or move it to a local path that it separate from the other user data.


We've also done a thorough analysis of all of the file i/o that occurs when processing a message received from an SMPP client. And we did find that there was a significant amount of unnecessary and duplicate checks and lookups that we were able to remove.

Removing the unnecessary network file i/o requests should provide some help. Whether or not it is enough for your setup, I don't know. If it is not, I'd suggest performing the charging in the accounting callbacks (especially since you're already currently providing us with the charging information already in these callbacks).

A preliminary update with reduced file i/o when accepting a message from an SMPP client is at http://www.nowsms.com/download/nowsms20100602.zip.

-bn
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 113
Registered: 07-2006
Posted on Thursday, June 03, 2010 - 09:36 am:   

Hi Bryce!

We reporting increased performance (80-140 SMS/sec) on inbound queue but it's not a maximum like we have on local folder with normal SATA disks.

DisableUserLog=Yes we have in our system by default. But setting DisableUserQuota=Yes is not suitable because we always compare financial data from DB and NowSMS.

Is there any chance to get more updates on that field?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2198
Registered: 08-2008
Posted on Thursday, June 03, 2010 - 04:18 pm:   

Hi Alex,

Would it be feasible to have the user quota files on local disks? And then you manually compare by combining them?

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 114
Registered: 07-2006
Posted on Thursday, June 03, 2010 - 06:31 pm:   

Hi Des,

We can make that without any problems. But as i sad before, even with DisableUserQuota=Yes speed isn't match if everything stores locally. So, that is a big performance lack, distributed system works slower than a single node. As far as I know, your licensing policy doesn't limit inbound speed, only outbound.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2205
Registered: 08-2008
Posted on Thursday, June 03, 2010 - 07:09 pm:   

Hi Alex,

I would not expect the network performance to equal local performance. (That said, I'm not an expert on network performance, and I'm sure that there are ways that your current performance level can be increased through network and/or disk drive configurations.)

Without a client/server architecture (which introduces additional points of potential failure), there's not much that we can do here.

However, the reason for the accounting callbacks is to provide hooks for an external charging mechanism. Essentially, it offers a type of client/server architecture for this scenario.

I do have a few other ideas that I'm going to discuss with our engineering team, but I'm not expecting miracles. (When an SMPP client submits a message, perhaps we could do an exclusive lock that would block that client from submitting to other servers on that same cluster for 1/2 second, which would give us time to process more messages.) Maybe we'll have some other ideas, but I still think the accounting callback is where it makes the most sense.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 115
Registered: 07-2006
Posted on Friday, June 04, 2010 - 03:49 pm:   

Hi Des,

Please excuse me but I don't understand last sentence: "I still think the accounting callback is where it makes the most sense."

What do you think about caching user balances locally and sync every 10 sec. Also you can make non-cacheable level to prevent billing errors.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 117
Registered: 07-2006
Posted on Wednesday, June 09, 2010 - 01:55 pm:   

Hi Des & Bryce,

Can we expect any improvements in that field soon?

Your answer is much appreciated!

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2222
Registered: 08-2008
Posted on Thursday, June 10, 2010 - 02:44 am:   

Hi Alex,

Bryce is on holiday this week and I'm going to defer to him as to whether or not there is anything further that we can do.

Our opinion based upon precious discussions was that you would be better served by handling charging directly within your accounting callback handlers.

Syncing caches local and server based balances would appear to present too many challenges. However I don't think we've fully evaluated the other idea that I presented earlier.

I don't expect to have any further followup until mid next week at earliest.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 118
Registered: 07-2006
Posted on Tuesday, June 15, 2010 - 04:08 pm:   

Hi Des,

Any good news with that issue?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2233
Registered: 08-2008
Posted on Tuesday, June 15, 2010 - 05:52 pm:   

Hi Alex,

Maybe.

We managed to come up with a solution that worked great with a single user streaming messages.

But then we mixed it up with multiple submitting user accounts and multiple front-ends, and traffic became bursty with lower throughput.

We're testing a hybrid solution. The same user account submitting to multiple front-ends will occasionally block. But it doesn't block other user accounts. And even with the occasional blocking, overall throughput is around double previous efforts. (Maybe better, we're testing with a rather slow back end server.)

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 119
Registered: 07-2006
Posted on Tuesday, June 15, 2010 - 10:10 pm:   

Hi Des,

Thanks for your support!

We also have made many tests and seem that problem is in SMB protocol, disk IO is quite low all the time. As far as I understand, user balances is the only resource which need to be accessed concurrently in multi-server configuration. Maybe you should replace shared folder access for something more productive?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2251
Registered: 08-2008
Posted on Thursday, June 17, 2010 - 05:26 pm:   

Hi Alex,

The alternative approach is a client/server approach. It introduces other significant potential problems.

As you are already using HTTP accounting callbacks, you could modify your accounting callbacks to perform their own quota management, rather than setting the quotas in NowSMS. That is an alternative that you may want to consider.

That said, we have been able to speed up the quota tracking and message counters for this type of shared disk environment.

The update is available at http://www.nowsms.com/download/nowsms20100617.zip.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 120
Registered: 07-2006
Posted on Thursday, June 24, 2010 - 05:02 pm:   

Hi Des,

We have tested the release and found that NowSMS issues HTTP 403 (Forbidden) on have traffic submissions. Please check it.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2271
Registered: 08-2008
Posted on Thursday, June 24, 2010 - 10:56 pm:   

Hi Alex,

I still don't see how the 403 errors are related to any of this.

However ... the underlying issue is that at this point, we don't know why you are occasionally seeing 403 errors.

Although we have serious doubts that this is the problem, it could be a false error from reading the user account balance, as you suggest.

However, it could also be a connectivity error relating to the processing of accounting callbacks, or a problem with the accounting callbacks themselves.

To troubleshoot this further, we need to know why the 403 error is being returned.

Toward this end, I have asked our engineering team to modify the text returned with the 403 error to include information about why the error was returned. Currently there are only clues in the SMSDEBUG.LOG, however they are like a "needle in a haystack" on a busy system.

I don't think it will be a very difficult effort to add this information to the 403 response. Hopefully it will be ready tomorrow or over the weekend. In the meantime, I would suggest that your engineers do whatever is necessary to save the entire HTTP response that accompanies the 403 error so that we can see what the error message is.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2295
Registered: 08-2008
Posted on Tuesday, June 29, 2010 - 05:53 pm:   

Hi Alex,

An update which adds additional information to indicate what triggered the 403 response is at the following link: http://www.nowsms.com/download/nowsms20100628.zip

You can either have your application save the complete HTTP 403 response to find the error message, or if the SMSDEBUG.LOG is enabled, look for MessageAccountingReject followed by a text description of why the message was rejected.

This will help determine why you are sometimes seeing the 403 quota errors in situations where they are not expected, and we can troubleshoot further from there.

--
Des
NowSMS Support