Shared UserDir | Search |
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through July 28, 2010 ⬆ |
◄ ► |
Author | Message | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 110 Registered: 07-2006 |
Hello Des & Bryce! We faced performance problem again, location of UserDir has incredible impact on accepting message. If you store that folder locally - no problem, inbound speed 300 SMS/sec but if you have it shared, in our scenario speed fell to 30-60 SMS/sec. I'm pretty sure if we’ll use optic fiber channel and other expensive stuff we will get more throughput but that is not a solution. Many customers send huge bulk batches and getting throttling errors and what about redundancy and scalability policy scenarios. Sharing of users balances is essential and very useful feature for commercial systems. Please take a look how to improve that situation. Regards, Alex K. | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 111 Registered: 07-2006 |
My thoughts, SMB protocol is not so good for rapid file updates. | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 112 Registered: 07-2006 |
or maybe you know any perfromance tips? | |||
Bryce Norwood - NowSMS Support Board Administrator Username: Bryce Post Number: 7916 Registered: 10-2002 |
Hi Alex, In my opinion, with this type of scenario, performing the charging in the accounting callbacks is what makes the most sense. (That's why I was initially not so excited about the "RouteCharge" scenario.) There are a few things that can be done to improve performance, but I would encourage you to rethink managing credits in the accounting callbacks. Here are some settings that can impact performance: DisableUserQuota=Yes This setting disables any checking of daily/monthly quotas. It also disables the statistics counter for how many messages a user account submits per day. (There is a performance impact updating this file over a network connection, and if you are managing with credits, this is redundant.) DisableUserLog=Yes or UsersLogDir=c:\path Either disable the user log that is created in the USERS directory structure for each user account, or move it to a local path that it separate from the other user data. We've also done a thorough analysis of all of the file i/o that occurs when processing a message received from an SMPP client. And we did find that there was a significant amount of unnecessary and duplicate checks and lookups that we were able to remove. Removing the unnecessary network file i/o requests should provide some help. Whether or not it is enough for your setup, I don't know. If it is not, I'd suggest performing the charging in the accounting callbacks (especially since you're already currently providing us with the charging information already in these callbacks). A preliminary update with reduced file i/o when accepting a message from an SMPP client is at http://www.nowsms.com/download/nowsms20100602.zip. -bn | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 113 Registered: 07-2006 |
Hi Bryce! We reporting increased performance (80-140 SMS/sec) on inbound queue but it's not a maximum like we have on local folder with normal SATA disks. DisableUserLog=Yes we have in our system by default. But setting DisableUserQuota=Yes is not suitable because we always compare financial data from DB and NowSMS. Is there any chance to get more updates on that field? Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2198 Registered: 08-2008 |
Hi Alex, Would it be feasible to have the user quota files on local disks? And then you manually compare by combining them? -- Des NowSMS Support | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 114 Registered: 07-2006 |
Hi Des, We can make that without any problems. But as i sad before, even with DisableUserQuota=Yes speed isn't match if everything stores locally. So, that is a big performance lack, distributed system works slower than a single node. As far as I know, your licensing policy doesn't limit inbound speed, only outbound. Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2205 Registered: 08-2008 |
Hi Alex, I would not expect the network performance to equal local performance. (That said, I'm not an expert on network performance, and I'm sure that there are ways that your current performance level can be increased through network and/or disk drive configurations.) Without a client/server architecture (which introduces additional points of potential failure), there's not much that we can do here. However, the reason for the accounting callbacks is to provide hooks for an external charging mechanism. Essentially, it offers a type of client/server architecture for this scenario. I do have a few other ideas that I'm going to discuss with our engineering team, but I'm not expecting miracles. (When an SMPP client submits a message, perhaps we could do an exclusive lock that would block that client from submitting to other servers on that same cluster for 1/2 second, which would give us time to process more messages.) Maybe we'll have some other ideas, but I still think the accounting callback is where it makes the most sense. -- Des NowSMS Support | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 115 Registered: 07-2006 |
Hi Des, Please excuse me but I don't understand last sentence: "I still think the accounting callback is where it makes the most sense." What do you think about caching user balances locally and sync every 10 sec. Also you can make non-cacheable level to prevent billing errors. Regards, Alex K. | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 117 Registered: 07-2006 |
Hi Des & Bryce, Can we expect any improvements in that field soon? Your answer is much appreciated! Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2222 Registered: 08-2008 |
Hi Alex, Bryce is on holiday this week and I'm going to defer to him as to whether or not there is anything further that we can do. Our opinion based upon precious discussions was that you would be better served by handling charging directly within your accounting callback handlers. Syncing caches local and server based balances would appear to present too many challenges. However I don't think we've fully evaluated the other idea that I presented earlier. I don't expect to have any further followup until mid next week at earliest. -- Des NowSMS Support | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 118 Registered: 07-2006 |
Hi Des, Any good news with that issue? Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2233 Registered: 08-2008 |
Hi Alex, Maybe. We managed to come up with a solution that worked great with a single user streaming messages. But then we mixed it up with multiple submitting user accounts and multiple front-ends, and traffic became bursty with lower throughput. We're testing a hybrid solution. The same user account submitting to multiple front-ends will occasionally block. But it doesn't block other user accounts. And even with the occasional blocking, overall throughput is around double previous efforts. (Maybe better, we're testing with a rather slow back end server.) -- Des NowSMS Support | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 119 Registered: 07-2006 |
Hi Des, Thanks for your support! We also have made many tests and seem that problem is in SMB protocol, disk IO is quite low all the time. As far as I understand, user balances is the only resource which need to be accessed concurrently in multi-server configuration. Maybe you should replace shared folder access for something more productive? Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2251 Registered: 08-2008 |
Hi Alex, The alternative approach is a client/server approach. It introduces other significant potential problems. As you are already using HTTP accounting callbacks, you could modify your accounting callbacks to perform their own quota management, rather than setting the quotas in NowSMS. That is an alternative that you may want to consider. That said, we have been able to speed up the quota tracking and message counters for this type of shared disk environment. The update is available at http://www.nowsms.com/download/nowsms20100617.zip. -- Des NowSMS Support | |||
Alex Kaiser Frequent Contributor Username: Alex_k Post Number: 120 Registered: 07-2006 |
Hi Des, We have tested the release and found that NowSMS issues HTTP 403 (Forbidden) on have traffic submissions. Please check it. Regards, Alex K. | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2271 Registered: 08-2008 |
Hi Alex, I still don't see how the 403 errors are related to any of this. However ... the underlying issue is that at this point, we don't know why you are occasionally seeing 403 errors. Although we have serious doubts that this is the problem, it could be a false error from reading the user account balance, as you suggest. However, it could also be a connectivity error relating to the processing of accounting callbacks, or a problem with the accounting callbacks themselves. To troubleshoot this further, we need to know why the 403 error is being returned. Toward this end, I have asked our engineering team to modify the text returned with the 403 error to include information about why the error was returned. Currently there are only clues in the SMSDEBUG.LOG, however they are like a "needle in a haystack" on a busy system. I don't think it will be a very difficult effort to add this information to the 403 response. Hopefully it will be ready tomorrow or over the weekend. In the meantime, I would suggest that your engineers do whatever is necessary to save the entire HTTP response that accompanies the 403 error so that we can see what the error message is. -- Des NowSMS Support | |||
Des - NowSMS Support Board Administrator Username: Desosms Post Number: 2295 Registered: 08-2008 |
Hi Alex, An update which adds additional information to indicate what triggered the 403 response is at the following link: http://www.nowsms.com/download/nowsms20100628.zip You can either have your application save the complete HTTP 403 response to find the error message, or if the SMSDEBUG.LOG is enabled, look for MessageAccountingReject followed by a text description of why the message was rejected. This will help determine why you are sometimes seeing the 403 quota errors in situations where they are not expected, and we can troubleshoot further from there. -- Des NowSMS Support |