UsersDir parameter in multi-server config

UsersDir parameter in multi-server config SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 54
Registered: 07-2006
Posted on Thursday, December 03, 2009 - 07:57 pm:   

Hi!

We tried to implement multi-server load balancing scenario.
1st server stores and shares(can be accessed anonymously) SMSUsers.D2A/D2I files
2nd server has parameter
UsersDir=\192.168.0.20\NowSMS.Users

We can add/update/delete user accounts from 2nd server but when we try to send SMS from 2nd we got authorization error. Haven't tried with SMPP access but suppose that situation would be the same. Any ideas where to dig?

Another question, are you sure that it won't be any file locks to shared files(*.ctr, *.qta)? Accounts can be simply misbilled. We are using NowSMS ver. 20091001.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1499
Registered: 08-2008
Posted on Friday, December 04, 2009 - 10:22 pm:   

Hi Alex,

This requires further investigation. I can't give a quick answer.

The counters and quota should be ok.

But I'm at a loss to explain the user auth error.

My thought is that there may be an issue with how the user interface program adds a user. It keeps the SMSUSERS file open, and this may cause it not to be flushed to disk for the other server to recognise the added user.

We need to do more testing, but at this point, that is the initial thought on this issue.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 55
Registered: 07-2006
Posted on Monday, December 07, 2009 - 07:32 pm:   

Hi Des,

By these explanations, you mean that multi-server config isn't possible atm? Do you have any forecasts when situation would change?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1513
Registered: 08-2008
Posted on Tuesday, December 08, 2009 - 12:24 am:   

Hi Alex,

No, that's not what I mean.

I think if you exited the NowSMS configuration program after adding the user, this would flush the users file so that the new user could be seen.

I think the issue is the way that the configuration program keeps the users file open when adding SMS users.

--
Des
NowSMS Support
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1514
Registered: 08-2008
Posted on Tuesday, December 08, 2009 - 01:31 am:   

Alex,

Is it possible that it is a configuration issue?

You say that 2nd server has a "UsersDir" setting. Does the 1st also? (By default, SMSUSERS files are in the NowSMS directory, not user NowSMS\Users.)

Have you modified the service to use a login as a user instead of "localsystem"?

Is the server a NAS or a Windows server? If its a Windows server, the remote system needs to be able to access it while running as a service, so login credentials can be required.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 56
Registered: 07-2006
Posted on Sunday, December 13, 2009 - 01:59 pm:   

Hello Des,

We followed your recommendations and it works :-)

But balance checking with NowSMS doesn't work anymore. Please check.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 57
Registered: 07-2006
Posted on Monday, December 14, 2009 - 08:45 am:   

We made some config mistakes, now problem solved.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 58
Registered: 07-2006
Posted on Monday, December 14, 2009 - 09:33 am:   

And one more question, have you made any performance tests? Accessing files over shares is fast enough?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1553
Registered: 08-2008
Posted on Tuesday, December 15, 2009 - 01:19 am:   

Hi Alex,

You have to keep in mind that local area network transfer speeds (gigabit) are far faster than disk drive transfer speeds.

What can be a problem though is that more load is placed on the underlying disk subsystem on the server. So its transfer speed is significant.

RAID systems tend to be a good idea for faster disk subsystems.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 59
Registered: 07-2006
Posted on Tuesday, December 15, 2009 - 01:41 pm:   

Hi Des,

We made some performance tests.
We submitted 500 sms (1 sms = 1 request) over HTTP to the local server with different "UserDir" settings. "UsersDir=<local path>" is 4 times faster than "UsersDir=<shared path>". I believe, accessing files on shares isn't so quick as sending sql stored procedures. My opinion - current NowSMS's file IO system is legacy and must be replaced with sql engine (even free open-source) :-(

Regards,
Alex K.
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 66
Registered: 06-2004
Posted on Tuesday, December 15, 2009 - 09:25 pm:   

Alex, Des:

We have a different view on this: file-based I/O provides for inherent simplicity and redundancy, simply because the filesystem itself is a simple thing and file storage can be easily made extremely redundant. This isn't true about SQL engines and databases in general, which require a lot more maintenance and in the end rely on the same filesystem. Real-time systems built around SQL engines are more complex, less reliable, not necessarily faster and must be maintained by the same pro-s who have designed and built them. Example: NowSMS can be used by any engineer experienced in messaging, with no support by the developer apart from occasional forum posts. The truth is that it never breaks down. A 100 SMS/sec. SMSC built on an SQL or Oracle DB could cost in support fees to the SMSC developer as much as you'd have paid for a 100 SMS/sec. NowSMS license, PER MONTH. And SMSC-s simply don't work without support from their developers.

A simple network file share though is indeed too slow for the purpose. I would correct Des: a raw speed of a 2U RAID array with 24 modern 2.5" SAS or SATA SLC SSD disks can be 60 times that of a gigabit Ethernet link, even before the TCP/IP overhead. We typically connect these via 40gbps Infiniband or multiple 10gE links to level the bandwidth of disks arrays with that of the storage fabric and the hosts.

So if you use a real high-performance shared storage (Fibre Channel SAN, or an "iSCSI SAN" over redundant 10gE links) for your MowSMS queue, stats, logs and other fast-changing files you would never notice any disk I/O overhead even if you have 10 NowSMS hosts running at full speed on top of one such shared array. It is expensive, but still a lot less (and 100s times simpler) than maintaining DB-based solutions.

Kind regards,
Ashot
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 63
Registered: 07-2006
Posted on Monday, December 21, 2009 - 08:15 pm:   

Hello Ashot & Des!

Thanks for your responses!

I'm not talking about hardware performance in general, we can buy even 100 SSD disks array but I just suppose that NTFS file system wasn't created for such fast read/write/modify IO file operations. Of course, maintenance of DB costs but I believe if you're using free DB like MySQL and 1 experienced DB administrator it won't be a waste of money. Even NowSMS team can adjudge advantages of such approach. Many new tasks can be performed, like messages resending, statistics withdrawals, account backups and etc.

In meantime, we still have performance issue. In our test environment, we have 2 test servers (2003 server), both servers has RAM drives. all NowSMS's folders ("%APP%", "UsersDir", "QDir", "SMSInDir", "MessageIDTrackingDir") installed there (for ultimate IO performance, near to SSD disks array), only "LogDirectory" installed on RAID5 15k SAS disks. 1st server is sharing "UserDir", 2nd server has "UsersDir=\192.168.0.20\NowSMS.Users". Both server started with different user account (non "local system"). In that configuration submission speed (over SMPP) near 20-40 SMS/sec on each server. But in standalone configuration (started with "local system" account and non-shared "UserDir") performance about 100-300 SMS/sec.

Maybe we’ve missed something or any other system tune-ups are possible?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 64
Registered: 07-2006
Posted on Monday, December 21, 2009 - 08:26 pm:   

just forgot to say, that we're using 100MB network adapters, maybe here could be a problem?
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 77
Registered: 06-2004
Posted on Monday, December 21, 2009 - 11:38 pm:   

Hi Alex,

Your problem may indeed be the 100mbps Ethernet. It's plain too slow for an application relying on a filesysem for essential stuff.

If your servers have dual 1G Ethernet adapters, I'd suggest you connect them through the 2nd ports with a crossover cable (unless you have a 1G switch) and make each server's storage visible to the other via NetBios. Chances are it'd improve performance to the point you'd no longer see much difference between "local" and "network shared" storage scenarios.

Another very important difference between SAN (e.g. Fibre-Channel based) shared storage and IP-based (a network share or iSCSI) is TCP/IP overhead. We just did comparative tests: throughput and IOPS difference between IP-based and local or FC can be 3-fold. You can actually see the extra TCP/IP load due to file activity on network shares or iSCSI in Windows Performance monitor, if you make it show the relevant parameters.

Hope it helps.

I agree with you that a bespoke DB or whatever other SMPP solution can be loads more efficient and useful than than NowSMS or any other off-the-shelf product. But if it's a universal middleware product it's hardly possible to make it DB-based and keep the day-to-day maintenance and support costs as low as they are with a file-based solution - for both the developer and the customer.

It's realtively easy to write a full SMPP platform and make it awfully fast. 3-4 experienced coders would do a working one in 3-4 months. The trouble is that for the next 3-4 years they'd be fixing bugs and inconsistencies. So it really depends on what your priorities are.

Kind regards,
Ashot