.LCK Files Build-Up

.LCK Files Build-Up SearchSearch
Author Message
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 81
Registered: 06-2004
Posted on Monday, March 08, 2010 - 06:58 pm:   

Hi Bryce, Des

The trouble might be related to issues reported in http://support.nowsms.com/discus/messages/1/41522.html and http://support.nowsms.com/discus/messages/1/33317.html

Afer update (from v. 2009.08.31) to v. 2009.12.08 we are seeing thousands of .lck files accumulating in \q\ folder. For example, a file named SAR-+2302595xxx-9.LCK is stuck, while a corresponding segmented message submitted normally:

2010-03-08 12:37:50,SAR-+2302595xxx-9-2-1.req,94.249.xxx.xx,+2302595xxx,OK -- SMPP - aaa:1234,SubmitUser=sfcom675;Sender=+13472779961;SMSCMsgId=5446194928367970870;U DH=050003090201;Text="+91919313651xxx: <2/3> b paid member plz let me know r u ready for dat or not ,day before Yesterday u told me dat today u r friend karteek wil pay but t"
2010-03-08 12:37:50,SAR-+2302595xxx-9-2-2.req,94.249.xxx.xx,+2302595xxx,OK -- SMPP - aaa:1234,SubmitUser=sfcom675;Sender=+13472779961;SMSCMsgId=5446194928367971126;U DH=050003090202;Text="il now no reply plz cal "

With the previous version, barely any .lck files would get stuck.

Ths is causing a problem, as if not cleaned daily these files cause a parasite CPU load and slow down the server.

FYI, we don't use the "Separate User Queues" setting and the messages in question had normal (as opposed to reverse, e.g. RerouteReceived) flow.

Can you please look at it?


Kind regards,
Ashot
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1919
Registered: 08-2008
Posted on Wednesday, March 10, 2010 - 08:56 pm:   

Hi Ashot,

Can you confirm that there is no external direct manipulation/renaming of the .REQ files?

We spent some time discussing and studying this ... and can't see how any stray .LCK files could ever be left behind.

I can say that one change between versions was that .LCK files are no longer immediately deleted ... but we now wait 2 minutes before deleting.

On a busy system, that could indeed be very substantial.

I wonder if that is what you are seeing? That they aren't actually strays, but a collection of left overs from the past two minutes pending deletion.

Our intent with the 2 minute delay was to deal with the situation where a subsequent part arrives on the server after we've already processed the first part ... so we end up routing the message out via a different SMSC connection. A 2 minute delay may be far too long.

I guess what needs to be determined in your case is whether or not all of these .LCK files are mostly less than 2 minutes old.

--
Des
NowSMS Support
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 82
Registered: 06-2004
Posted on Wednesday, March 10, 2010 - 09:43 pm:   

Hi Des,

Looked at it just now - 4 lck files, two 1 minute and two 2 minutes old. Another four - for failed messages (rejected by SMSC and .err files left in q).

So looks like it's fine now!

When I first checked it last week the folder had almost 80,000 files, and that was positively not for the last 2 minutes. We've deleted them and next day it had 7,000, at which point I've made that post on the discussion board.

After cleaning it up I made some configuration changes and restarted the service. I guess that fixed it.

We've not rebooted that server for a year, maybe it's time.. Although I must admit it's holding up pretty well, compared to our Linux Debian servers. One last weird fluke we've had with one of them - the TIME STOPPED! Seriously, the system clock in the OS stopped and won't run for almost a day until the platform crashed, SMS stopped processing, we got our alerts and noticed the warp, to our amazement.

Kind regards,
Ashot

P.S. We're slowly progressing with the Data_SM/ESM_Class spec and are now testing live with one operator the new version where you've corrected the ESM_Class for the reversed DLR. I'll post in the relevant thread when the spec's ironed out, and with results of the tests. Sorry it's taking much longer than I thought it would. We're pretty backed up..
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 83
Registered: 06-2004
Posted on Sunday, March 14, 2010 - 04:49 pm:   

Hi Des,

Noticed the same problem: at a certain time this morning .lck files started building up in \q\ folder, at several files per minute.

We haven't made any changes to the server at that time. It happened by itself, possibly because of some crooked message.

What in your opinion would be the best means of diagnosing the problem? I've not touched it, so the files keep building up now.

Kind regards,
Ashot
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1934
Registered: 08-2008
Posted on Monday, March 15, 2010 - 04:21 pm:   

Hi Ashot,

I think you may need to update before we can troubleshoot this further.

This problem might be related to a change made for http://support.nowsms.com/discus/messages/1/41960.html.

As I recall, we found that the first implementation of this logic had a flaw that could stop the LCK files from being deleted.

We're looking back through version changes to pinpoint the change. Is the server that is having this problem still running 2009.12.08, or has it been updated?

--
Des
NowSMS Support
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 84
Registered: 06-2004
Posted on Tuesday, March 16, 2010 - 08:15 pm:   

Hi Des,

It's a version 2009.12.08, which handles the issue of misrouting segments of one message correctly.

Strangely, I thought 3-4 weeks ago we have updated it on this server to the 2010 version which had the issue with a wrong esm_class for revrsed DLR fixed. Apparently we haven't though, it still shows the 2009 version.

Earlier today we've received alerts of a couple of thousand .lck-s in the queue folder. Now I'm not seeing any, and this morning I've told everyone specifically not to touch the server or remove the files. Looks like they get deleted erratically by NowSMS.

I've tried stopping/starting the service to see if I could recreate the scenario. It did stop very slowly (2 minutes) and started normally. Next thing I did was I stopped it by unticking the "run as service" and closed the NowSMS interface. Launched it again - and it won't let me activate the service again! Attached please find the screenshot showing the error message (I was logged in as the administrator.)

After several unsuccessful attempts I've rebooted the server and only then was able to start the service.

Another interesting observation: the last section of the install.log in \nowsms\ folder, please see attached. There are many messages in it indicating some files not being copied. Is this normal?

If you think we should update it it'd be great if you could post a link to the latest stable version. Thanks!

Kind regards,
Ashot



application/pdf
Doc182.pdf (216.2 k)
text/plain
install_log.txt (19.7 k)
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 85
Registered: 06-2004
Posted on Tuesday, March 16, 2010 - 08:17 pm:   

BTW,

Can you confirm that there is no external direct manipulation/renaming of the .REQ files?

No, not on this server.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1943
Registered: 08-2008
Posted on Tuesday, March 16, 2010 - 08:35 pm:   

Hi Ashot,

I have to walk away from my computer, so this will be quick, but I wanted to get back to you with version info.

http://www.nowsms.com/download/nowsmsupdate.zip is a good version. 2010.02.09

We are currently using this version as the standard new download as well.

There are some later builds, but they address very specific/minor issues.

install.log is normal. EXE and DLL files are always updated (allows for easily going back and installing older versions if necessary). However, other text or template files are not updated, just in case the customer has made some custom edits.

I'm not sure about the other issue. Error 5 is access denied.

I'm guessing that maybe there is an issue that the service is still in the process of shutting down, and is pending deletion, so the new instance can't be recreated.

I haven't seen it show up in an error this, but I have seen instances where the service is taking a long time to stop. We have investigated a few of those situation. I know we identified one situation late last year and fixed it, but there may be others.

--
Des
NowSMS Support
ashot shahbazian
Frequent Contributor
Username: Animatele

Post Number: 86
Registered: 06-2004
Posted on Tuesday, March 16, 2010 - 11:10 pm:   

Hi Des,

Couldn't update it: got an error message "Software license Agreement Expired".

Kind regards,
Ashot
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 1944
Registered: 08-2008
Posted on Wednesday, March 17, 2010 - 07:28 pm:   

Hi Ashot,

Send a private e-mail to nowsms@nowsms.com with "Attention: Des" in the subject line. Include the serial number and installation reference code, and I'll see if I can get this sorted.

--
Des
NowSMS Support