System Error 0x08

System Error 0x08 SearchSearch
Author Message
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 243
Registered: 07-2006
Posted on Tuesday, October 25, 2011 - 12:47 pm:   

Hi Des,

For some unknown reason, NowSMS returns (System Error error code 0x00000008) in response to submit_sm instead "OK". What could happen? After reboot - problem disappears.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3557
Registered: 08-2008
Posted on Tuesday, October 25, 2011 - 07:53 pm:   

Hi Alex,

This could be caused by two possibilities:

1.) Accounting callback failure

2.) Inability to create a message file (insufficient disk space)

Should the problem reoccur, I would suggest enabling the SMSDEBUG.LOG to see what information it is logging.

We have reviewed the above conditions, and the SMSDEBUG.LOG would contain information that makes reference to either SMSAccountingPreAuth or SMPPServerReceiveMessageCallback in the log with additional details before returning this error.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 245
Registered: 07-2006
Posted on Tuesday, October 25, 2011 - 08:35 pm:   

Hi Des,

Thanks for response.
What particular we should look into SMSDEBUG.LOG?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 246
Registered: 07-2006
Posted on Tuesday, October 25, 2011 - 08:38 pm:   

How callback can failure message submission?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3559
Registered: 08-2008
Posted on Tuesday, October 25, 2011 - 09:06 pm:   

PreAuth accounting callbacks can be used to reject a message. (Recall PreAuth=Deny)

As such, any failure processing a PreAuth accounting callback also results in an same error. In this case, error 8.

In recent versions, and invalid SMSCRoute= response in a PreAuth callback will also result in this same error.

I believe the text "SMSAccountingPreAuth" will only be present in the SMSDEBUG.LOG if a message is being rejected because of a PreAuth callback error situation.

If it is a disk related failure, the following text should be logged: "SMPPServerReceiveMessageCallback: User outbound queue file cannot be created".
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 247
Registered: 07-2006
Posted on Tuesday, October 25, 2011 - 09:32 pm:   

Thanks Des.

We enabled SMSDEBUG log and try to catch the exception? But if that is a disk related issue? What should we do? Any suggestions?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 248
Registered: 07-2006
Posted on Wednesday, October 26, 2011 - 01:10 am:   

We ran more tests with callbacks and without but that error occured. What that could mean? Disk problems?

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 249
Registered: 07-2006
Posted on Wednesday, October 26, 2011 - 12:13 pm:   

we found the follwing:

13:44:27:999 [156] ThreadProcessModem: Processing X98D65E23.req...
13:44:28:015 [55] SMPPServerReceiveMessageCallback: X:\Queue\RT_-_SMSGW105_-_6203_xdyn4###3E9A2\
13:44:28:015 [55] SMPPServerReceiveMessageCallback: User outbound queue file cannot be created
13:44:28:031 [598] SMPPServerReceiveMessageCallback: X:\Queue\RT_-_SMSGW105_-_6203_xdyn4###3E9A2\
13:44:28:031 [620] WaitForSocketClose: WinSock reported ioctlsocket complete
13:44:28:031 [598] GetAccountingKeepAlive: Re-using keep-alive socket

But the system has sufficient resources.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3560
Registered: 08-2008
Posted on Wednesday, October 26, 2011 - 06:31 pm:   

Hi Alex,

The thing that strikes me is that it references the "X:" drive, which I assume is a network drive.

It would seem that you are losing your mapping to this drive.

I'd suggest referencing the path using the UNC format of \server\path\ instead of a drive mapping.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 250
Registered: 07-2006
Posted on Wednesday, October 26, 2011 - 06:44 pm:   

It's a local drive for NowSMS DRs, Queues, IN and user files. Formated NTFS 512 cluster size.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3562
Registered: 08-2008
Posted on Wednesday, October 26, 2011 - 06:45 pm:   

You say you have to reboot to recover?

If you stop/start the NowSMS service, does this also recover?
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 251
Registered: 07-2006
Posted on Wednesday, October 26, 2011 - 06:54 pm:   

Yes, that's strange. We created local RAM disk for storing such files because our VDS has very very low IO. Maybe here is the problem?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3565
Registered: 08-2008
Posted on Wednesday, October 26, 2011 - 09:35 pm:   

Then it is possible the RAM disk ran out of storage?

What if you explore the RAM disk when the problem occurs? Does it still seem to have capacity?

I suppose it is also possible the RAM disk driver may not be able to keep up.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 252
Registered: 07-2006
Posted on Wednesday, October 26, 2011 - 10:00 pm:   

RAM disk works fine in virtual environment, errors happens occasionally with some SMPP clients. HTTP isn’t affected. Capacity is under control - from that side all is fine. Another issue, NowSMS doesn't work with FAT file systems. There’s only problem with message submission, outbound SMS is fine.

Regards,
Alex K.
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 255
Registered: 07-2006
Posted on Tuesday, November 22, 2011 - 09:13 am:   

Hi Des,

Is it possible to make some protection against that issue?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3610
Registered: 08-2008
Posted on Tuesday, November 22, 2011 - 02:07 pm:   

Hi Alex,

I'm sorry, but I don't understand. That error is the protection.

If we are unable to create files on that volume, it is necessary to return an error.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 256
Registered: 07-2006
Posted on Tuesday, November 22, 2011 - 03:31 pm:   

Hi Des,

But from the other side, other messages keep going fine. It happens occasionally. What it could be?

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3611
Registered: 08-2008
Posted on Tuesday, November 22, 2011 - 06:54 pm:   

Hi Alex,

Ok. I think I need a better understanding of what is happening.

When this problem occurs, does it affect all SMPP accounts, or only one (or some)?

Is an affected account able to submit via HTTP without issue?

I am wondering if perhaps the virtual RAM disk software has a smaller directory size limit (# of files allowed in any one directory).

By default, a new directory is created every 10000 submissions for a given account. This value can be adjusted with a MaxQDirEntries=##### setting ([SMSGW] section of SMSGW.INI). Perhaps a lower value for that would help if there is such a directory size limit.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 257
Registered: 07-2006
Posted on Tuesday, November 22, 2011 - 07:39 pm:   

Hi Des,

We found only few accounts influenced by but I guess there are more accounts involved. Submission was on small speed 3 sms/sec, I’m not sure that IO issue. RAM driver allows more than 10k files in same directory, we also checked that. Maybe that caused by callbacks? We found another bug here, I'll describe it later.

Regards,
Alex K.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 3612
Registered: 08-2008
Posted on Tuesday, November 22, 2011 - 08:12 pm:   

Hi Alex,

I don't think it's an accounting callback issue. These lines from SMSDEBUG.LOG are indicating a failure to create a file in the directory listed. (If the directory doesn't exist, an attempt to create it would occur, then a retry to create a message file, before this would be logged.)

13:44:28:015 [55] SMPPServerReceiveMessageCallback: X:\Queue\RT_-_SMSGW105_-_6203_xdyn4###3E9A2\
13:44:28:015 [55] SMPPServerReceiveMessageCallback: User outbound queue file cannot be created

I did just review this with one of my colleagues, and we did come up with an idea of a situation where if multiple different multipart messages, sent to the same recipient address, using the same sequence number, could result in this error. However, the segments would have to be sent out of order in order to trigger this error. (And the error would clear automatically after 5 minutes.)

So, we'd like to see more SMSDEBUG.LOG to determine if this is the problem. Or maybe there is something else that we are missing.

Using the above log excerpt as an example, here's what we'd like to see.

Wherever you see, "User outbound queue file cannot be created", note the [xxxx] that appears in that same line. Filter the SMSDEBUG.LOG so that it only shows lines that contain the same [xxxx].

An easy way to do this is:

find "[xxxx]" smsdebug.log > xxxx.log

You can e-mail the logs to nowsms@nowsms.com with a subject line of "Attention: Des" as they will likely contain other information that should not be shared publicly.

--
Des
NowSMS Support
Alex Kaiser
Frequent Contributor
Username: Alex_k

Post Number: 258
Registered: 07-2006
Posted on Wednesday, November 23, 2011 - 08:37 am:   

Hi Des,

Unfortunately, I’ve deleted these logs but I’ll try to catch that issue again. From my observations:

- Error affects SMPP clients in TRX modes (I guess)
- Error wouldn’t clear after 5 minutes or more
- Error clears if we restart service
- Error clears in 50% cases if SMPP client restarts

Regards,
Alex K.