NowSMS Message ID duplication

NowSMS Message ID duplication SearchSearch
Author Message
Rik Clews
New member
Username: Rikclews

Post Number: 3
Registered: 08-2006
Posted on Wednesday, November 03, 2010 - 02:10 pm:   

We are having an issue with message ID duplication (currently using version 2010.10.22), we do use two servers and we have found the MessageIDPrefix which will alleviate the issue but there are currently 155,000 duplicate IDs in our database with a count of 3 or 4 occurrences. We store 8 weeks worth of messages which is 15 - 20 million at the moment.

Can you tell me how the message IDs are generated? Are they seeded and/or recycled? Could it be due to installing a newer version of NowSMS? How many combinations should there be? All our IDs start "4", which would appear to cut the possible combinations down.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2616
Registered: 08-2008
Posted on Thursday, November 04, 2010 - 02:05 am:   

Hi Rik,

They are seeded on first use, then incremented (and eventually will repeat).

Each server is independent, so there will be duplicates if there are multiple servers. That is why you need to use the MessageIDPrefix setting if you need unique IDs across more than one server.

That said, there is an issue with multipart message IDs (where we generate that SAR-x-x-x format ID), where these can be duplicated. There was a recent update to address this.

--
Des
NowSMS Support
Rik Clews
New member
Username: Rikclews

Post Number: 4
Registered: 08-2006
Posted on Thursday, November 04, 2010 - 10:00 am:   

Thanks Des,

But why do we have a large number (equalling over 1%) of IDs that occur 4 times? I'd assume that's 2 from each server. Does a new install trigger the seeding? If so can it be changed so it carries on if the install is an upgrade - otherwise every upgrade we do we could encounter the same duplication issue, if we have to upgrade "too soon" after the last.

Added the MessageIDPrefix and thats working lovely, with it we are matching ID and mobile numbers on receipts for uniqueness, but its not foolproof and is a large amount of extra processing we could do without, so a solution would be useful.
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2618
Registered: 08-2008
Posted on Thursday, November 04, 2010 - 06:37 pm:   

Sorry Rik,

I missed the bit where you said that there were 3 or 4 occurrences of the same ID. 2 I could understand, but not more than 2 (assuming 2 servers).

Do you still have the SMSOUT-yyyymmdd.LOG files from this time period that you could grep or find your way through.

From a command prompt:

FIND "xxxxxxxxx" SMSOUT*.LOG > TEMP.TXT

(I suggest rerouting output to a file because otherwise FIND echoes to the screen for each file it processes, whether or not it finds a match.)


Actually, as I think this through, I am wondering what you are considering to be a duplicate.

Are these outbound message IDs generated by NowSMS, or inbound receipts? (Maybe you are receiving duplicate inbound receipts?)


The seed would only be reseeded if a file named UID.DAT is deleted. An upgrade install does not delete this file. Uninstalling and re-installing would (just as you would lose other configuration files), but a typical update install has no effect.

I want to make sure I understand this, so please explain further.

--
Des
NowSMS Support
Rik Clews
New member
Username: Rikclews

Post Number: 5
Registered: 08-2006
Posted on Friday, November 05, 2010 - 12:00 pm:   

Ah thanks, suddenly everything is clear.

We recently had an issue with older receipts not finding their corresponding message ID - it appears whilst I was on holiday we rolled NowSMS back to a previous version, ran for half a day and rolled back a further version, before updating to the current version a few days later. Turns out our rollback involves uninstalling the current version and reinstalling the older version, so we were losing the UID.DAT file, the created date for which corresponds nicely with the pairs of duplicates.

Its not something we need to do often, but I assume keeping a copy of the UID.DAT and copying it back in after install would stop it happening if we ever needed to rollback again?
Des - NowSMS Support
Board Administrator
Username: Desosms

Post Number: 2631
Registered: 08-2008
Posted on Friday, November 05, 2010 - 06:56 pm:   

Rik,

Keeping a copy of that UID.DAT file and copying it back will preserve order.

In general, even if you are going back to an earlier version, we do not recommend uninstalling.

All of our installs are designed to restore everything that is needed for the older version to operate properly.

We take extreme care to make sure that any changes to configuration settings are backward compatible (new settings might be recognised by newer versions, but are designed not to cause a problem with an older version).

--
Des
NowSMS Support