| Many connections established but don't close |  Search
|
|
NowSMS Support Forums ⬆ NowSMS Support - SMS Issues ⬆ Archive through April 13, 2018 ⬆ |
◄ ► |
| Author | Message | ||||
| Ahmed Negm New member Username: Anegm Post Number: 8 Registered: 04-2014 |
Hi, I have connected to one of mobile carriers by establishing 6 connections with one IP and 6 ports. This part was easy, but we noticed that there are some undelivered messages. So, we asked the operator and they replied: "All messages has been forwarded to you as per logs, and we have seen that you established many connections to us, which might cause this issue". Although, they pushed the messages to us, we didn't get them yet, so, they said: "It seems that your application doesn't close the connection properly" Is there any suggestions or recommendations we can do or we can ask them about? | ||||
| Ahmed Negm New member Username: Anegm Post Number: 10 Registered: 04-2014 |
[UPDATE] I also allowed 1TX and 1RX for each connection, but they also saw many connections established and not closed properly from our side, so, all of my connections are down because there are many connections which are up on the server's side. Any idea about that? | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5828 Registered: 08-2008 |
Hi, That really doesn't make any sense. If you run the command netstat -a -n on your server, do you see a large number of connections? This is a long shot, but is there a file named EXCEPT.LOG in the NowSMS directory that has been updated recently? What is the NowSMS version? Is async mode enabled? (If no, try enabling it. If yes, try disabling.) Regarding missing messages, the provider should not consider a message delivered unless acknowledged with deliver_sm_resp. It might help to look at some debug logs or traces. Enable the SMSDEBUG.LOG. I'm not sure what we're looking for yet...please comment on the above first. -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 12 Registered: 04-2014 |
Hi Des, I checked the netstat and I saw my connections only as per my sessions configurations, so we are fine with this, but I asked the operator to check the same at his end, and he tole me that there are many connections established by my gateway. Also, I check the EXCEPT.log, and I fount the below: Exception occurred in thread #60 Thread Description: ThreadSmppSubmitAsyncHelper Thursday, May 4, 2017 @ 2:55 PM I am using NowSMS 2012. And for SMSDEBUG.log, there was nothing odd there. Also, I enabled ASYNC, and nothing changed there. Any other actions/procedures should I take? | ||||
| Ahmed Negm New member Username: Anegm Post Number: 13 Registered: 04-2014 |
Also, they asked me to check why my application doesn't close the connection properly before create new one. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5829 Registered: 08-2008 |
Please enable the SMSDEBUG.LOG. If EXCEPT.LOG gets updated again, send me SMSDEBUG.LOG/BAK and SMPPDEBUG.LOG/BAK Also send me EXCEPT.LOG Send via email to nowsms@nowsms.com with Attention: Des in the subject line. What is your async window size? If large, try lowering. Also try restarting Windows ... with this oddity, it is possible the TCP stack has crashed. In fact, please try that before we look at debug logs. -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 14 Registered: 04-2014 |
Hi Des, I contacted the service provider and he sent the below reply after sharing our TCP trace log with him during an hour test session : ========== As we told you previously we are getting “deliver_sm response: ok” but the problem is different. You are establishing more than allowed connections it’s because of that you are not getting MO from us. I’ll tell you what the exact issue here. when we receive MO from user we will pick your currently established IP and port and forward that MO to your end but you might be waiting to receive that MO in an already established stale connections port and you are not receiving MO there. But we are forwarding that MO to your latest established port this is what happening right now that’s the reason we asked you to resolve doing multiple connection establishment. Please try to compare both the TCP trace and please try to trouble shoot this issue from your end and avoid establishing multiple connections at our end to get rid of this issue. ========== So, we are talking about establishing many connections and don't close them properly when trying to open a new one, so, we lost messages and also get the connectivity down as there is no capacity for new pools/connections. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5830 Registered: 08-2008 |
Hi Ahmed, Please refer to my previous reply. Are there many updates to EXCEPT.LOG and have you restarted the Windows server? -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 15 Registered: 04-2014 |
Hi Des, I checked the EXCEPT.log and there's no updates in the file since I shared the above content. Tomorrow morning I will restart the server, but do you believe that is necessary? Also, if sharing the TCP trace of both us and service provider would be helpful, I can share them with you. BTW, today we conduct a test and we noticed something weird. We noticed that the message didn't reach our end although the service provider got deliver_sm resp: ok from our side as per their logs. How could this be happened? If there are any other recommendations or anything I can ask the SP about, Kindly share the same with me. I really appreciate your kind support on this. | ||||
| Ahmed Negm New member Username: Anegm Post Number: 16 Registered: 04-2014 |
Another point, can we have something to keep the connection always up/alive instead of closing and establishing a new one from time to time? | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5831 Registered: 08-2008 |
Hi Ahmed, Thank you for the logs. I am glad that EXCEPT.LOG is not being updated, as that would suggest more serious problems.
What you are requesting is normal expected behavior. Connections are kept open unless: - timeout error waiting for response from partner - connection terminated by partner - connection terminated by TCP stack The logs do NOT show a timeout problem. (If a timeout does occur, every attempt is made to close it gracefully so that the partner recognizes the disconnect.) The logs do show unexpected connection termination. This makes the Wireshark traces interesting...in fact, it is very interesting that you have traces from both your side and the service provider side...because here is what I see... In your trace, around the time of disconnects, Wireshark reports many ACKs for missing packets and retransmissions. A TCP RSET is transmitted *to* NowSMS to terminate the connection. On the provider side, they are receiving a TCP SYN ECN CWR notifying them of network congestion. I believe network congestion between the systems (most likely on your side) is the root of the problem. One potential quick fix...if you are running a Windows version earlier than Server 2012, ECN (Explicit Congestion Notification) is disabled by default. Instructions for enabling can be found here: https://social.technet.microsoft.com/wiki/contents/articles/20204.how-to-enable- and-disable-explicit-congestion-notification-in-windows.aspx -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 17 Registered: 04-2014 |
Hi Des, We are using Microsoft Windows Server 2012, and we have configured all above instructions, but we still have some missing messages due to many connections issue. On the other hand, we forwarded your reply to SP technical team and I highlighted their reply in YELLOW (PFA). I believe that NowSMS has no issue, but we need your thoughts and recommendations about the issue.  | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5833 Registered: 08-2008 |
Hi Ahmed, I have no knowledge of what IPs belong to which party, so I may be confused as to which side initiates the RSET. After further analysis, the trace on your end does seem to show a RSET being received from their system. Here is one of many examples:  You are sending the enquire_link, and they are sending the response ... so I believe that I am correct to say that this RSET appears to be coming from them. At an application level, this is consistent with what NowSMS is seeing, and this is why it initiates a new SMPP connection. The service provider trace shows nothing...just an idle 60+ seconds. They receive no notification that the connection has been reset, and they do not appear to be initiating the RSET. But it is coming from somewhere. Regarding their "RSET, ACK" comment, that is not consistent with the cases I have looked at. For example, consider this:  Wireshark says this is an ACK to an enquire_link (frame 204), and a RSET ... not an ACK of a reset. Please understand, I am not trying to place blame anywhere...only trying to understand the problem. I do not think the service provider is doing anything wrong. And I do not think NowSMS is doing anything wrong. Both are acting as expected. (Except as it relates to missing messages. A connection dropping or timing out and being reestablished is cause for delayed delivery, but not cause for aborted delivery.) This type of issue is why SMPP has enquire_link...eventually, the provider should timeout the connection for not receiving enquire_link or other activity. What I cannot explain is where this RSET is coming from. I can only suspect network congestion and/or a router that is overloaded and dropping packets. It concerns me that the trace on your end shows so many missing packets, acks for unseen segments and retransmissions. -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 18 Registered: 04-2014 |
Hi Des, Here you are the IPs if this would help. SP IP: 185.127.160.6 and our IP: 192.168.10.47 I really appreciate your kind help on the same. | ||||
| Ahmed Negm New member Username: Anegm Post Number: 19 Registered: 04-2014 |
Dear Des, I am sorry to tell you that the mentioned problem is persists. So, we expand our traces and got a lot of TCP dump files to get the source of the issue, the last trace was yesterday, and we noticed the issue there: For MO: we noticed that the SP deliver the MO to us through another port which is different from the one we send Enquire_link, and when we asked him to do "netstat -na | grep ..." we noticed that we are already having many established connections there and he deliver the message to one of them which is not currently active at our end. For MT: we have no issues as all MTs have been delivered successfully. We did some searches and we noticed that there is an old version of NowSMS allowed to specify "Sender or Transceiver port", so, can we have the same in 2012, 2016 or 2017 through SMSGW.ini configuration file, because I couldn't get it in the interface? On the other hand, kindly find the attched TCP trace file with the following notes: 1. Our TCP called "Y2D_Trace" and our IP is "192.168.10.46", so the other IP is SP's one. 2. SP TCP called "TIMWE" and our masking IP there is "10.200.3.170", so the other IP is SP's one. 3. You can filter by typing SMPP and you will see the above issue of delivering MOs in different port. I really appreciate your kind and prompt support regarding to this. Thanks in advance.
| ||||
| Ahmed Negm New member Username: Anegm Post Number: 20 Registered: 04-2014 |
Hi Des, We noticed that NowSMS opens more than one port during the connection to the SP and I have no issues with that as it works fine with other SPs but it doesn't work with this SP. After reading the Advanced INI Configurations topic, I see ReceivePort parameter which can enable me to specify one fixed port to be used during connection, I just need your confirmation if this parameter can help me to get the issue fixed and avoid establishing more than one port. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5834 Registered: 08-2008 |
Hi Ahmed, The problem is that on our end, a RSET has been received. This means that the connection is closed and we can no longer access it. The operating system closes the connection because of the RSET. I have no idea what is causing this RSET, but this is out of our control. When the provider attempts to send data over that connection, they should encounter a TCP error. If they do not, they still will not receive deliver_sm_resp...after a timeout they should assume the connection has been dropped, and try a different active connection. My point about enquire_link was that if a server does not see any activity (including enquire_link) within some timeout period, it should assume the connection is dropped. Using the SenderPort or ReceiverPort setting is an interesting idea. But I don't know whether this will help or not. For a connection that drops unexpectedly, it normally enters a TIMEOUT_WAIT state in the TCP stack, and another connection between the same set of ports is not allowed for 120 seconds. In your case, this might help the provider side recognize that there is a problem with the connection. But I would still expect error retries for 120 seconds after connection drop. -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 21 Registered: 04-2014 |
Dear Des, We have applied the ReceiverPort parameter, and the below is one of our connections configurations: [SMPP - 185.127.160.6:15031] RouteName=OM_P15031 SMPPVersion=v3.4 UserName=59 Password=m4d1n SenderAddressOverride=Yes Receive=No ReceiveMMS=No UseSSL=No LongSMSAlt=Yes SourceTON=1 SourceNPI=1 DestTON=1 DestNPI=1 WindowSize=5 KeepAlive=580 ReceiverPort=16031 aa} | ||||
| Ahmed Negm New member Username: Anegm Post Number: 22 Registered: 04-2014 |
then we get a TCP trace, and I didn't see my configured ports in the trace yet, I saw another ports. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5837 Registered: 08-2008 |
ReceiverPort is only used when there are separate sender & receiver connections. You are using transceiver mode (simultaneous send + receive). The SenderPort setting will apply to a transceiver connection . Note, however, that only one connection can use a port in this way, so multiple connections will not work, only one connection. -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 23 Registered: 04-2014 |
Do you mean with separate connection for send and receive to create the same connection twice in SMSC tab (one for MO and the other one is for MT)? In v2006, I see that I can set both senders in the same SMSC connection by assigning connection to have TX and RX. So, please clarify if I need to create the connection twice. If so, I will have the same issue again as I will establish 2 connections at least to the same SMSC. | ||||
| Ahmed Negm New member Username: Anegm Post Number: 24 Registered: 04-2014 |
Hi Des, We applied the same, but after sometime we got TimeOut error as we have reached the limit of allowed connections at the SP's side (~10-15 connections), although, I have one single RX and one single TX with ReceiverPort and SenderPort. I just need advice, if there's anything can we ask the SP about to get this issue done, I believe that there's something missing but we don't know what it is. | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5843 Registered: 08-2008 |
Hi Ahmed, Stop using the SenderPort/ReceiverPort settings. It was an interesting idea to try...but not one that I would have suggested. Overall, that setting is making things more confusing. Unfortunately, I have no other recommendations.
Whether or not enquire_link is sent is unimportant..it does not directly relate to the issue. They are free to send deliver_sm on any active connection. But in the most recent logs, I see NowSMS sending unbind...closing the connection with FIN...and they still try to send deliver_sm over the connection. That said, I have been reviewing trace_files.zip, and I am confused. I see multiple instances of NowSMS sending bind_transceiver...getting an OK response...then sending unbind immediately. This is not normal. The configuration interface will do this when you save/test a connection. (If the UI asks if you want to test a connection, I would suggest you answer no. If you are frequently stopping/starting the NowSMS service, do not do this.) -- Des NowSMS Support | ||||
| Ahmed Negm New member Username: Anegm Post Number: 27 Registered: 04-2014 |
Hi Des, For your last part about the not normal behavior of immediate unbinding after bind_transceiver OK. Our test session had the following steps: 1. Start the service 2. Click "Test" button in SMSC tab. 3. Sending MOs 4. Sending MTs 5. Click "Test" button in SMSC tab. 6. Finally Stop/Start the service, then close the tracing session. | ||||
| Ahmed Negm New member Username: Anegm Post Number: 28 Registered: 04-2014 |
Unbind occurs after clicking Test button in SMSC tab, is this wrong? | ||||
| Ahmed Negm New member Username: Anegm Post Number: 29 Registered: 04-2014 |
Unbind occurs after clicking Test button in SMSC tab, is this wrong? | ||||
| Des - NowSMS Support Board Administrator Username: Desosms Post Number: 5850 Registered: 08-2008 |
No, that is expected. (...and for most SMSCs, not a problem. In your case, I would recommend never using this button.) The "Test" button causes the server to unbind all active sessions for that connection. It then creates a new bind...reports an error if this fails...if no errors, it immediately unbinds. The server then creates new binds based on configuration settings. -- Des NowSMS Support | ||||
