Forum Settings
       
Reply To Thread

DAOC TCP has bugs, causes needless link deaths.Follow

#1 Mar 31 2004 at 7:58 PM Rating: Decent
I've tried support@DAOC, the feedback forms on herald, and am running out of ideas as to where to send this information to get a response.

Essentially while trying to diagnose whether my firewall or my ISP were dropping data causing my link deaths, I put a lan analyzer on either side of my firewall while running DAOC. Indeed I get occasional dropped packets coming from my ISP(once every few minute or so). What is disturbing is that a single dropped packet on the TCP socket DAOC uses for part of its communication is sufficient to cause a link death. The DAOC client appears to recognize the missing packet from the server, and sets its TCP ack to point to the missing packet, thereby requesting the server retransmit it. The server responds to each such request with an acknowledgement making it clear that the request was received, but does NOT resend the requested packet. This is bizarre because this retransmit behavior should be part of the operating system on the server, and not even part of the DAOC server application itself. After 30 seconds of requesting a retransmit, the client eventually closes the socket and exits to the link death screen.

I've repeated the experiment with the lan analyzer running 5 times now, and every time is the same. Despite the fact that client and server are still exchanging messages, the client has to exit 30 seconds after a packet from the server is dropped because the server fails to restransmit the missing packet.

This is absurd. This should not happen. Mythic should be able to fix, but I can not figure where to go with the information to get something done about it. Any other internet application can handle missed packets like this so my ISP has no motivation to fix the dropped packets. But even if they fixed it I would not expect any internet link to be completely without dropped packets.

This is _so_ frustrating. I have multiple sets of lan logs I could give to developers to help diagnose problem, and can't even get a response.
#2 Apr 02 2004 at 5:32 AM Rating: Decent
*
139 posts
I very rarely go LD while playing DAOC and I haven't heard many people complain of frequent LDs, so it may be a problem on your end. From the camelotherald support site about frequent LDs:

I am constantly going link dead within Dark Age of Camelot. I am using a DSL or cable modem connection. Could my network card have anything to do with this?

For Windows 2000/XP users:

Right-click My Computer and select Properties. Next, click on the HARDWARE tab and the click on the DEVICE MANAGER option. Under the section for NETWORK ADAPTERS, right click your network card and select Properties. Under the advanced TAB under MEDIA TYPE, set the value option on the right to 10mb half duplex mode.

For Windows 9x/ME users:

Right-click My Computer and select Properties. Next, click on the DEVICE MANAGER option. Under the section for NETWORK ADAPTERS, right-click your network card and select properties. Under the advanced TAB under MEDIA TYPE, set the value option on the right to 10mb half duplex mode.

Also from that site:

Please note: Mythic Entertainment will not be able to assist with the setup for users running behind a PROXY server or any home network setup. Please contact your ISP, computer manufacturer or networking hardware manufacturer for further assistance on configuring Dark Age of Camelot to run through your PROXY server or Home network.

All UDP ports 1024->65535
TCP ports 1280, 10500 and 10622

Make sure to open 2-way traffic to the following class C:

208.254.16.0 / 255.255.255.0 (netmask)
a.k.a.
208.254.16.0 / 24 (universal)

So, basically, if it's a problem with the way you're set up, you're on your own. You might want to read more articles in the support section.

You could also check message boards at other sites like http://vnboards.ign.com/Tech_Tavern/b20934/ I don't doubt others have run into the problem.
#3 Apr 02 2004 at 4:05 PM Rating: Decent
I know my ISP has 400 users on one cable. Probably with one router at the head of the cable. I have every reason to believe that my LD problems are frequent due to that.

This doesn't change the fact that a single packet, packet loss should not cause an application using TCP to fail, because TCP has built in recovery.

I've been working this problem for a month and have indeed looked at the tech support articles on the herald support page.

1) Its not the firewall. Still have the problem if bypass the firewall entirely. Can see that the packet was lost from the server size of the firewall(between firewall and modem), indicating it was before it got to firewall.

2) Tried the 10/100 half/full duplex change and makes no difference. Lan logs indicating the loss is detectable on other side of firewall on inbound packets would seem to confirm that the connection from PC to firewall is not the issue.

The problem very much seems to be the server is not retransmitting a lost packet when the client asks for it. Despite every evidence that the server did get the request for the lost packet.

Only parameters I cand find that would affect this would be server side parameters. Some O/S's allow tweak of the retransmit timeout and of the number of requests required prior to retransmitting a packet prior to the timeout. Both of these parameters would have to have been set absurdly large for the server to not retransmit the packet after 30 seoncds of requests at around 2 requests per second. The retransmit timeout should be computed automatically from the estimated round-trip time. I suppose there is could be some oddity causing the round-trip time to be miscomputed so that the server thinks it needs to wait absurdly long time, but that alone wouldn't change the fact that server should retransmit lost packets if more than three requests for the same lost packet are received.

I can't find anything that I could change on my side that would cause the server to be more likely to retransmit the dropped packet.



#4 Apr 02 2004 at 9:01 PM Rating: Decent
*
139 posts
I'm not a techie so don't know where to go from here, however, try the tech tavern at the ign boards (link in previous post). Quiet as it may be kept, I don't doubt that Mythic techies drop in now and again to address issues that Mythic doesn't officially address.

If possible, try another ISP (even dial-up) and see if there is improvement. Try using your ISP at off-peak hours to see if there is improvement. If so, I'd start yelling at my ISP.

#5 Apr 03 2004 at 7:47 PM Rating: Decent
*
130 posts
I have been playing DAoC for over 2 years, and I have gone LD maybe a total of 10 times. If your going LD that often it's a problem with your ISP or settings.
#6 Apr 03 2004 at 11:20 PM Rating: Decent
Well my ISP has apparently finally (after month of complaining) fixes something and according to my lan analyzer there are now _ZERO_ dropped packets.

I still don't think a perfect _ZERO_ dropped packet link should be required to play DAOC, and Mythic is probably loosing customers (ones that never play for more than short time because get kicked off a few times) because they broke the TCP internet protocol's normal ability to recover from single dropped packets.

Oh well, hopefully my ISP will remain 100% and I will no longer have any ability or need to test such issues. I have my doubts though.
#7 Apr 05 2004 at 6:13 AM Rating: Decent
*
139 posts
I'm sure everyone gets dropped packets now and then without going LD - that's part of the nature of the internet, however, as your ISP has "finally fixed something" my guess is that it was doing more than just dropping an errant packet here and there. Maybe it only reported one dropped packet when it had actually dropped a bunch of them? Or, it wasn't accepting retrieved packets. Whatever the problem, sounds like it was the ISP's fault.

Glad to hear you got a happy resolution! Hope you enjoy the game.
#8 Apr 05 2004 at 9:42 PM Rating: Decent
Thanks, Yes had been enjoying for 44 levels before troubles. Hope to enjoy for at least 6 more <grin>.

I'm trying to get accross to as many as possible that it only takes a single dropped packet. I logged the data with an external lan analyzer (ethereal on a linux box) and can see exactly what happend 5 different sessions. The TCP header has sequence numbers, and the IP header has a sequential message ident field and both of these fields show that a single message was dropped coming from the server, and the server refused to retransmit the packet. My ISP would drop a packet about once ever 2 to 15 minutes, but only single packet drops.

Yes you shouldn't have to have a perfect link, but I've still be running lan analyzer even when I was playing for hours without LD and analysis of the files produced shows that there were _ZERO_ dropped packets. Script analysis of the files showed a few out of order receptions, but none of these were long enough delayed to have been retransmits. Most appared to be result of fragmented packets where the first in pair was MSS size and received after the second.

Try it your self, all you need is an extra computer running linux, (and ethereal), a couple of spare network cards, and tools to build your own ethernet cable with spliced wires so the linux box can listen in. (Or a older ethernet hub that doesn't attempt to isolate ports, but I had a hard time finding one so made cable instead.)

Note I'm only analyzing the TCP packets, dropped UDP packets areharder to detect due to lack of TCP style sequence numbers. But UDP connections won't hang up if UDP packet is lost and not retransmitted like a TCP one will, and it was TCP packet loss that instigated LD every time I've seen it.

#9 Apr 08 2004 at 8:28 PM Rating: Decent
ouple pieces of additional information in case someone is actually trying to solve the problem:

1)
My ISP admits they have a minor packet loss problem. Out of 400 users on this cable line, there are 5 complaints. 2 of those are people trying to maintain VPN tunnels, and 3 of them are people trying to play DAOC. Hmm, either speaks to popularity of DAOC, or to it's fragility to packet loss.


2)
In analyzing lan logs I have indeed seen one case where the TCP retransmit recovery functioned correctly and a dropped packet from the server was retransmitted by the server, and the game proceeded for a consderable length of time(several minutes) untill another dropped packet failed to be retransmitted, and resulted in link death after the typical 30 seconds of the client requesting the retransmit.



#10 Apr 15 2004 at 10:25 PM Rating: Decent
Thanks to some very astute help on ign boards, the problem has been identified. It has not yet been fixed, but is only a matter of time. For benefit of anyone with a similar "link death" during which you can't chat, interact with npcs, use styles, etc, but can see mobs and toons moving about, see damage taken by mobes and toons change, and for 30 seconds or so other various indications that you are about to die even though still connected.

The problem is a subtle router setup error on the part of the ISP. There is a cisco recomendation that routers be set up to block 92 byte ICMP messages because this size message was used in an internet worm a couple years ago, that would cripple networks with excessive traffic. Some poeple (my ISP included) misunderstood this rule and instead of blocking 92 byte ICMP messages blocked all 92 byte message. This mostly works as 92 bytes is a fairly small IP packet, but the DAOC server often sends messages this size on its TCP socket, and when they get blocked it may well try to retransmit, but then that gets blocked as well. Since all retransmits of the 92 byte message will be blocked the TCP socket cannot recover. Even so they continue to exchange data on UDP sockets, as well one-way communication to the server on the TCP socket.

Receiving a /send text of 21 characters from another toon in game was able to reliably recreate this scenario. (Size of fatal chat text depends on who sending in a manner I haven't tried to determine.) Also can test for this particular problem out of game by creating a 52 byte file and ftping it through your ISP's routers. I can reliable ftp get a 51 and 53 byte file, but the 52 byte file requres a 92 byte message to send and will reliably hang.

FWIW, hopefully this will help others.
#11 Apr 16 2004 at 6:55 AM Rating: Decent
*
139 posts
That's amazing! I'm glad your persistence has paid off. I'll try to remember that if I ever run into someone else with the same problem. I hope your ISP sorts out the problem. If not, I guess you'd better switch to a different ISP :)
#12 Apr 16 2004 at 9:30 AM Rating: Decent
35 posts
That's impressive detective work. So does the fix require the cooperation of every ISP, or is Mythic going to change their code to never send 92-byte packets? :)
Reply To Thread

Colors Smileys Quote OriginalQuote Checked Help

 

Recent Visitors: 6 All times are in CST
Anonymous Guests (6)