Losing connectivity every 2 hours

Hello and Happy Thursday! I hope you’re having a great week! I also appreciate you taking a moment out of your day to reach out to us on our Business Community Forums! You’ve reached the best place and perfect person to help with virtually all of your account needs! Going forward, I'm truly sorry to hear about the IPv6 packet loss you're experiencing with your services as it's definitely not what we want for you! You mentioned there could be some impact from the WAN DHCP lease expiration, are you able to set up a static IP for this specific port to see if that helps? If not, we can also take a closer look on the backend to see what other options we have to help. Please send us a private message by clicking my name "ComcastJosephW" then select "Send a Message" on the right side with your name and service address to get started. Thanks again for both your time and patience!

I don't control the WAN DHCP expiration.  This is an address that the modem retrieves from Comcast's own network automatically.

Everything within my internal network uses leases that are handed out by my own router behind the modem.  The router itself has a static IP address.  None of this has changed recently.

On further inspection, I am actually losing IPv6 connectivity every hour (not every two hours).  Rebooting the modem restores IPv6 connectivity for an hour and then the modem stops routing traffic again.

So my theory about the problem being related to 2 hour WAN DHCP leases may be incorrect.  (Renewing a lease should not break connectivity but the modem's firmware is buggy enough that anything could be the culprit.)

Regardless, looking at my logs makes it very obvious that something changed upstream around 4:53 pm PST on 2021/2/24 and I have lost IPv6 connectivity every hour since then.

I don't think the hardware is broken.  The Comcast modem's firewall was already dropping IPv6 traffic every couple of days, now it's just happening every hour very predictably.

As for affected devices...

- The modem itself can ping IPv6 hosts on the Internet.  I tested this using the modem's built-in diagnostic features.

- The router behind the modem can ping IPv6 hosts on the Internet.  I tested this using the router's built-in diagnostic features.

- The devices behind the router can ping IPv6 hosts on the Internet, but only for one hour, after which time the modem's firewall drops all inbound IPv6 packets sent to them until I reboot the modem.  I tested this by running pings and connecting to IPv6 sites on those devices.

As far as I can tell, ONLY rebooting the modem fixes the problem (temporarily).  Rebooting the router or devices behind the router has no effect.

This appears to be an issue with the Comcast modem's firewall and its support for IPv6 prefix delegation, as I previously described here: https://forums.businesshelp.comcast.com/conversations/ipv6/ipv6-prefix-delegation-workaround-and-lingering-reliability-issues/5fe0a64fc5375f08cd9aba6d

I spent many hours with customer support on this issue before, got absolutely nowhere, and gave up.

What I don't understand is why as of yesterday the Comcast modem is getting into a bad state in a span of one hour whereas previously it routed IPv6 packets for about two days between reboots.  So now it's pretty unusable.

This problem should be easy for anyone at Comcast to replicate in the lab.  It just takes a modem, a router configured for prefix delegation, a device downstream from the router, and a stopwatch.

I haven't changed anything on my modem, router, or network topology in at least a few weeks.

What do you mean by "remove and replace the IP"?  I have static IPv4 and IPv6 addresses.  As far as I can tell, they're still valid.

Could it be that Comcast remotely pushed a configuration change or firmware update to my modem within the last 48 hours which affected IPv6 routing or firewall behavior?

Recent network changes:

- Several months ago, I set up monitoring tools and wrote some scripts to automatically reboot the Comcast modem when it stops routing IPv6 traffic.

- A few weeks ago, I tried modifying the DHCP lease expiration timeout on the Comcast modem to see if it would have an effect on IPv6 routing issues.  It did not have an effect, so I reverted the change.  The modem continued to have IPv6 routing issues every two days or so.

- Yesterday, I noticed that the modem was losing IPv6 routing and being rebooted every hour as of that afternoon.  I posted to this forum but didn't make any changes.

- Today, after seeing that the modem was still losing IPv6 routing every hour, I temporarily disabled my script so it wouldn't reboot the modem anymore because it was too much of a service interruption.  And since I had nothing better to do, I backed up settings, upgraded my router firmware, and performed other routine maintenance.  IPv6 routing is still broken.

Summary: Same modem, same router, same topology, more frequent outages as of yesterday.

I haven't tried bypassing the router.  However, the router itself is able to ping IPv6 hosts even when downstream hosts can't so I suspect that if I plugged a computer into the Comcast modem directly, then it would work.  However, my network topology is complex enough to require a router (multiple vlans, firewall, services) so I can't just plug everything into the modem.

My guess is that devices plugged directly into the Comcast modem are using link-local addressing to reach the gateway and the gateway's routing tables handle link-local routing fine but not prefix delegation to devices further downstream.

When I first reported this issue to Comcast last year, I used a packet sniffer and could see the Comcast modem dropping inbound IPv6 packets instead of forwarding them to the router.

No, the router is fine.  Like I said, I ran packet sniffers and I could see that after some period of time the modem simply stops forwarding downstream traffic to the router.

How else would you explain that my IPv6 routing issue goes away when I reboot the modem?  ONLY when I reboot the modem!  Touching the router makes no difference at all.

Similarly, I made no changes to my network at all and then suddenly IPv6 routing issues start happening every hour instead of every two days.  There's only one piece of equipment in my network whose configuration I don't completely control and that's the modem (and whatever Comcast infrastructure it talks to).  How else would you explain the change in frequency of my outages?

So as far as I can tell, the problem isn't beyond the demarcation point, it's right on the demarcation point.  Lots of folks have reported similar problems in this forum.  Isn't it time to stop punting these problems onto customers and instead start investigating their common cause?

I am having the same exact issues as @jeffb.  After working with no issues for almost three years, this started happening to me about two weeks ago, although it is occurring about every 24 hours or so. I have not had over 24 hours of stable IPv6 functionality for going on three weeks now. I have escalated to tier two, and they have yet to be able to solve the problem. Like the others, traffic just stops moving (timing out). I am just fetching a prefix and default route, and DHCPv6 seems to be working properly. I can ping both the Comcast the local-link and default route GUI addresses. Like @jeffb I seem to be able to ping and traceroute to nodes on the Internet when logged into the Comcast modem.

Nothing on my router has changed, and once it stops working, no configuration modification change fixes it, including setting up static routes. I have rebooted my router hundreds of times, each time it gets a prefix and default route... but traffic "times out". My nodes believe they have IPv6 connectivity, but they don't.  The only thing that gets traffic moving again is to reboot the Comcast Modem. We generally move a lot of IPv6 traffic and this is not good.

IPv6 traffic stopped again last night at a little after 10:30p PDT. I did not reboot my modem this time and have re-routed traffic so that support might be able to look at the modem/head-end current state and try to fix this.

-Scott

Early last week we had our Modem replaced with the latest Cisco modem/router, running the latest software (Cisco model DPC3941B). Exactly the same symptoms. My modem firewall is: "Disable entire firewall", yet I have similar firewall log messages as @jeffb : 

The dates on the above logs are "later" than when the current outage began (2012/5/21 22:40:00) and continuing to increament, leading me to believe that packets are possible actively being blocked by the modem, even-though the firewall should be totally disabled.

-Scott 

The previous modem looked exactly the same as the current one with the exception that it said "Technicolor" on the rear instead of "Cisco". I asked the repair tech if I could get a different model, and they told me this was all they had.

At this point I will absolutely try a different Gateway model. It seems that the prefix we receive might be dependent on the Gateways MAC interface, so every time it is swapped out, I have some extensive re-addressing to deal with... but....if that is what it takes to fix this issue, so be it.

I have including pictures of the current Gateway.

By the way, I added a second independent test router to the Gateway. Since I have static /29, it was easy.  I enabled IPv6 on this router and fetched a unique prefix. This router is having the same issues communicating with IPv6 as the other. 

Well, I am still down and not sure about if I can get a new modem or not. I followed the chat instructions and still waiting for a reply. 

It seems like you are saying we should try a different type of modem, but there is a chance there is possibly no new modem available, in which case I will still be down with the problem unresolved...