Skip to content
io's profile

New problem solver

 • 

4 Messages

Tuesday, September 29th, 2020 3:00 PM

jitter, latency spikes, packet loss

Small office with a few employees on 35Mb/5Mb in Albuquerque with several years of mostly trouble-free internet service that's been stable enough for using a 3rd party voip service hosted out of state.  Around August 26-27th an outage affected the area that looked like this: 

 

forum01.pngComcast technicians performed repair work following a support call and service was restored.  Since that outage every day there are latency spikes or packet loss between roughly 8am-3pm which looks like this:

 

forum02.png

 

 

forum05.png

 

 

 

The 3rd party voip service has been unusable from this location for the past month due to the jitter and packet loss impacting call quality.  Ping to the Comcast .1 ip that is the gateway for the Comcast modem at the office always looks stable but this is an example of ping to the modem ip or its corresponding wan ip during office hours tested from an isp out of state:

 

 

Reply from *.*.*.*: bytes=32 time=76ms TTL=55
Reply from *.*.*.*: bytes=32 time=138ms TTL=55
Reply from *.*.*.*: bytes=32 time=253ms TTL=55
Reply from *.*.*.*: bytes=32 time=212ms TTL=55
Reply from *.*.*.*: bytes=32 time=96ms TTL=55
Reply from *.*.*.*: bytes=32 time=98ms TTL=55
Reply from *.*.*.*: bytes=32 time=91ms TTL=55
Reply from *.*.*.*: bytes=32 time=341ms TTL=55
Reply from *.*.*.*: bytes=32 time=108ms TTL=55
Reply from *.*.*.*: bytes=32 time=73ms TTL=55
Reply from *.*.*.*: bytes=32 time=65ms TTL=55
Reply from *.*.*.*: bytes=32 time=78ms TTL=55
Reply from *.*.*.*: bytes=32 time=78ms TTL=55
Reply from *.*.*.*: bytes=32 time=279ms TTL=55
Reply from *.*.*.*: bytes=32 time=63ms TTL=55
Reply from *.*.*.*: bytes=32 time=85ms TTL=55
Reply from *.*.*.*: bytes=32 time=143ms TTL=55
Reply from *.*.*.*: bytes=32 time=72ms TTL=55
Reply from *.*.*.*: bytes=32 time=201ms TTL=55
Reply from *.*.*.*: bytes=32 time=135ms TTL=55
Reply from *.*.*.*: bytes=32 time=369ms TTL=55
Reply from *.*.*.*: bytes=32 time=133ms TTL=55
Reply from *.*.*.*: bytes=32 time=186ms TTL=55
Reply from *.*.*.*: bytes=32 time=134ms TTL=55
Reply from *.*.*.*: bytes=32 time=275ms TTL=55
Request timed out.
Reply from *.*.*.*: bytes=32 time=148ms TTL=55
Request timed out.
Reply from *.*.*.*: bytes=32 time=65ms TTL=55
Reply from *.*.*.*: bytes=32 time=120ms TTL=55
Reply from *.*.*.*: bytes=32 time=97ms TTL=55
Reply from *.*.*.*: bytes=32 time=127ms TTL=55
Reply from *.*.*.*: bytes=32 time=56ms TTL=55
Reply from *.*.*.*: bytes=32 time=89ms TTL=55
Reply from *.*.*.*: bytes=32 time=134ms TTL=55
Reply from *.*.*.*: bytes=32 time=80ms TTL=55
Reply from *.*.*.*: bytes=32 time=178ms TTL=55
Reply from *.*.*.*: bytes=32 time=169ms TTL=55
Reply from *.*.*.*: bytes=32 time=87ms TTL=55
Reply from *.*.*.*: bytes=32 time=60ms TTL=55
Reply from *.*.*.*: bytes=32 time=58ms TTL=55

 

 

 

 

Normally, high LAN << -- >> WAN utilization would be suspected as the cause of latency like this but WAN + tunnel interface utilization on the firewall in snmp doesn't seem excessive or consistently at capacity and netflow hasn't shown high traffic:

forum03.png

 

 

This speed test screenshot shows full capacity.  Latency does ramp up above the usual bad numbers while this is run which is expected.  The speed test down/up results also decrease when traffic shaping is in effect on the firewall to confirm that's effective.  The comcast account web portal displays the service as "business internet - starter" "25Mb/5Mb":

forum06.png

 

 

The problem persists with or without traffic shaping down to ~20Mb/3Mb in the firewall config behind the modem.  The problem persists with everything disconnected from the Comcast modem during testing.  The troubleshooting process starts from scratch on each inbound support call so to keep those conversations brief the modem has been physically unplugged from the firewall on a few calls and power cycled followed by monitoring response time to demonstrate the latency spikes continue on the Comcast modem ip / wan ip without the office connected.  The goal of that test is to eliminate office traffic from consideration.  A Comcast point of escalation observed this and agreed there is a problem not related to the LAN or customer equipment and arranged tech dispatch that didn't resolve the problem.

 

Support calls and technician visits over the past month have had different findings:  First tier support reps ask for the number of devices on the LAN and then recommend a service upgrade as the solution.  It's hard to prove or disprove that would fix the problem without upgrading so I've asked why the starter tier by default with nothing connected is being affected by the excessive jitter and there isn't an answer.  One rep felt that high utilization at a comcast node hitting 90-97% is the problem and they escalated for that reason but a point of escalation that followed up thought that is not actually a problem.  Bad static ip build causing an issue with the headend was another idea but a static rebuild had no effect.  Visiting technicians did outside repair or drop line replacement and also replaced the original Netgear CG3000DCR modem with a Cisco DPC3939b after a few visits which is bridged with wifi off.  Most recent technician visit last week left employees with the impression there would be a followup bucket truck coming out last Saturday for more work but they didn't spot anyone and the tickets have been closed since then.  

 

An escalation technician at some level above first tier who seemed knowledgable had been making outbound calls to us for testing last week but I didn't get their contact info.  A support rep on an inbound call this morning advised tickets were closed and no work was in progress and they opened another new ticket which may start this from scratch again. 

 

Any suggestions?

Accepted Solution

Gold Problem solver

 • 

421 Messages

4 years ago

Thank you for getting back to us and letting us know about this. I'd be glad to follow up on the progress of the node split. Please click on my handle (Comcast_Gina) and send a private message with your name, the business name, the complete service address (including city, state, ZIP, suite number, etc), and the phone or account number, and any pertinent details (for example, the ticket number) so we can help. 

Accepted Solution

New problem solver

 • 

4 Messages

4 years ago

Problem resolved after Comcast node split completion a couple weeks ago.  

Official Employee

 • 

276 Messages

4 years ago

Hello, how are you? I hope my reply finds you at a good time and your Tuesday is going well. I am sorry to hear that you have experienced ongoing trouble since a service interruption. I know this has been a huge inconvenience for your business, especially since your VoIP service is not operating ideally.

This is always the right place to talk and get support because our dedicated team will always do all we can to provide a great experience, even when challenging issues like this are taking place. I promise I will do all I can to help with this connection issue. Are you able to send a private message so we can take a closer look at the account? This will allow us to get more details without speculating about what's going on. Please include your name exactly as it appears on your account, your full-service address, and your account number? (or phone number)

New problem solver

 • 

4 Messages

4 years ago

Messaged you 09-29-2020 and got a call from support later that day.  They found signal issues in the area and also opened a ticket with xoc to request a node split due to high utilization.  Modem as of today:

 

forum07.png

New problem solver

 • 

4 Messages

4 years ago

done. 

New Contributor

 • 

3 Messages

4 years ago

Hey there, I'm a Business customer in Los Alamos (not too far out from Albuquerque) and I've been having similar issues since around the time you reported. Did you have any luck working with the team? I had an outage as recently as yesterday afternoon. It's typically 8am-3pm on weekdays and presents either as significant upstream signal degradation or outright disconnections, but short enough that the automated system typically won't register them.

Problem solver

 • 

348 Messages

4 years ago

Hi there, thanks so much for taking the time out to reach out to the Digital Care Team here through Twitter. I am so sorry to see that you are having some issues with your service as we do understand having reliable internet is super important. Do not worry, you have reached an amazing team to help. Can you please send us a private message with your name, the full address, and the phone or account number?