packet loss, high latency, and garbled VoIP

Hello AStephens and welcome,

Do you use straight up Netgear 3000 (NG3K) DHCP or StaticIP address for both your VoIP Server and phones ? If you are using DHCP for either, you MUST log into the NG3K and go to LAN to set your LeaseTime=FOREVER. Also, make sure that your LAN DNSs pri=75.75.75.75 and sec=75.75.76.76.  If your VoIP server has built=in load balancing you will need to disable this for sure. 

Look forward to hearing from you.

It's a static IP, of course.  The phones and VoIP server are both local (not a cloud-based VoIP).

No problems with DNS.

The smokeping results clearly shows the packet loss and inconsistent latency. 

Is it a problem with the rack, the line, or the local modem?  Is an on-site evaluation needed to do testing?  We need this resolved ASAP.

The Netgear CG3000DCR is a great cable modem except that it has a known bug with SIP and ALG and should NEVER be used on a cable line where your running VoIP.  This is an issue known to Comcast support and if you call into support and ask the support tech to ask 2nd level support about it they will concur.

You can substitute either an SMC router or a Cisco BWG.

Sadly, the problem continues.  I've had a tech out four times so far.  A resistor was added to decrease the download gain.  Then the connection went down completely.  Then the modem was changed to an SMC.  No help.  Maintanance has worked on the line once, which restored the connection to its "bad" state... at least it's connected.

Another tech is scheduled for this morning.

You can see things go from bad to worse, back to bad on the 10-day graph.  After the first tech visit, the connection went down for most of 2.5 days!

http://www.dslreports.com/smokeping?target=bc338a83bc43bf79f314a7a442c2cbfa.Kansas

Are you graphing bandwidth also?  If not, I think you should be - there might be a coorelation.

I agree that some of the jumps in latency could be associated with bandwidth saturation.  However, the analysis so far does not show that happening in this case.

Unfortunately, we do not have a mechanism in place to log bandwidth at a resolution that can be compared to the smokeping graph over the long-term.  There is logging in place, but it's aggregate data by *day*.  There is also cumulative logging by device.

The only high-resolution bandwidth graph available is a "live" monitor (~5min history, graphed) in the router.  I have kept an eye on that in conjunction with smokeping.  There is no correlation.  When smokeping shows packets dropped and/or heavily variable latency, our bandwidth usage is very low.  

Some automated backup routines push data offsite to two self-managed locations (not "cloud").  I have compared the graph to the backup log (start/stop times) and see no obvious correlation.  A smokeping for the main backup site is rock-steady.  However, I still agree that some of the overnight 0-2hr blips could be backups running.  

FYI, I was contacted by Comcast yesterday and found out that Maintenance found a line with a cracked sheathing along the road.  The Utilities group is to be scheduled to repair/replace the line.  The schedule is TBD.  (Hopefully it won't take too long, as internet is important for business.)

Once the line is repaired, I plan to continue monitoring the connection and to eventually post an updated smokeping graph for reference.  (I really wish I had logged a smokeping from a few months ago, so I knew what "normal" was.)

Thank you for your time and input/ideas.  

We have been experiency high latency on several traceroutes, always hanging up at ar01.area4.il.comcast.net.  If we run a traceroute to our VOIP voice server it hangs up at this part of the comcast network EVERY TIME, with high packet loss.   Trace routes and MRT's to other addresses and sites show it hangs up there bad some times, but when tracing the route to our VOIP server it's almost as if some kind of new traffic management software is mis identifying our legitmate VOIP traffic as P2P file sharing?  It's strange because the connection is otherwise fine.   I am beginning to wonder if comcast is slowing down competing VOIP traffic? I really wish Comcast would allow QOS management for customers on their side, so we could prioritize VOIP traffic in the best way.  Then they would loose the only advantage to paying a lot more just to have comcast VOIP lines. We do have the SMC router.  We never have high latency or packet loss on hops just before or just after Comcast modem.  Delays always on comcast side. Problem is, comcast does not allow customers to file complaintes with "tier 2" or internet "maintenance" without a "truck roll first".  Even when they agree the evidence shows the problem is not on premisis, comcast says they have to send a truck because, "that's the policy, that's the way we do it".  Makes no sense to send techs and trucks for situations where all equipment on site is peforming as it should.  I spent all day hearing all kinds of crazy, contridictory information from comcasts customer service, tech support, and supervisors. Finally at 9PM after a day of this nonsense someone finally was able to file a ticket with dispatch for our area.  Problem was resolved this morning one way or another.