IPsec site to site keeps dropping

This problem has been happening since we signed up for service about a year ago. Basically, at random, my connection will stop accepting udp port 500 traffic. It will work fine for a week or even a month, then it will happen a few times in one day. We are using the same site to site config on Fortigates between multiple other sites, some of them also on Comcast and we do not have this problem. I confirmed with Fortinet when the problem was happening that traffic was going out from one side but not being recieved at the site in question. A modem reboot always fixes the problem. The other strange thing is that rebooting the firewall also fixes it. It almost seems as if it's something to do with the connection between the two devices. At one point we replaced the ethernet cable to the firewall and thought that might have fixed the problem. But it started happening again a month or so later. Any ideas? Comcast's support has been worthless since every time I call in they just want to reboot the modem and since it is currently in a working state, they don't think anything is wrong.