Intermitent connectivity problems with google services

Hello ​@Antant! Welcome to the KPN Community! 

It's unfortunate that you're experiencing issues reaching certain Google services. I'll be honest and say that my knowledge in this specific area is somewhat limited, but I'd be happy to think along with you to figure out what's causing this and how we can resolve it.

Could you also tell us a bit more about your home network setup? Are you using your own modem/router, and have you made any changes to the DNS servers you're using?

It's already great that you've done so much investigation yourself, and I think knowledgeable community members such as ​@TDN or ​@bnh may already be able to draw some preliminary conclusions from the information you've provided. However, having a bit more insight into your setup may help us narrow down where the issue could be originating.

Hopefully you'll be able to share some additional information with us. 🙂

Hi Arjan,

Thanks for getting back to me. Yes, I’m using my own router but the setup has been stable for nearly a decade now. 

I made changes to the DNS servers while trying to troubleshoot this issue. I was originally using the KPN servers (195.121.1.34 & 195.121.1.66) when the problems started. I’ve since tried google (8.8.8.8), quad9 (9.9.9.9) as well as my own unbound.

DNS was also my forst suspicion but I gave up blaming it when I realised that a) all of them return the same IP addresses and b) my monitoring tool shows errors even when it tries to reach the IP addresses directly, bypassing the DNS.

I have to say I’m really perplexed. The very suspiciously consistent outage durations of just around 15 minutes, tells me that it’s probably the side effect of some process.  

It’s hard to believe that it’s my end of the network when I can reach the other 7 of the 8 IP addresses when it happens. 

Also hard to believe that it’s some CI/CD process on the google side. Those don’t usually stop servers from responding to pings and outages like that usually make the news when they happen once, let alone this often.

Hard to believe that it’s the KPN connection to google. That would make the Dutch news just as fast.

This by the way, is what it looks like as I type.

So far, no other external testing tools have shown the outage and I have not seen anyone else reporting issues between KPN and google. I’d be willing to accept that I’m going mad but that does not explain the red bars either. :D

If I had the keys, the next thing I would check is what does my gateway (195.190.228.57) see at the time.

I look forward to any other ideas I can try.

Regards,

Antonios

I wonder if these are not normal outages from Google.

Normally I have uptime kuma and smokeping running here, and I don't see such outages (but they check on hostname, not directly on IP). Do you have a download for your config.yaml? I can run gatus (in docker) for a few days besides the other ones to see if I get the same.

thanks for thinking along ​@rvk01. Outages on the google side would definitely be a candidate. Scheduled service rollouts too. Network failovers anywhere along the path, they all fit the pattern. The possibilities are endless...

My understanding of the process is: User goes to www.goole.com → browser asks DNS for an IP → DNS gives back all 8 → browser picks one at random → during the 15 minute outage, there is 1 in 8 chance that the browser will pick the IP that it can’t reach and time out.

I put the config in a gist along with a sample compose file. The last entries in each group are indeed by hostname and they also show the problem, 1/8th of it.

Ok. Running.

The reason I was thinking about outages is because the direct IP scans show about 15 minutes of downtime while the host scans only show 1 minute down. Which makes me believe the DNS only shows IP’s that are up (or at least host picks the one that is up). And the TTL for Google service is 5 minutes.

thanks! look forward to hearing what you see on your end.

My guess is that the host scans show 1 minute because all scans are configured to run every minute so that would be the time between runs, and then pick another IP for the next run. 

I really hope that the issue does not show on your end. It would narrow down the scope to within KPN (including my network), behind the point where our connections converge on the way to google, on my side of the path.

I see your 142.251.152.119 endpoint had some troubles today. For me they didn't. The only hickup was yesterday when 142.251.150.119 had a spike but that was just for 50s. Your points show multiple outages of 14 minutes. 

~ $ traceroute 142.251.152.119
traceroute to 142.251.152.119 (142.251.152.119), 30 hops max, 60 byte packets
 1  mijnmodem.kpn (192.168.2.1)  0.644 ms  0.692 ms  0.754 ms
 2  static.kpn.net (195.190.228.11)  4.203 ms  4.273 ms  4.347 ms
 3  * * *
 4  * * *
 5  142.251.152.119 (142.251.152.119)  5.980 ms  6.246 ms  6.227 ms

Haha, yeah, I’ve fallen in such a trap too in the past. For me I have nftables setup with a monitor to check HTTP, SSH and other traffic. If something is fishy, that IP goes in the blocklist for 2 to 7 days (depending on the severity). I also had it happen to me that an IP got on that list by mistake. Always takes me a while to think about it, why doesn’t it work….

In your case, a simple single request which is dropped or delayed by Google (which can happen once in a while) will result in a complete shutdown of that IP for 15 min. Maybe building in a number of (re)tries would help there 😁

Glad you figured out where the problem lies 😀

yes, I’m trying 2 things now.

Multiple “stages” of blacklist. If in stage1, add to stage2, if in stage2 add to stage3 etc and eventually block stageX.

A Whitelist, or more accurately, a blacklist bypass list. Also dynamically generated by ip addresses requested from the LAN side that are in the blacklist.

The 2 can live together but if I had to pick, I think I prefer the 2nd option. It feels a bit like a fix to a problem I create in the first place but that actually applies to my whole network so it doesn’t count :D

The multiple stages only mitigate the problem. I could set up 8 stages and 8 rogue packets could still break google. The bypass should in theory eliminate the problem and therefore make the multiple stages obsolete. That means fewer/simpler rules and a potential attacker blacklisted from the 1st packet…

The logs and the monitoring page over the next few days will give me an idea which way to go. My better half will most likely cast the deciding vote :D 

It's great that you and ​@rvk01 were able to identify the root cause together! Thanks a lot for your help rvk01!

The silver lining is that you've learned something about firewalls and the Google blocks that can sometimes occur. So in the end, all that troubleshooting did pay off.