Debug Packet drops between GCP and colocation servers

October 12, 2026

The short version

Production TCP connections to our colocation datacenter were timing out for 40–60 seconds, recovering, then failing again hours later. No alerts fired during most events. We found two completely separate problems sharing one interconnect:

A 1 Gbps attachment cap being exceeded by bursty traffic - fixed with a traffic shaper
Periodic L2 data-plane blackouts on the exchange path - BGP hold timer determined how long users felt them

We fixed the first one, watched anot]her blackout hit on an idle link, and had to start over.

The setup

Our GCP environment (europe-west1) connects to a colocation datacenter via Partner Interconnect through a public internet exchange. The VLAN attachment is provisioned at BPS_1G. All GCP→colo traffic routes through a single gateway VM acting as a WireGuard endpoint.

Problem 1: the invisible policer

GCP’s per-VM traffic metrics showed bursty egress - 60-second averages peaking at 700–790 Mbps, shaped like batch jobs running every few minutes.

BPS_1G means GCP enforces a 1 Gbps egress cap at its edge, regardless of the physical handoff speed (10 Gbps). When traffic exceeds 1 Gbps, GCP shapes it - which means silently dropping packets. Microbursts were crossing the cap, but we couldn’t see it in GCP’s metrics: the per-VM sent_bytes_count metric samples at a 60-second interval, which averages away any sub-second spike. A burst that saturates the link for 500ms looks like a modest blip at 60s granularity, and the associated drops are never surfaced at all - GCP has no dropped_packets metric for Partner Interconnect attachments. We only confirmed it by watching TX rates at 1-second resolution directly on the gateway VM, where TX would plateau at exactly ~1000 Mbps for 10-second stretches during bursts.

The ping trap. During these events, our ICMP monitors to the colo stayed green. Pings survived. A 64-byte ICMP packet sails through a policer that’s drowning in bulk TCP flows. Ping surviving during congestion tells you almost nothing about TCP health.

Fix for problem 1

cake (Common Applications Kept Enhanced) is a queuing discipline in the Linux tc stack that does fair queuing plus active queue management. We deployed it on the gateway VM’s outbound interface to rate-limit just under the 1 Gbps cap:

tc qdisc replace dev ens4 root cake bandwidth 950Mbit

TX peaks went from ~1910 Mbps to ~950 Mbps. Over-cap seconds dropped from ~886 per day to 0.

We shipped this as a fix. It was correct. It was not complete.

Problem 2: the blackouts we couldn’t explain

Two hours after deploying the shaper, a blackout hit while the link was nearly idle - peak 7 Mbps in the surrounding 2-minute window, shaper active, 0 over-cap seconds. Both the WireGuard tunnel ICMP probe and the underlay ICMP - a separate probe riding the raw interconnect path outside WireGuard - died simultaneously.

If this were a WireGuard problem, the underlay would stay up. If this were the gateway VM stalling (CPU never exceeded 26% softirq on any core), the underlay would stay up. This was not congestion.

Building the monitoring stack

We deployed coordinated monitoring on both sides simultaneously:

GCP side (gateway VM):

0.5s ICMP probes to the colo router, logged with timestamps
TCP connection tests every ~8s via nc (-w2 timeout), curl (—max-time 2), and telnet (no timeout) to port 22
GCP Packet Mirroring: all traffic from the gateway VM’s NIC mirrored to a collector VM via an internal load balancer, captured with tcpdump

Colo side:

Identical ICMP and TCP probes back toward GCP

The packet mirror captures at the VM’s NIC before GCP’s network stack - it proves whether packets actually leave the VM or whether the VM itself is the problem.

What the captures showed

Two distinct phases emerged from the packet captures:

# Phase 1: Full blackout (~3s) - ICMP and TCP both dark
01:47:03  GCP > colo: ICMP echo request  [NO REPLY]
01:47:04  GCP > colo:22: Flags [S]       [TCP SYN - NO SYN-ACK]
01:47:05  GCP > colo:22: Flags [S]       [SYN retransmit - NO SYN-ACK]

# Phase 2: ICMP recovers, TCP SYN still dark (~16s)
01:47:06  colo > GCP: ICMP echo reply    ← ICMP RESTORED
01:47:08  GCP > colo:22: Flags [S]       [SYN - NO SYN-ACK]
01:47:09  GCP > colo:22: Flags [S]       [retransmit +1s - NO SYN-ACK]
01:47:11  ...                             [retransmit +2s - NO SYN-ACK]
01:47:15  ...                             [retransmit +4s - NO SYN-ACK]

# During Phase 2: colo→GCP works fine
01:47:15  colo > GCP:22: Flags [S]       [colo-initiated SYN - succeeds]

# TCP restored after 4 retransmits
01:47:24  colo > GCP:22: Flags [S.]      ← SYN-ACK - TCP RESTORED

No RST, no ICMP unreachable - silent drops throughout.

The Phase 2 asymmetry was the clearest signal: connections initiated by the colo toward GCP completed normally during that 16-second window. Only GCP-initiated TCP SYNs dropped. Consistent with a forwarding table on the interconnect switch updating for one direction of traffic before the other - a MAC or ARP entry that sees colo-initiated frames and recovers, but hasn’t yet recovered the return path.

A dual-side simultaneous capture confirmed the failure was bidirectional at onset: both endpoints logged TCP SYN timeouts toward each other at the exact same timestamp.

The BGP smoking gun

We had been watching the wrong metric for BGP health.

GCP’s Cloud Router publishes session_up at a 60-second sampling interval. Our blackouts lasted 32–64 seconds. A BGP session that goes down and recovers within 60 seconds is invisible in that metric - session_up shows 1 throughout. We had been confident BGP never flapped. It was a metrics granularity artifact.

The actual sequence from BGP logs for one blackout:

T+0s - data plane dies, ICMP and TCP go dark
T+42s - colo BGP peer sends NOTIFICATION 4/0 HOLD_TIMER_EXPIRED
T+43s - BGP tears down, route withdrawn
T+46s - pings recover after BGP reconverges

BGP hold timer was set to 60 seconds. The data plane died and BGP held on for 42 seconds, retransmitting keepalives into a dead path before declaring it gone. The route withdrawal and re-advertisement is what ended the visible outage - not the interconnect recovering, but BGP reconverging and pushing a new route.

The outage length from a user perspective equals the BGP hold timer, not the actual path-down duration.

BFD would have cut this to ~5 seconds. We had BFD configured on the GCP Cloud Router side (1s detection intervals × 5 = 5s). But the colo had not enabled BFD on their end. GCP was sending BFD packets into a void - 35,000+ sent, 0 received. One-sided BFD does nothing.

Root cause

Problem 1 - Congestion: BPS_1G attachment policer + bursty gateway VM egress. Fixed by cake shaper at 950 Mbit.

Problem 2 - Periodic L2 outages: Data-plane blackouts on the exchange path, every 4–8 hours, 32–64 seconds each. Traffic-independent - hit at idle. Bidirectional at onset, silent drops, BGP hold timer determines user-visible outage length. Root cause is L2/forwarding on the exchange/colo switching layer - likely STP topology changes, port flaps, or MAC table issues. Escalated to the colo with exact timestamps and dual-side packet captures as evidence.

When you hit something similar

Ping is green but TCP is failing? Ping is not a TCP health check. ICMP sails through saturated policers and can recover before TCP SYNs do (Phase 2). Add an actual TCP connectivity monitor.

BGP dashboard says “up” but connections fail? Check the sampling interval of your BGP metric. GCP Cloud Router session_up at 60s will miss any flap shorter than that. Read BGP logs directly.

You fixed the obvious problem but outages continue? Verify the fix in isolation before closing. Two problems can share identical symptoms while being completely independent.

Trying to prove where a break is on an interconnect? GCP Packet Mirroring to a collector VM lets you confirm packets leave the VM’s NIC. If they leave but never arrive, the break is past GCP’s edge, not inside your VM or GCP’s internal network.

BFD enabled but detection is still slow? Verify your peer has enabled it. One-sided BFD is inert. Check packets sent vs. received - 0 received means it’s not active regardless of your local config.

Bursty traffic on a BPS_1G attachment? That tier is a policer on a 10G port. Bursts get silently dropped. Shape at source with cake or raise the attachment tier.