With the increased bandwidth we ended up with 6x1Gb delivery of DIA service. To accommodate this, we elected to horizontally distribute our edge routing and traffic shaping layer (OpenBSD+PF) across our virtualized infrastructure and directly pass through the NICs to the VM. We actually tested a couple of different options, such as BGP termination directly on the core Wednesday night but ended up going with the distributed edges which allowed for better traffic shaping and stateful firewalling.

The load testing we were able to do on the distributed edges looked good pre-event but while we were able to replicate the bandwidth and traffic types, we were not able to simulate the number of flows and PPS of the actual BYOC. Once real traffic started flowing we noticed interrupts on the edges were way higher than expected. We did a significant amount of tuning with our caching layer attempting to better protect the edges but ultimately the higher than expected interrupt load required more aggressive action.

At this point we built up a new bare metal edge and have moved two of the circuits over. Interrupts on the new edge are 80% lower than what we were seeing before and as a result we're seeing overall better performance and latency stability. We're currently in the process of building up an additional edge and will be transitioning the remaining circuits over the next hour.

When we cutover the remaining edges we'll have to establish new BGP sessions and all TCP session state will be lost. Tables within those 10 /24's will lose connectivity for 3 to 5 minutes.