Some thoughts on CL outage, sorry @ioshints too lazy to blog this right now 🙂
First problem was input validation. It shouldn't have been possible to enter wildcards, but the validation failed (buggy code). It would make sense to add more logic here... 1/x
First problem was input validation. It shouldn't have been possible to enter wildcards, but the validation failed (buggy code). It would make sense to add more logic here... 1/x
It shouldn't be allowed to filter traffic belong to CL infra, BGP, ISIS, loopbacks, management etc etc...
Second problem was this was implemented without running tests (from what I can tell). The rule could have been tested on a virtual device first. CP is easy to simulate. 2/x
Second problem was this was implemented without running tests (from what I can tell). The rule could have been tested on a virtual device first. CP is easy to simulate. 2/x
The fault should have been caught in these tests and and the rules should not have made it to production.
Furthermore, there should be a ruleset, a safety net, of rules you can't override. You shouldn't be able to filter out traffic from the router itself. Think CP. 3/x
Furthermore, there should be a ruleset, a safety net, of rules you can't override. You shouldn't be able to filter out traffic from the router itself. Think CP. 3/x
Having a global FlowSpec implementation seems VERY risky. There should be interfaces where you don't implement FlowSpec. It should be used for Edge, not Core. From what I can tell, far too many interfaces were affected.
4/x
4/x
If customer advertises something to you, you don't want that to affect your own infra or other customers. Once again, global implementation seems very risky. This should be scoped to where customer has their interfaces... 5/x
An option to having FlowSpec on devices directly, would be to have FS speaker somewhere else. Adverties NLRI to it, then use standard automation tool Southbound to implement standard ACLs. Maybe not as fully featured but probably a safer implementation. @ytti 6/x
In the end, shit happens. And it was on a Sunday. Still, 5h of outages for a pretty basic problem. Not nearly as complex as the DWDM outage some years ago. This should have been resolved faster.
Many lessons to learn on improving MTTR. 7/7.
Many lessons to learn on improving MTTR. 7/7.
