Thread by @bradarkin

I had the opportunity to testify yesterday to the Senate Committee on Homeland Security & Governmental Affairs about log4j. A 🧵 on the experience.

My prepared statement: hsgac.senate.gov/download/testi…

Video of session: hsgac.senate.gov/hearings/respo…

Rocky start. First time wearing suit & tie since covid. Two years of WFH in sweatpants & hoodies hasn’t done me any favors. Realized to my horror the extra lbs meant I couldn’t get the top button of my shirt fastened! Borrowed a safety pin from the front desk and was on my way.

My bag check tag from the hotel seemed a little ominous given I was meeting with the Senate committee responsible for oversight for Homeland Security.

And then the cab meter read $6.66 when we pulled up in front of the Dirksen Senate Building… <gulp>

The U.S. Surgeon General, Dr. Vivek Murthy (@Surgeon_General), was behind me in line for the security line to enter the building. Made everything seem very DC even if he wasn’t giving me six feet of space…

The chair of the committee, @GaryPeters, & @SenatorHassan schmoozed in advance of the session with me, @jadefh and @ke4qqq. (Trey Herr from @CyberStatecraft was dialed in as the fourth witness.)

Because of covid, each witness was allocated only 2 guests. I rolled heavy with @EricWenger and @WaldoMcMillian from the Cisco Government Affairs team. You can see them sitting behind me in this picture. They sat close to hand me notes during the hearing.

We were sworn in and then each witness read a 5 minute version of their (longer) written testimony submitted a few days earlier.

Some of the key points I wanted to get across:
1) Bugs happen. The next log4shell event could involve a commercial or closed source component - we shouldn't focus narrowly on open source here. Downstream organizations that use software need to be prepared.

2) @CISAGov’s recently developed #JCDC shows promising signs that it can facilitate effective partnerships across industry and government during complex security incidents like #log4j.

3) Incident reporting legislation like @SenGaryPeters and @SenRobPortman introduced will ideally spur @CISAgov to accelerate timely sharing of actionable information at lowest level of classification possible to vendors who can fix the bugs.

4) @CISAgov’s Binding Operational Directive 22-01 correctly emphasizes importance of prioritizing efforts to remediate those known vulnerabilities with available patches or mitigations showing signs of active exploitation.

5) Executive Order 14028 drives two important areas of work—improving the security of software development so we reduce bugs, find/fix faster; and zero trust networking so we are more resilient when there are problems that require patching.

After each witness finished reading their prepared statements, we moved to Q&A. @ericgeller

https://twitter.com/ericgeller/status/1491064575852220416

and @aevavoom

https://twitter.com/aevavoom/status/1491061682399215629

published 🧵 s of the session.

Ranking Member Portman and I talked about Cisco’s experience shipping security updates to our customers. Cisco published details (tools.cisco.com/security/cente…) regarding impacted products and ship dates for each affected product.

We managed the Cisco response across three “swim lanes:” 1) identify affected “on-premise” products and publish security updates for customers to apply; 2) remediate any impacted customer-facing SaaS/cloud products and 3) remediate any impacted Cisco IT / back-office services.

The fastest teams were able to remediate within 24-48 hours and many teams had gotten their code patched fast enough that they got to do it all over again when the second log4j patch was released on Tuesday 14 Dec.

By the end of the first weekend of the response (~72 hours after the first log4j patch was released) we largely knew what work had to be done. Most updates were published to customers within 14 days of the initial log4j patch release.

Throughout the event we updated this page tools.cisco.com/security/cente… every ~4 hours as new status updates came in from the various teams across Cisco engineering and IT.

One thing I wasn’t able to work into the Q&A was Cisco’s leadership in a cool new open source project: @gitBOM gitbom.dev We see this as a complement to the work happening in the SBOM space and encourage folks to get involved.

The session ended about two hours after we started. I went back to the hotel to change out of my suit & tie as fast as possible.

I was the only person on the connection flight from DCA to EWR, which was really weird.

About my mask. @EricWenger freaked out when I showed up wearing a blue mask with 🍍 on it. He thought it was some subversive reference to the whole @C_C_Krebs #WarOnPineapple

https://twitter.com/c_c_krebs/status/1330282188777132035

For the record, I didn’t know anything about that.

I enjoyed meeting @jadefh and @ke4qqq at the event and was glad to share Cisco’s experience responding to the log4j event with the Committee. A big THANK YOU to all involved for helping pull the info together to make the session such a good use of time. <end>