Thread by @TubeTimeUS

this happened to a Boeing 787 while it was parked at Boston Logan back in 2013. yes, the infamous🔋🔥. 🧵

it's *very* fortunate that it was parked at the time because the smoke filled the cabin. firefighting was difficult for many reasons, including the fact that the battery disconnect switch wasn't accessible!

the NTSB investigated and found out some very interesting details but never found a definitive root cause. here's the inside of the battery bay where the fire occurred.

here is the battery box right after they removed it. notice the broken and distended lid? there was a lot of pressure inside and no safety vent (🤦‍♂️), so it bent and then ruptured.

upon lifting the (bent, broken) lid, this is what investigators found -- a charred mess! but this is the NTSB, and they are good at sifting through charred messes.

for contrast, this is what an undamaged battery looks like. it's basically a bunch of cells wired in series along with a monitoring circuit.

each of the cells looks like this. it's a metal can, and inside are three "jellyroll" assemblies wound together and welded to electrodes called "current collectors." electrically, it's like three cells wired in parallel.

so if you pull out a jellyroll winding and unwrap it, you can see it is like a sandwich. the metal foils (Al and Cu) are the electrodes, the separators prevent electrical short circuits, and the lithium forms part of the chemical reaction that stores the energy.

although nearly all the cells had caught fire and failed due to thermal runaway, there were two cells (#5 and #6) that looked like they may have failed first, causing the fire that then caused the other cells to fail as well. here's cell #5.

investigators have fancy equipment like a 3D X-ray CT so they can inspect the cell before tearing it apart. cell #5 has some strange distortions in the case.

in fact, both cell #5 and #6 had broken current collectors -- remember, these are the fingers that electrically connect each jellyroll to the main cell terminals. this is pretty suspicious too.

the next step was to take apart the cells. they chose to use a Dremel tool and worked with a plastic shield just in case the cell caught fire again.

(the sparks are from the cutoff wheel in the dremel sparking off the steel outer case)

here's what they pulled out of cell 5. lots of crispy carbon, mmm. i'm sure it smelled pretty terrible.

here are those broken current collectors. they are actually designed to act as "worst case situation" fuses if there is excess current flowing. but these look broken, not melted.

it's badly charred, but they could see that the cell's vent had also opened.

here's the vent in another cell that also burst so you can see what it looks like without all that carbon in the way. this releases the internal gas pressure during a failure so it doesn't turn into a pipe bomb and explode.

the investigators then painstakingly unwrapped each of the jellyrolls. here's the middle one.

something quite notable are the wrinkles that they found in the jellyrolls. due to the manufacturing process, the foils with their chemical coatings aren't always consistently flat.

they also found evidence of internal electrical arcs, meaning that the cell suffered some internal short circuits.

this led to the NTSB's first determination: that it was an internal short circuit in cell 5 or 6 that lead to the thermal runaway. beyond this, they don't have a precise root cause.

but back to those wrinkles: they visited the Yuasa factory where the cells were made, and found that the jellyrolls are wound as a cylinder and then later on there's a person who flattens it out by hand! this is likely the cause of the wrinkles.

i once chatted with a representative of another Li-Ion battery manufacturer who does odd-shaped cells for consumer products (like watches, fitness rings, etc). she was insistent that their cells are NEVER bent to shape due to reliability concerns.

https://twitter.com/TubeTimeUS/status/1223043332928000005

so was this the cause? i don't know. the NTSB also pointed out that the Yuasa factory performs welding in the vicinity of the jellyroll winding process and noted FOD (foreign object debris) that *may* be able to contaminate a jellyroll.

the more concerning part is that most of Yuasa's quality control inspections were done after the cell was fully assembled. the NTSB uses more diplomatic language here but they're basically saying that the inspector looks at the cell, shrugs, and says "looks good to me!"

but it's not the fault of the inspectors. they're put at a disadvantage since they are engaging in "passive vigilance" for 8 hours a day! sounds familiar...

in fact, i've written about this before, when passive vigilance contributed to the Tesla crash a few years ago in CA:

https://twitter.com/TubeTimeUS/status/1319744440571813888

apparently this cell design is very similar to an existing Yuasa design that has been used a lot for industrial applications. but Boeing extrapolated MTBF data from an installed base of only 14,000 cells (not identical either, but similar). oops.

they also only did a single nail test. for a battery in such a critical place, you'd think there would be more failure testing like this.

by the way, the nail test isn't quite like those YouTube videos. here's the "nail" that UL uses for battery penetration testing (pentesting?) it's very fancy, with controlled dimensions and even a built-in thermocouple!

this is how they get fun photos like this one. lots of acrid battery smoke!

anyway, it turns out they had battery fires before, while the battery charging circuit was being designed, and had to redesign it. i doubt the root cause is the same, but it makes me wonder about that battery charger, and also about the battery management circuit.

i don't believe the NTSB reviewed the firmware in either device. is it possible a software glitch caused the problem? the hardware is supposed to be designed carefully, with built-in redundancy.

here's another worrying fact about the battery charger. my knowledge about oscillatory waveforms (such as PWM) being used to charge batteries is about 15 years out of date, but this bothers me and I think it ought to be tested more thoroughly.

as a result of the NTSB investigation, Boeing (and their conga line of suppliers and vendors) made a number of changes to the design to mitigate the issue.

they added a vent to the battery box, and then stuck it inside another, bigger box. this box is made out of thick steel with lots of bolts. they pressure tested it to 3x the rated pressure (which is why the one in the photo is bulged). this box also vents to the outside.

it's just a hilariously overengineered pressure chamber. but i suppose when there is that much scrutiny and you just don't want to mess it up...

here's the battery box vent on the bottom of the aircraft. it reminds me of something...

it kind of reminds me of a dolphin's blowhole.🐬

you can find the official NTSB report here: ntsb.gov/investigations… along with lots of other documentation.

another good document is the Materials Laboratory Factual Report, which is very long but has some great photos of the cell disassembly process.
ntsb.gov/_layouts/NTSB/…

i want to conclude with the usual disclaimer: i'm not a battery expert nor am i a professional engineer (PE, as defined by the state of CA). i'm leaning entirely on the NTSB report. take with salt, as usual.

and i don't want to be *too* hard on the engineers at Boeing, Thales, Yuasa, and Securaplane. working with Li-Ion batteries is really, really tough, particularly in high reliability applications like this.