Thread by @NoahHaber

Systematic reviews and meta-analyses are like plywood. While they often have a pretty veneer, they are only as useful as the layers of materials they are made of and how it's all put together.

In this essay I will

Hm, might do this, since plywood is so, so much cooler than people give it credit for, and there's some good analogy making with how cross-grain layers are complementary and hold things in check.

I have nerd sniped myself.

Ugh, fine, I'll do it.

Systematic reviews and meta-analyses are like plywood, and plywood is crazy cool stuff that I bet you never even thought about.

But first we have to talk about wood.

Thread!

Wood is not the inert solid block stuff you might think it is.

Solid wood is sneaky evil stuff. It is constantly changing shape, twisting, expanding, contracting, and trying to ruin what you have made. But there are some rules to how it "moves."

The ways it expands and twists have mostly to do with what kind of wood it is, and the direction of the grain This is worthy of many a thread rant by itself, but alas.

thewoodwhisperer.com/articles/wood-…

Think about these twists and movements as something like "bias" in a study.

Like in studies, you can often deal with these biases, but it takes planning and work and skill, but often we just need something that stays put.

Enter: plywood, because plywood is a goddamn miracle.

Plywood is made of sheets of wood (or wood-like stuff) glued together.

And much like a meta-analysis, the strength and stability of the wood depends entirely on what those layers are made of, and how they are put together, and its WAY cooler than you think.

Obviously, plywood made of junk wood is junk (but may be good enough to get the job done)

So it's always good to know how good "good enough" is.

But the construction of ply really cool; it isn't just glued together in sheets; it's glued together in sheets of CHANGING DIRECTIONS.

By alternating the directions, the layers hold each other's movement in check.

A whole bunch of bendy wood that is identical and just layered on top of each other and glued would be just as bendy and unstable.

The results of a meta-analysis of a bunch of studies which all share the same biases will share the same biases of its component parts.

Meta-analysis is strong and stable only when it's made of strong and compatible/complementary study material. If it has gaps ("voids" in woodwork parlance), it'll be weak.

But ply also does another thing: it hides what's in the middle via its outer veneer.

If you just look at it from the outside, you can't tell what's up. A lot of nice looking ply is just very very thin veneer of nice looking stuff containing sandwiched junk.

A whole lot of systematic reviews look much stronger than they are, and you have to look inside.

But not all ply is the same! Different kinds of plys work better for different tasks, and what its made of determines what it's good for.

For example; MDF core ply has two outer wood veneers, and the center is a solid layer of MDF. MDF is cheap, but it has some cool properties.

MDF is cheap, heavy, and stays VERY stable / flat (unless it comes in contact with water), but has poor shear strength and is very brittle.

So, MDF core plywood is a great choice for stuff needs to stay very flat, stable, and void-free, but is well supported and well protected.

A meta-analysis for clinical practice might select more "pragmatic" studies, rather than typical phase-III type RCTs.

Pragmatic trials may or may not have hyper control over adherence, blinding, etc, but can be the right tool for the task at hand.

It's not all just good v bad.

Don't you worry @jamesheathers, I'm getting there.

https://twitter.com/jamesheathers/status/1373310366361485320

Brief story that just about everyone who does statistical consulting will understand.

I was working in a public workshop, and someone came in and wanted to trim a sheet of wood so he could make it into a cutting board.

What he had in his hands was a piece of big box store ply.

Ply in general isn't gonna made for a good cutting board for all sorts of reasons, but one HUGE one is this: industrial ply glues are made of nasty stuff you REALLY don't want in your food, like formaldehyde.

I had to regret to inform him that I couldn't help.

There are plenty of food-safe glues out there, and that's what your wooden cutting board is put together with.

But in general, you have to trust the manufacturer to have used the appropriate stuff to put it together, since you can't really tell.

The glue here is the analytical/statistical methods. It's the code that holds it all together. If it's made of the wrong stuff, applied badly, etc, it all falls apart, and may be dangerous.

But it's often really hard examine the details of the stats and code.

Plywood gets a bad rep in part because it's often cheaply made crap (and some serious no true Scotsman-ing from traditional woodworkers).

If you get (and pay for) the right stuff, you'll get great results.

Meta-analysis is the same way. You get what you pay for.

So, whether you're going out to build something w/ ply or reading a meta-analysis, carefully examine what it's made of.

Bring a wood nerd ("expert") along to check it out too, since they'll likely be better able to critically evaluate what you have.

/thread

Thread revived to mention something else that's super cool about ply and glue. Wood glue is insanely strong, provided that you are gluing faces/edges, and aren't trying to glue end grain.

(image credit: thepowertoolwebsite.com/best-glue-for-…)

Gluing end-grain is a no-go; it'll be super weak and break.

But ply has alternating directions. Its "edges" are always part end grain, part edge grain.

That means that if you glue ply by the edge, it'll always be a medium strength joint (at least on that side of the joint.

No analogy here, it's just a super cool plywood fact that people don't think about.