Thread by @palewire

Our open-source archive of California COVID data has expanded again.

A nerdy new milestone: 100% of our web scrapers are running free and open via @github's action framework.

That's 80+ routines, written by a half-dozen team members, running in unity.

github.com/datadesk/calif…

@github GitHub's action system, which came into its own during the pandemic, has changed the web scraping game.

This approach allows us to store, script and schedule our data-gathering systems. For free.

@github Here's how it works:

1️⃣ Scrapers repo acquires data.

2️⃣ Private repo pulls it in, processes and publishes tracker. Also via actions, and (almost) entirely automated.

3️⃣ Processed files are pushed to open-source data archive for reuse.

github.com/datadesk/calif…

@github Today's release includes dozens of scrapers that pry data tracking places, neighborhoods and ZIP Codes from county-run dashboards.

They're monitored by an ingenious @rdmurphy hack that automatically files a pull request if something goes wrong.

github.com/datadesk/calif…

And, to be clear, when I say free, I mean both gratis and libre, both beer and speech.

That means this portion of our most ambitious data gathering effort ever has zero cost.

That's because @github doesn't charge for open-source repositories.

docs.github.com/en/billing/man…

This morning, I expanded our export pipeline to include a few more files.

You can now find cleaned up and regularly updating data tracking hospital load and county-level demographic trends in vaccination.

github.com/datadesk/calif…