Share this page!

Ben Welsh Profile picture
Twitter logo
Dec 16, 2021, 9 tweets, 11 min read
🛰️🔥 #CODE 🔥🛰️

Our very own @irisslee polished, expanded and re-released the open-source tools @latimes uses to track wildfires.

Here's NASA’s satellite hotspots, updated hourly

observablehq.com/d/365549932102…
@irisslee @latimes Here are the rest of our utilities, which fetch, clean and store data from a variety of government sources.

github.com/datadesk/noaa-…

github.com/datadesk/nifc-…

github.com/datadesk/inciw…

github.com/datadesk/air-q…

github.com/datadesk/nws-w…
@irisslee @latimes They are now all scheduled to run scrapers as @github Actions, making the latest data available to all in a standard format via live, free URLs.

Here's an example of how easy this new technique makes it.

github.com/datadesk/nasa-…
@irisslee @latimes @github The packages here are integrated into a larger, private repository where we merge, refine and republish the data for the public on our live wildfires map.

latimes.com/wildfires-map/
@irisslee @latimes @github While GitHub Actions don't run frequently (and reliably) enough to power some rapidly updating commercial products, we believe they are a powerful new model for easy, low cost scraping in journalism.
@irisslee @latimes @github Another example is our coronavirus tracker, which is powered by dozens of data scrapers. 100% of them run free and open on GitHub.

github.com/datadesk/calif…
@irisslee @latimes @github @irisslee and I are so proud of this pattern, we're eager to spread the word.

That's why we're linking up with @aadittambe at #NICAR22 to offer a first-of-its-kind bootcamp.
@irisslee @latimes @github @aadittambe We're calling it First GitHub Scraper.

It's a new three-hour class we're developing that will teach you the fundamentals of everything covered above.

Stay tuned for when #NICAR22 schedules drop via
@IRE_NICAR. Then sign up.

See you in Atlanta.
@irisslee @latimes @github @aadittambe @IRE_NICAR The @observablehq noteboook that makes the globe at the top of this thread is now published as well. You can find it here. observablehq.com/@datadesk/nasa…

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Ben Welsh

Ben Welsh Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @palewire

Ben Welsh Profile picture
Jan 1,
It's Public Domain Day! As of today, copyright works from 1926 are free for reuse.

That includes "is 5," E.E. Cummings’ fourth book of verse.

We are now free to add its contents to my open-source archive of the poet’s work. ⌨️ Data entry is underway!

cummings.ee/book/is-5/
I launched the site earlier this year.

With the help of numerous volunteers, we've already published three public domain books.

Read all about it in my earlier thread.

I'm putting out a new call for contributors. If you have a copy of the book and would like to keypunch a poem, here are instructions.

github.com/palewire/cummi…

If you're new to GitHub and need a hand getting started, just let me know.
Read 4 tweets
Ben Welsh Profile picture
Oct 26, 2021
This promotion is the capstone to a complete, fundamental change in how the @latimes makes the charts and maps that accompany stories.

The effort involved dozens of people all across the newsroom. Let me tell you a little more about it. 🧵
@latimes When we started, there was a clear consensus change was necessary.

1️⃣ Every chart was produced separately for web and print, often by different people. 2️⃣ Web charts weren't responsive. 3️⃣ Copy edits were done at night on paper. 4️⃣ We staffed until the print deadline year round
@latimes Those practices all had their time and place, and the output was indeed high quality.

But the urgent need to grow our digital audience called out for investing in more ambitious efforts like visual stories and news applications.

Streamlining this process was an obvious way in.
Read 28 tweets
Ben Welsh Profile picture
Oct 25, 2021
I am pleased to announce that @thomas06037 has been promoted to serve as the @datagraphics assistant editor for graphics. 📈

latimes.com/about/pressrel…
@thomas06037 @datagraphics In this newly created role, Thomas will:

📊 Oversee our overhauled system for creating charts and maps to enrich day-to-day news coverage

⛓️Direct improvements in how graphics integrate with our growing array of digital publishing platforms
@thomas06037 @datagraphics Thomas joined the @latimes in 2000. In the time since, authored hundreds of graphics, created data applications and coordinated work for some of our most ambitious projects.

latimes.com/people/thomas-…
Read 6 tweets
Ben Welsh Profile picture
Oct 25, 2021
Día de Muertos is almost here.

To celebrate, the @latimes has created a communal digital altar.

Upload a photo and share an ofrenda for a loved one.

By our very own @VanessaMartinez

latimes.com/projects/contr…
@latimes @VanessaMartinez The page will grow to include as many reader tributes as we receive.

It's been seeded by 10 offerings posted by @latimes staff, including my memorial to my oma.

latimes.com/projects/contr…
I didn't appreciate Day of the Dead until I moved to Los Angeles.

The range of artistry that can appear is incredible, and the recent boom in using altars as a platform for activism, showcased by @SHG1970, proves the form's flexibility.
Read 6 tweets
Ben Welsh Profile picture
Oct 25, 2021
New from our crew, and the latest @latimes effort to chronicle California's historic vice president.

By our very own @thomas06037, @maloym and @stiles
The data we've assembled is powering our reporting, like this analysis by @Noahbierman and @stiles.

latimes.com/politics/story…
@Noahbierman @stiles It's all made possible by a custom computer program, developed by our team.

It parses, logs and classifies the schedules emailed out each day by the White House.
Read 5 tweets
Ben Welsh Profile picture
Sep 10, 2021
Our open-source archive of California COVID data has expanded again.

A nerdy new milestone: 100% of our web scrapers are running free and open via @github's action framework.

That's 80+ routines, written by a half-dozen team members, running in unity.

github.com/datadesk/calif…
@github GitHub's action system, which came into its own during the pandemic, has changed the web scraping game.

This approach allows us to store, script and schedule our data-gathering systems. For free.
@github Here's how it works:

1️⃣ Scrapers repo acquires data.

2️⃣ Private repo pulls it in, processes and publishes tracker. Also via actions, and (almost) entirely automated.

3️⃣ Processed files are pushed to open-source data archive for reuse.

github.com/datadesk/calif…
Read 6 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

Like this author's thread?

Get emails when @palewire has new unrolls!

Check out other threads from @palewire!

Read all threads by @palewire

:(