Thread by @ralsina

So, let's talk a bit more about watching TV using Apache Airflow (which is not a very good idea legally, but has good potential technically!)

This is the first thread: threadreaderapp.com/thread/1229070…

When we last looked, we had two airflow tasks looking for the latest episodes of two series, and one task that was using xcom_pull to get that information and find torrents.

Well, that was not really working well.

Why? Because of this line in the downstream task.

So, xcom_pull "pulls" data that was "pushed" by the upstream tasks. But ... we are pulling once. If two tasks pushed, then we will only pull the latest value, and search for a single series' episode.

So, we have to do better.

Turns out xcom_pull can take a "task_ids" argument, which if it's a list of tasks will pull the value required from EVERY ONE of those tasks.

Luckily, operators know the ids of their upstream (and downstream) tasks, so this is easy to do.

So, I can get all that data, iterate, and process each one as before.

Another interesting bit is that there is no need to explicitly declare each "search for latest episode of series X" task in the DAG. That's what we have config files and loops for (notice the use of the dag as context manager)

So, now we have a WORKING thing that given a list of series names ends up with a list of torrents and series metadata.

The next logical step is to download those things somehow.

One way to do this is:

1) Start transmission-daemon
2) Configure however you want
3) Let's use transmission-remote to add torrents to it

Since transmission-remote is just a command, we can use PythonOperator's more rustic brother BashOperator

Basically, BashOperator runs commands.

Here is what works (will explain)

DAFUQ, you may say. And indeed.

But basically: BashOperator is **templated**. That string there? It's ran through jinja with the context (remember the context?) and thus things are replaced.

So it ends up being something like this (but much longer) because it's extracting the torrents from the context.

So, our DAG now looks like this.

And what happens when you trigger that DAG?

Coming in part 3:

* So, what can you do with that?
* What value is airflow adding?

@threadreaderapp please unroll