Thread by @jasonnochlin

Our #RubyOnRails apps at GaggleAMP have published content to 3rd Party APIs for almost 10 years. Here are some things we've put in place to reliably perform these API calls from our apps’ background job workers:
.
.
Thread👇

We use a state machine to control the process. The diagram below shows a simplified version of our state machine (some states for app-specific functionality are not shown). At the row level, we store the current state along with a timestamp to allow future scheduling. 2/

A recurring job lives in our job queue that enqueues individual API jobs. This recurring job queries the database for all API jobs in the approved or retry state with schedule timestamps in the past and enqueues them. It recurs every 10 seconds. 3/

To ensure the recurring job didn’t die, a cron job runs every few minutes to verify that the recurring job still exists in the job queue. 4/

The individual API call jobs are then picked from the queue by a worker. The worker performs a number of checks before executing the API call. If any fail, the worker cancels the job: 5/

🔓 Lock Acquisition - the worker must be able to acquire the lock to prevent multiple workers from executing the same job. 6/

✅ Validation - the worker performs checks to ensure that the job _should_ still be executing, such as billing status and global pause (customer's can stop all activity, e.g., in the event of a crisis) 7/

👮‍♂️ Authorization - there worker checks that the associated API tokens exist in our system and are not known to be invalid so the job _can_ be executed 8/

If all checks pass, the API call executes... and either succeeds 🎉 or fails 😧 9/

For every API call execution, we have instrumentation to measure the aggregate number of attempts, successes and failures; and logging to record individual request/response results (we use a database table, but a log aggregator would also work). 10/

What causes failure? Could be:
* A network error stopped our HTTP connection to the 3rd party
* The 3rd party returned an error (e.g., Twitter has 65 different error codes: developer.twitter.com/en/docs/basics…)
* A timeout occurred (the trickiest b/c the request may have succeeded)
11/

We classify all errors into 3 categories:
* Unavailable - the 3rd party was unreachable
* Unauthorized - the 3rd party returned an error indicating the user’s API credentials are not valid
* Forbidden - the 3rd party refused the request
12/

Errors in each category are handled differently:
* Unavailable - retry the API Call with exponential back-off (github.com/mperham/sideki…)
* Unauthorized - cancel the job and notify the user to re-authorize with the 3rd Party
* Forbidden - cancel the job and notify the user
13/

In the Unavailable case, we limit the total number of retries. If the final attempt fails, the job is canceled and returned to the draft state for the user to review. 14/

If a new error occurs that has not yet been classified into one of the categories, details are sent to our exception reporting service for bug triage by our support team. 15/

Feel free to ask questions on any of the above!

A friend building a new application asked me for advice on how to manage API and inspired this thread. Hopefully the points above save others from the headaches we went through early on 😅