Curious what everyone’s approach is for handling cron jobs that depend on external APIs.
Had a data sync job that pulled from a vendor API every hour. Worked great for months until the vendor started rate-limiting us during peak hours. The job would fail, retry on the next cron tick, hit the rate limit again, and basically spin for 3-4 hours until traffic died down.
What I ended up doing was adding exponential backoff inside the job itself — first retry after 5 min, then 15, then 60. But it felt wrong because now the job’s execution time is unpredictable and can overlap with the next scheduled run.
The other approach I considered was just letting it fail and having a separate “catch-up” job that runs less frequently and handles any gaps. Cleaner separation but more moving parts.
For those of you running scheduled jobs that depend on third-party services: do you build the retry logic into the job, handle it at the scheduler level, or just accept that some runs will fail and deal with it downstream?
Also wondering if anyone’s hit issues with jobs that silently succeed but return partial data — like the API responds 200 but only gives you half the records because of an undocumented pagination change. That one took me a while to catch.
I’ve never used it, but this looks like it’s aimed at this problem:
Have it put information about last run details in a persistent file. When the cron job runs again, check the file first to know how the next iteration should run.
For stuff this complex, maybe cron isn’t the best choice in the first place? A constantly running background service might be more flexible, better able to do things like checking for partial response and retrying if the data is incomplete.
Two ways I’ve handled this. If you have some orchestrator, like kubernetes, you can use the job feature to retry. In that case you just let the script exit with error status.
If we’re staying with just cron and shell scripts, look at using flock(1) with timeout in a bash script (example). You can create a lock to prevent multiple copies running and it cleans up automatically. You can combine this with time-stamped files to enforce a minimum amount of time since last run, or whatever you like.
I prefer having the retry handled outside the job itself, but I’ve used both approaches. I’ve grown to prefer separating the code for “doing the thing” from the code for “orchestrating the thing”, but it depends on what environment it runs in, how critical it is, and how who else may be supporting it.
Finally, set up something to monitor it and alert you if it hasn’t run successfully for some time.
I’ve been in a similar situation, and what I wound up doing was what you describe as a “catch up” job. I do as much work as I can before I can’t anymore, store the result, and then continue from the failure point at a later time. It does feel a bit disgusting with a lot more moving parts but you gotta get a little creative with unreliable APIs. Gotta be careful with this though, because if you try and run this catchup at the start of every new run of the job then you can see how the backlog could quickly grow impossibly large if you don’t allow requests to ultimately fail and drop out of the queue entirely.
Also there’s not really anything you can do if the pagination methodology of the API is changed without warning. If the provider has some way to subscribe to API changes, like a newsletter, then you could get on that. Otherwise all you can really do is set up some alerting if you can reliably recognize when you’re only getting a partial data set.
