Curious what everyone’s approach is for handling cron jobs that depend on external APIs.
Had a data sync job that pulled from a vendor API every hour. Worked great for months until the vendor started rate-limiting us during peak hours. The job would fail, retry on the next cron tick, hit the rate limit again, and basically spin for 3-4 hours until traffic died down.
What I ended up doing was adding exponential backoff inside the job itself — first retry after 5 min, then 15, then 60. But it felt wrong because now the job’s execution time is unpredictable and can overlap with the next scheduled run.
The other approach I considered was just letting it fail and having a separate “catch-up” job that runs less frequently and handles any gaps. Cleaner separation but more moving parts.
For those of you running scheduled jobs that depend on third-party services: do you build the retry logic into the job, handle it at the scheduler level, or just accept that some runs will fail and deal with it downstream?
Also wondering if anyone’s hit issues with jobs that silently succeed but return partial data — like the API responds 200 but only gives you half the records because of an undocumented pagination change. That one took me a while to catch.
I’ve been in a similar situation, and what I wound up doing was what you describe as a “catch up” job. I do as much work as I can before I can’t anymore, store the result, and then continue from the failure point at a later time. It does feel a bit disgusting with a lot more moving parts but you gotta get a little creative with unreliable APIs. Gotta be careful with this though, because if you try and run this catchup at the start of every new run of the job then you can see how the backlog could quickly grow impossibly large if you don’t allow requests to ultimately fail and drop out of the queue entirely.
Also there’s not really anything you can do if the pagination methodology of the API is changed without warning. If the provider has some way to subscribe to API changes, like a newsletter, then you could get on that. Otherwise all you can really do is set up some alerting if you can reliably recognize when you’re only getting a partial data set.