What is the correct way to deal with pipelines?

Question

I have an application where once a job succeeds it needs to go through a series of steps, some are CPU intensive (ffmpeg, etc.), others require communicating with an external API that may fail (Discord), etc.This is not a massive data-engineering project. I just need to ensure the steps get run to completion.My current plan to do this is to just use Redis lists as queues, and then have steps go from one queue to the next, but I'm wondering if there's a better way.

lantry · Accepted Answer

"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas:
- https://camel.apache.org/
- https://www.windmill.dev/
- https://github.com/huginn/huginn
Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.

dylanhassinger · Answer

Buddy.works is my favorite CI tool