Is it a good idea to use AWS Lambda for ETL Pipelines?

Question

We are working on an OSS for ETL (Typhoon data orchestrator) and we are thinking about using AWS Lambda as the backend. The benefit is that it has a built in scheduler and we don't need to build our own. We can also spawn multiple Lambdas to scale concurrency.Do you think this would be a sensible option? Would the 15 minutes running time be a deal breaker?

ignorantguy · Accepted Answer

We use lambda in our production environment for ETL. It works really well. In our case when we were using it, we modeled the code to account for 15 minute window. Hope that helps.

vlahmot · Answer

You can always use step functions or some janky recursive calls to work around the 15 min timeout.I would recommend using docker containers and writing it in a way that you can swap out lambda/ecs/k8s/etc as the backend infrastructure. That way you you can punt on doing any fancy work to try and break the jobs up across lambdas under that 15 min constraint.