But there's a part of it I haven't been able to wrap my head around: How to set up automated/scheduled tasks.
I'm most familiar with Python and Node/JavaScript, and that's what I'm planning to use as a starting point for the back end of my system.
In the past I've set scripts running on a crontab, and scheduled tasks leveraging Celery, RabbitMQ, and Redis.. but I've never built something with scheduled data fetching that fed that data into an event sourced system. Using the tools I know don't feel quite right.. like they're the nail to my hammer and I'm missing something.
To simplify: Basically I have one service that'll rely on an external data source (the data I need is available over a RESTful API), and another service that uses the resulting state of that data to create events in its own context.
Does anyone have experience with this sort of architecture? Could you offer any advice or point me at some resources?
Thank you in advance!
There could be some other options. It depends on what the task needs to do and where it is in the process. There are triggers one could use, push style notifications from a prior step, etc. It could even make sense to keep a certain task as batch depending on how it fits in with the other processes.
I am failing to see what the friction is between scheduled tasks and an event sourced system. If a customer were logging in every day at 10am (on a schedule) and issued their own events based on some external state, your system would be none the wiser to the "scheduled" nature of their behavior.