How have you implemented human-in-the-loop workflows?

Question

I&rsquo;ve seen a few startups now re-implement the same things for human-in-the-loop workflows.These are where a product is backed by a human somewhere in the world working a queue. Content moderation is the classic example, where when a user &ldquo;flags&rdquo; a social media someone on the other side of the world looks to decide if it&rsquo;s violating or not.I&rsquo;ve seen each one re-implement reviews (QA), task assignment, worker productivity measurement, Slack notifications, SLA/due dates, and so on.I&rsquo;ve written a draft of a framework at http://tinyurl.com/opsqueue-framework, but before building anything want to learn more about how other software engineers have solved this problem.If you&rsquo;ve implemented human-in-the-loop workflows before, how did you do it?

matt_s · Accepted Answer

How is this problem any different than a task/ticket/todo tracker?I don't see custom built todo trackers as adding value to that workflow. What features are you thinking would require something that any of the 100's of SaaS offerings out there don't do that would be needed?

cotera-grant · Answer

I come from Ruby on Rails world at a medium sized software company. I think we got a lot of leverage from how good our wrapper was around sidekiq. We asked our engineers to break down their business logic into small to medium idempotent "jobs" and then we'd give them a class they could inherit from that would hook up all the alerting / observability / rate limiting / retry / SLA stuff.I really liked the "Job Queue" abstraction, because it gave our engineers a good mental framework to think about scalable computter processes. Curious if there's some parallel where the workers are humans instead of ruby processes?

koliber · Answer

Would it help to think of it as two workflows? The first workflow ends with a human. The second workflow is triggered by the human. To build those two workflows, use whatever tools allow you to build such no-human-in-the-loop workflows.

fswd · Answer

Yes, I have implemented HITL. There's a task queue(psql skiplock). You make a request (insert row). It builds a "workflow graph" of your request and compiles it, a combination between a FSM and graph program. It runs an iteration on the workgraph, and then if successful, submits the next step in the queue as "pending". the user then inspects and accepts the tasks, which then run another iteration of the workgraph. If the user rejects it they can click "playground" and manually do the step.

zurawiki · Answer

What about just using Linear or another off-the-shelf task management software?Seems like most of the features are covered there

nekudotayim · Answer

I worked at a CRM/BPO company doing something like this for customer support inquiries, everything was pretty bespoke. We did all the productivity measurement off of, IIRC, Redshift tables fed by Kafka events (probably some airflow in there to ETL) published during task assignment.

oytuntez · Answer

I have my eyes on https://temporal.io/ for similar purposes.

How have you implemented human-in-the-loop workflows?

How is this problem any different than a task/ticket/todo tracker?
I don't see custom built todo trackers as adding value to that workflow. What features are you thinking would require something that any of the 100's of SaaS offerings out there don't do that would be needed?

Would it help to think of it as two workflows? The first workflow ends with a human. The second workflow is triggered by the human. To build those two workflows, use whatever tools allow you to build such no-human-in-the-loop workflows.

What about just using Linear or another off-the-shelf task management software?
Seems like most of the features are covered there

I worked at a CRM/BPO company doing something like this for customer support inquiries, everything was pretty bespoke. We did all the productivity measurement off of, IIRC, Redshift tables fed by Kafka events (probably some airflow in there to ETL) published during task assignment.

I have my eyes on https://temporal.io/ for similar purposes.