How to make server side code atomic?

If sending the email is mission critical you could start a transaction, update the DB, send the email, commit the transaction. Maybe slightly better is: start the transaction, update the DB, put the "send email" job in a job queue, commit the transaction. And the job queue takes care of sending the email, retrying on fail, notifying the admin if sending the email keeps failing etc.

Gotta love the state of today where "I need to solve a basic state-execution flow" translates to "Let's pay a third-party service to do it".

Almost makes me forgive the interview process tech-companies demand just to check dev's can actually cover the basics.

I don't know how it's done now or what APIs support, but here's an old-school design:

During the first database commit, set a field indicating that the associated email is pending. After sending it and getting a success, do another database commit setting the flag to sent.

You will need a server-side task, maybe a cron job, to periodically query the database for pending emails and attempt to resend them, then set the flag to sent. Also, hopefully the API you use to send email has a way to avoid duplicate emails, perhaps by attaching a unique ID to the request that's used to detect duplicate sends.

Welcome to distributed systems. The answer is "you can't, you can only increase the chance of the correct thing happening".

It boils down to choosing between:

    - possibility of sending 0-1 emails
    - possibility of sending 1+ emails (in reality is 0+,your service could still go down permanently)

The correct solution is to make this event based and not sequential.

You have service A that checks if it needs to send an email.

Service B submits a value to DB and adds a task to the email system inside a transaction. If the DB is down, the service will retry later. If the email service isn't responding, it can roll back the DB change.

I see a few solutions in the comments: but one thing that is not acknowledged is: "What do you do if the email service executes successfully but your server crashes before the POST is returned". I guess this isn't the worst thing in the world since that just means you send the email twice.

Edit: Actually the worst case is pretty bad, you could accidentally double-charge a user using Stripe.

  begin transaction;

  write to db;

  post /api

  commit;

Assuming rollback is implicit when an exception occurs, failure at any point results in the whole operation not executing.

This is a great article going into more specifics. https://brandur.org/idempotency-keys

Try searching online for "outbox pattern", I think you'll find your answer there.

Background queue pulling from the db and pausing / retrying failed jobs

SQS