How do you iterate on and manage prompts in production?

Question

I'm curious about how people handle managing and iterating on their prompts, especially for production-facing prompts that many users depend on.- Where do you store your prompts? - Do you use version control? - How do you test prompts after editing? - Where do you store your test sets? - Do you evaluate results? If so, how? - Are you fine-tuning models like GPT-3.5 for better/cheaper results?

BWStearns · Accepted Answer

We have a script and a input library that has a bunch of scoring dimensions and allows a head to head comparison of a new candidate prompt vs what's in prod. It takes a configuration (prompt, which LLM to use, temperature etc.) and then gets run with all the various inputs and makes a json blob of the outputs for scoring.Most of the score dimensions are deterministic but we've added some where we integrated an LLM to do the scoring (... which brings the new problem of scoring the scoring prompt!). We also do a manual scan of the outputs to sanity check. Not doing any fine tuning yet as we're getting pretty good results with just prompting.

ivanpashenko · Answer

"7 likes / no comments" --> should I read it as: people interested in others people experience, but have nothing to share about their own? - No prompt on production? - No testing or other routines about it yet?Please share your current status :)