What problems do you face while using NLP models in production?

Question

As someone whose launched a business using generative nlp models, I found my fair share of problems with them.For those of you who used models such as gpt-2/3.5, gpt-neox, etc for a service or in prod, what problems do you currently face while using the nlp models?

vazgenzohranyan · Accepted Answer

In our case, the biggest problem is to deploy, monitor and view the track of the model in the production. Most problem is that it is not well fit to our platform that DS use for their model deployments,and we need to change it for gpt like models. The most important problem is that, you need to have unified schema for saving inputs, outputs and then calculate metrics on them vor further development and alerting.