HACKER Q&A
📣 rococode

How can I set up an ML model as a scalable API?


I have a custom ML (PyTorch) model that I would like to set up as a service / API - it should be able to receive an input any time and promptly return an output. It should be able to scale up automatically to thousands of requests per second. The model itself takes around a minute to load, an inference step takes around 100ms. The model is being called only from my product's backend, so I have a bit of control over request volume.

I've been searching around and haven't found a clear standard/best way to do this.

Here are some of the options I've considered:

- Algorithmia (came across this yesterday, unsure how good it is and have some questions about the licensing)

- Something fancy with Kubernetes

- Write a load balancer and manually spin up new instances when needed.

Right now I'm leaning towards Algorithmia as it seems to be cost-effective and basically designed to do what I want. But I'm unsure how it handles long model loading times, or if the major cloud providers have similar services.

I'm quite new to this kind of architecture and would appreciate some thoughts on the best way to accomplish this!

  👤 calebkaiser Accepted Answer ✓
I work on a free and open source project called Cortex that deploys PyTorch models (as well as other frameworks) as scalable APIs. It sounds perfect for what you're looking for: https://github.com/cortexlabs/cortex

Cortex automates all of the devops work—from containerizing your model, to orchestrating Kubernetes, to autoscaling instances to meet demands. We have a bunch of PyTorch examples in our repo, if you're interested: https://github.com/cortexlabs/cortex/tree/master/examples

👤 gautamkmr89
You can use sagemaker to perform ml model deployment. https://aws.amazon.com/sagemaker/

Sagemaker takes care of infrastructure for you. It has also been integrated with various orchestrations like k8s, airflow etc.

https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_...

👤 rankam
Maybe you could try to save the pre-trained model to a storage bucket (e.g. s3) and then use flask (or whatever framework you like) to create the endpoints. When the flask app starts, the model can be loaded into memory from the storage bucket, and then you could create, for example, a /predict endpoint that accepts whatever data is needed to make the prediction. Deploy this to some PaaS (Heroku, AWS EBS, GCP App Engine) that has auto-scaling as a feature and you're sorted.
👤 streetcat1
So you can go with Kubernetes. This is my preferred tool.

With Kubernetes, you can either wrap your model inside a container or mount it into the container from a persistent volume.

As for scaling you have two options:

1) Horizontal Pod Autoscaler https://kubernetes.io/docs/tasks/run-application/horizontal-...

2) Knative, which is Kubernetes serverless on-prem solution.

👤 doppenhe
Algorithmia here . What are you concerned about license wise? You own all ip always. There is some restrictions if you choose to commercialize on our service (mostly guarantee you won't take it down on users). System was built for this. Happy to answer questions
👤 mailslot
Tensorflow has a decent C++ layer, just sayin.

PyTorch? Dunno. Last I spoke to those ppl, they had a solution too.