HACKER Q&A
📣 earthnail

Embedded NN Libraries


Dear HN,

Do you have recommendations for libraries for running NNs on embedded devices?

- must compile using a C or C++ compiler

- must be efficient (runtime speed on embedded obv matters)

- must support conv, lstm

- easy export from PyTorch

- bonus points for rfft / torch.stft support

Any advice on how to choose/evaluate libraries would also be welcome. So far I've had a look at tinygrad, ONNX, and ExecuTorch. It seems ONNX is the most supported, stable option, but I'd very much welcome advice from people who have actual experience running NNs on embedded devices.

  👤 fwsgonzo Accepted Answer ✓
I've embedded llama.cpp for a while now. There may be a point where it's not embeddable anymore, but for now it works great. My experience with the others is don't bother. The build systems and dependency management is truly awful.

Also, I'm obviously talking about porting here. You will need to tailor the code to your needs, thinking of it as a library. You cannot run an unmodified program.