OS-es with dynamically-linked libs in separate address spaces?

Question

I've been wondering about the problem of debugging applications, when some of your libraries are closed-source (provided by another company). If there is a memory-corruption, code from a library may corrupt memory used by your own code. I think ideally different libs should run in their own address spaces, so if you have a function foo1() from lib1, which calls a function foo2() from lib2, the call involves a switch of entries in the page table, to protect the address space of lib1. The call would still be blocking. And if lib2 crashes, it wouldn't crash the whole application, but would deliver an exception to foo1() in lib1. Now the question is: has anything like that been done before?

toast0 · Accepted Answer

I haven't seen an OS like this. It's going to be difficult to make this work in general, because libraries often take pointers to structures that have pointers etc. You're going to need to somehow map datastructures that are exchanged between libraries into the various address spaces, but if lib2 corrupts a pointer that lib1 uses, lib1 crashes through no fault of its own.If you did strict serialize/deserialize between libraries, you could make it work, but at that point you need to refactor everything as message passing (think Erlang), and that may be a hard job. A lot of libraries are built around shared access to large data structures, which is not a good fit for message passing.

duped · Answer

Pro audio applications usually let users install third party plugins distributed as shared libraries. A buggy plugin can crash the host.Today applications solve this by spawning a daemon to host plugin libs in a separate process, communicating back with the main host via some kind of IPC. It has some performance/latency tradeoffs but it's better than a crash.

emteycz · Answer

Wasm is probably close to this ideal