How to make a game that will be easy for others to mod?
I want to develop a game and make it a hassle free experience for other developers to make mods for it.
However, I don't want to open source my game because I would like to sell it.
My idea was to create a kind of API that others could hook into to add and modify content in the game. The idea would be similar to a frontend using a REST API to interact with the backend.
I wonder if there are best practices when it comes to making your game modable. How is it usually done by professionnal game devs?
Make the core content of your game a mod itself. That kind of design will ensure your API is always up to date with the game.
Embedding a Lua (or similar) interpreter is a somewhat common method in the games world. This allows modders and develipers to easily add scripts which can use APIs exposed to Lua.
Observe Factorio and do what they do, as far as you can. Expose everything to Lua that you can, document well, if you can cultivate a community then it can guide you.
Look at what Doom and Quake did before they were open sourced. I think the keys there were releasing the data formats, open sourcing dev tools, and permitting mods to be loaded easily. Also engaging with the community helped.
You could consider making only the game's code open source, keeping its assets proprietary. That way you are still able to sell the game normally.