The project would be self hosted by the institution, and I want to profit from it. Is there a way to sell licenses for usage even though the source code is available freely? Can someone fork my project, change a few tiny details and the name and provide it for free without licenses?
Would having the project be free of price, but selling "support" worth it, or would they just rely on their IT department and not bother with buying it?
The idea of FOSS sounded better as a consumer than as a supplier tbh, so I would like to know your opinion about this.
I think it's very difficult to make money from a Free Software project. It is somewhat possible from a Source Available project, but not easy.
The elephant in the room is Moodle. Everyone is using Moodle. It is GPL3. It is an pragmatic/opinionated project, so you can find people that don't like it or that want some special feature.
Perhaps you want to read about the AGPL licence, it forces people to share the changes in they servers, even if they don't redistribute them. [I prefer Apache or MIT, but it is even more difficult to make money with them unless your project is huge.]
Some people share the [A]GPL version for free, but charge for the same code under a not viral licence. If you accept contributions, you probably need a ask for a CLA (contributor license agreement) so you can change the licence for the payed version. [I really don't like projects with a CLA. They are not free enough for me.]
You can definitely sell F/OSS for a profit. One could quibble about the semantics of whether or not you're really selling a license for "usage" or not, but I don't think that's a particularly useful distinction. Maybe the more important point is to understand that you're not really selling software in a sense. Yes, you could choose to not make binaries available unless the customer pays, but they could always build their own binaries, or get them from a third party.
So why do organizations pay companies for F/OSS at all? Well, there is the support aspect, but its not just "support" as in patches / version upgrades, and phone/email/web/whatever technical support. Those things are aspects of what an organization is paying for, but they're also paying for something more subtle. Organizations, especially larger ones, want stability and continuity. By paying your company, instead of just using your software for free, they are investing in the continued development and evolution of your project. And, yes, they're also paying to have "somebody to sue" if the shit hits the fan. Large organizations want contracts and legally binding commitments around things, and they understand that that costs money.
Can someone fork my project, change a few tiny details and the name and provide it for free without licenses?
Pretty much, yeah. You can protect trademarks, and you can always choose the route of not making the documentation available under a license that permits redistribution, and do other small things to discourage a third party from setting up to sell the same thing. But the licenses (I'm assuming some more or less standard, common-place, OSD compliant license) generally permit that.
Would having the project be free of price, but selling "support" worth it, or would they just rely on their IT department and not bother with buying it?
It depends. There would probably always be "cheapskate" orgs that would download and use the code and never engage with you. That's just part of the cost of doing business. But many organizations will buy your package, assuming it creates value for them, for the reasons mentioned above. Not to mention, their internal IT departments are probably already overworked, underfunded, and stretched thin. They would prefer to outsource the maintenance and evolution of the thing, as opposed to taking on that responsibility themselves.
The idea of FOSS sounded better as a consumer than as a supplier tbh, so I would like to know your opinion about this.
It's not for everybody. There are a lot of implications to "selling F/OSS" and you should think through them carefully and do your research before deciding. I kinda feel like this topic has been beaten to death online over the last 15 or 20 years, so there's plenty of discussion of these points for you to read over and consider. See:
https://www.google.com/search?q=open+source+business+models
One thing I would suggest, if you haven't yet, is to read
Open Source Licensing: Software Freedom and Intellectual Property Law by Lawrence Rosen.