Things like device shadows, choosing between MQTT/HTTP, how an MQTT broker works, the key challenges and good approaches around ingesting data, sending OTA firmware updates and new device settings down, the pros and cons of rolling our own vs e.g. using AWS's tooling.
I've googled around and have found piece parts. But are there any good ebooks or similar resources I could dive into?
There are pre-built platforms like Particle where you get OTA and a basic platform for a nominal fee. They have pros and cons as any platform does. Plenty others as well.
MQTT is nice, but not all embedded platforms will support it for a variety of reasons. Security is a major component to embedded work as well.
My opinion having done these projects. Define the needs of the device and the features it needs to support now and over the next 3-5 years (best the business can predict). Then define the hardware based on those needs, that will let you know what software architecture(s) are reasonable. That will also let you look at various IoT platforms that exist like AWS etc and match them to your requirements. This up front process is way more important than it is with something like web development.
I can share war stories about what happens when you choose hardware via the MVP methodology like you would for a web project. Let's just say it never ends pretty, hardware is harder, takes more time and requires a more methodical approach then rapid iteration of typical software engineering. Of course, if you are just trying to check the temperature and humidity at a location and log them, it isn't very difficult and is solvable using off the shelf components which are low risk/low overhead.
Happy to answer questions if I can.