My current approach is to create a small project – with real data – design an architecture for specific constraints, and estimate how far it could go. Later, I start thinking what I would do to improve it to reach the next level.
I wonder if there are other or more effective ways of doing it. With this approach, I constantly see myself paying too much attention to implementation details and not really practicing concepts or tools too out of my comfort zone.
What you can do, though, is make sure you employ good principles in all the work you do. Try to make things simple as opposed to complex. Make code as readable as you can. Add logging. Have some form of testing to make sure things are working all the time.
This will get you 95% there. The remaining 5% you will learn once you get there.
Lastly, less than 1% of coders work or will ever work in such applications so don’t worry too much about it.
I.E if C# Api you're building, looking at and recording Application Insight metrics and load testing with something like jMeter would give you an idea of scalability and help you identify bottlenecks, etc.