Why hasn't the industry just switched to better solutions yet? Is it because PDF is the thing you get regardless of the typesetting program you used (e.g., Word, Latex, Markdown, HTML->PDF (Save as PDF), etc.)? It seems to me that there must be a better way.
A PDF file is essentially pre-rendered. So the application creating the PDF file needs to do all the stuff mentioned above and the PDF itself just contains the instructions at what exact position on the page which glyph should be rendered.
This makes displaying or printing a PDF much easier (but still a hard task). And that is also the reason why editing PDFs is hard because all the additional information like what is a paragraph, a heading ... is usually not available.
FYI: Tagged PDF has all that structural information and there are developments to allow e.g. reflowing of PDFs on smaller devices.