Server-side HTML Templates based on DOM?

Question

For side projects I've used BeautifulSoup in Python to break down HTML documents into the DOM, manipulate them at the DOM level, then serve them to a browser.For instance the "main template" for a site could be an HTML page that has a inside it which is a placeholder for the content. The content is itself an HTML page, and the of the content is inserted into the , also the metadata gets merged so the passes through.You might write First Name: and the system would insert the firstName into that span, for instance.I've done the same thing in Java with tools like jsoup and HTMLUnit.This system can rewrite links and do other interesting transformations on the guest HTML.One thing I'll concede is that this is slow compared to the alternatives. In a crude test a system like this and not optimized might be able to transform 10-50 pages per second on a core whereas it is easy to get 3000+ with string based templates.I haven't seen other people do this, are there any examples you of know of DOM-based templating? Any thoughts?

fabianholzer · Accepted Answer

If you consider static site generators as a special case of side-server rendering, then that is basically the principle that underlies my personal ssg [0].Highlevel idea: It reads in all HTML files, parses them with JSDOM, builds a metamodel of the overall website, which (ab-)uses standard HTML meta tags as alternative to a frontmatter (e.g declares the particular file as the template for the category articles, while declares a particular file as part of the category articles. And then it uses a non-standard HTML tag to call JS plugins with the whole meta-model of the site, so it can plug together content and template, rewrite parts of the DOM (for example add bidirectional links when desired, run a syntax highlighter on tag, count words). When all plugins have run the DOM is serialized into HTML. Admittedly it could be a tad faster, but for my purposes it is good enough. I don't mind running a ton of JS on my machine, but still can serve mostly static HTML with very few sprinkles of client-side JS[0] https://github.com/fnh/intertwingle/

sntran · Answer

Cloudflare has a thing called HTMLRewriter that is both a streaming HTML parser and selector for changing it. It's very efficient and let you change only the places you need.For your example, you can just match in `[id^="insert"]` and add the actual firstName value in the handler.I use it alot on the server side, but it's for JavaScript side, or Rust if you use the lower level lol-html.

nchmy · Answer

I'm not quite sure what you're doing, but I wonder if a new-ish browser api - Declarative Shadow DOM - could be useful here. It allows for sending a sort of html shell to the browser that contains pre-defined placeholders, and then async sending the html fragments for those placeholders. No JS required.It recently got support from all the major browsers.- https://developer.chrome.com/docs/css-ui/declarative-shadow-...- https://lamplightdev.com/blog/2024/01/10/streaming-html-out-...I'm also considering trying something similar to you, and thought that perhaps GoQuery (essentially jQuery written in much faster Golang) could be useful here. https://github.com/PuerkitoBio/goqueryIf you need to do the scraping stuff, Colly could be used (which uses GoQuery under the hood). https://go-colly.org/You could also create a sort of hypermedia "single page app" if you serve the shell once, and then use something like HTMX (or even jquery) to swap the body and other html fragments without a full page reload.

conradfr · Answer

Is this really different than Jinja2 or Twig with template inheritance and blocks ?

MatmaRex · Answer

XSLT would be a classic answer, but it's a bit of a lost ancient technology these days. I've only used it in the browser for a lark, but I suppose you'd find mature (old) libraries for working with it easily enough.PHPTAL is another system I've heard of (but never used) that seems to be built on this idea. It is also very mature (old). https://phptal.org/In general working with DOM doesn't seem to be very cool these days. Everyone uses template systems that are glorified string concatenators, like Mustache, and doesn't care about security issues due to bad escaping.

solardev · Answer

I'm not sure what the advantage of this is vs any standard server-side templating (PHP, Twig, Mustache, Liquid, etc.)Converting from a string to a DOM is very expensive because of all the non-XML non-standard cruft HTML accumulated over the years. If you can do a simple string find/replace instead, it should be less computationally expensive. No reason to actually parse the HTML unless you need to actually render it or at least work with it semantically/structurally, which doesn't sound like the case here.

TheRealPomax · Answer

How does this differ from using a templating engine where the server gets a template, and instantiates it to "the final thing it should be" on a per-request basis?(also, if you need to transform 200 requests per second that's slowing things down, there's almost certainly a cache-layer missing?)

fzzzy · Answer

I made a web framework a long time ago that did this. https://twisted.org/documents/8.2.0/web/howto/woven.html

mk12 · Answer

The work-in-progress Zine SSG project https://zine-ssg.io/ calls its templating Super and it works just like this &mdash; directives are written in HTML rather than on top of it.

newzisforsukas · Answer

You would likely use other attributes aside from id, and there are a large number of JS based templating frameworks that do this.https://github.com/Floofies/cdaTemplate

rbanffy · Answer

The Zope application server uses something like that, ZPT (Zope Page Templates) and TAL (Tenplate Access Language), IIRC. Plone, a CMS built on Zope uses them extensively.

cxr · Answer

> Any thoughts?10&ndash;50 pages per second is already more output than a human can produce input for, so it shouldn't be considered a problem.

frmdstryr · Answer

enaml-web does this using lxml. It works fine for a limited number of concurrent users.

delanyoyoko · Answer

Probably raw Svelte can be used as a frontend template to your backend

cynicalsecurity · Answer

PHP.

noahlt · Answer

The end of this road is React's Server Side Rendering and Next.js.This approach only really has one major benefit over string-based templating, namely, it lets you re-use code across server and client side DOM logic.