Is the web for machines (/llm.txt) the one we wished we had as humans?

Question

I got really tired, as a human, of parsing the standard marketing heavy web we have today. I've always loved the simplicity of gopher and gemini web.Recently I found myself manually adding `/llm.txt` to most websites I visit because I find the content for LLMs strait to the point and clear. The only annoyance is web browsers like chrome do not render the markdown.So could the AI revolution actually fix the web for humans as a side effect?Do you find yourself doing the same?

realty_geek · Accepted Answer

What is an example of a site with a good llm.txt?

ahriad · Answer

We broke the web so badly for humans that we had to build a clean web for machines, and now humans will have to use machines to experience a clean web again.

cyanydeez · Answer

oh don't worry, in 5 years your AI will be unundated with context poison prompts that try to get them to spend all your bank notes and meta bucks on equally useless things.This is just a redeux of the early web.

marand23 · Answer

I never thought about it before now but the llm era could be a form of renaissance for blind people on the Internet. An alternative web where functionality of every page is described in short but detailed text instead of extremely verbose and non-linear html tree structure.

rickette · Answer

Does any of the LLM providers actually use llms.txt?If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.

skywalqer · Answer

Why didn't they place it in .well-known? Also, I couldn't find a website that has it.

mohamedkoubaa · Answer

It just hasn't been gamed yet

onion2k · Answer

The only annoyance is web browsers like chrome do not render the markdown.I imagine Claude could zero-shot a Chrome plugin for that.

gobdovan · Answer

Not really, but sounds interesting. Would you care to share some sites that offer better llms.txt than main web page? Or talk about some piece of info you easily found on llms.txt that was hard to navigate to on the regular website?

croes · Answer

No, the spammers are just at the beginning of ruining that toohttps://news.ycombinator.com/item?id=48411569BTW why should Chrome even consider rendering a .txt file as markdown?

tacostakohashi · Answer

Pretty much.There is an enshittification cycle at work. The web used to be good, predominately text, and useful, 25 years ago. Then... slowly... we added javascript, then AJAX, CSS, flash, interstitials, popups, marketing, social media, algorithms, doomscrolling... gradually but surely turn it into the unusable cesspool that it is today.Now we have AI! I think a big part of its utility is that it gets us back to text/information, and lets us bypass all the "beautiful" design / nonsense on the material it is trained on.However, AI is just beginning its enshittification cycle - now that it has a critical mass of users, it is an irresistible target to start slowly adding ads, misinformation, conspiracy theories, and whatever else people can dream up, until it also becomes unusable and the cycle repeats.

DeathArrow · Answer

I tried it: https://news.ycombinator.com/item?id=48410589`/llm.txtResult: no such item.From where do you got the idea that adding /llm.txt to urls will produce markdown?

Is the web for machines (/llm.txt) the one we wished we had as humans?

What is an example of a site with a good llm.txt?

We broke the web so badly for humans that we had to build a clean web for machines, and now humans will have to use machines to experience a clean web again.

oh don't worry, in 5 years your AI will be unundated with context poison prompts that try to get them to spend all your bank notes and meta bucks on equally useless things.
This is just a redeux of the early web.

I never thought about it before now but the llm era could be a form of renaissance for blind people on the Internet. An alternative web where functionality of every page is described in short but detailed text instead of extremely verbose and non-linear html tree structure.

Does any of the LLM providers actually use llms.txt?
If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.

Why didn't they place it in .well-known? Also, I couldn't find a website that has it.

It just hasn't been gamed yet

The only annoyance is web browsers like chrome do not render the markdown.
I imagine Claude could zero-shot a Chrome plugin for that.

Not really, but sounds interesting. Would you care to share some sites that offer better llms.txt than main web page? Or talk about some piece of info you easily found on llms.txt that was hard to navigate to on the regular website?

No, the spammers are just at the beginning of ruining that too
https://news.ycombinator.com/item?id=48411569
BTW why should Chrome even consider rendering a .txt file as markdown?

I tried it: https://news.ycombinator.com/item?id=48410589`/llm.txt
Result: no such item.
From where do you got the idea that adding /llm.txt to urls will produce markdown?