How are crawlers going to work when Google-like search dies?

Question

When you no longer have page rank (counting the number and quality of links to a page to determine a rough estimate of how important the website is), how do you best serve results?Imagine there's a new topic to talk about, that has not previously been seen on the internet: how do LLM chatbots rank the correct information (if there's no search engine to rank it for them)?

mrkramer · Accepted Answer

>Imagine there's a new topic to talk about, that has not previously been seen on the internet: how do LLM chatbots rank the correct information (if there's no search engine to rank it for them)?They follow trusted source? For example, would you quote WSJ or some random site? But imo PageRank was powerful in finding popular and somewhat good information but it was bad for discovering new content and new websites and that's why Web aggregators like Reddit began to thrive.