Building a Better Google

Question

If you had the 4 billion pages Google had on its index in 2004 in elasticsearch today, what would it take to replicate 2004-era web search quality using that static index?What relevant research or projects are trying to make these sorts of algorithms and data accessible as a future commodity people can build on top of?

immortal3 · Accepted Answer

I have been thinking about these a lot lately. There is lot more scope on improving current search results.
I believe simple two stage system might be enough to produce decent system. Stage - 1. query expansion and reverse index based retrieval Stage - 2. re-ranking based on few combination of heuristics. (page rank + word embedding + query analysis)
Would you like to talk more about this ? I have email address in my profile.

mikewarot · Answer

I'd use word2vec to embed all the words into a vector space to try to get more inclusive search results.I'd also try to separate out multiple meanings of phrases if possible... for example "hypertext markup" could mean the HTML language, or it could mean actually marking up hypertext (annotation). I'd let the user have some way to disambiguate the meanings.

ipaddr · Answer

In google cared about: - title, description meta tag - PageRank was just starting - Single word matching domains to search terms - Anchor TextThe quality of the search algo wasn't better the sites indexed were better.

blackcats · Answer

First I would ignore bad backlinks as they&rsquo;re not a good indicator of quality. The better your page, the more likely a competitor will start creating thousands of bad links.

Building a Better Google

In google cared about: - title, description meta tag - PageRank was just starting - Single word matching domains to search terms - Anchor Text
The quality of the search algo wasn't better the sites indexed were better.

First I would ignore bad backlinks as they’re not a good indicator of quality. The better your page, the more likely a competitor will start creating thousands of bad links.