Apart from Google's monopoly and the big technical challenges, what would a competitor need to defy Google?
I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible.
You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine.
Edit:
Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search.
We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.
Too much of the up-to-date high value information is posted onto semi-public channels. Like facebook groups, instagram posts, telegram channels, etc.
At the same time, too much of the publicly available websites are becoming clickbait and mindless marketing drivel.
Just try to research anything slightly obscure that has to be up-to-date information (from the last few years).
Like "how to get from Nairobi to Kisumu by bus". Google maps doesn't have this (not surprisingly), but what about the SERPs? Nothing there. 90% of the results are useless bus ticket sale websites that have no info, just SEO. (You can't buy this ticket online anyway).
Is Kenya just such an off-the-map destination that no backpacker has ever written about taking this route? No!
The information is out there. You'll find it inside the related facebook groups for travelers in east Africa. And if not? That's where you'll ask. But the answer? It'll never reach a SERP in the future.
Even more sad, is that if you keep looking on Google, you may find buried results from years ago. From travel forums and such. What's sad is that the info will be 10 years old, since no one uses public forums anymore. (Unless it's on reddit, pretty much the last public forum with any reach).
One thing a competitor needs is a new and innovative technical algorithm.
Back in 1998, Google's PageRank was an innovative algorithm that calculated relevance based on counting back links instead of parsing the word counts in embedded HTML text like other search engines. This created a noticeable improvement in quality of results.
Nobody seems to have The Next Big Idea for a better search engine yet. Somebody did a Shown HN of a new search engine based on whitelisted curated domains such as reddit discussions. But there are many technical problems with that (e.g. Goodhart's Law & Hawthorne Effect creates bad feedback loop of gaming the reddit discussions which then poisons the search engine.)
Another technical idea of crowdsourced decentralized search index creates a very slow query engine which is a hard sell when web surfers are used to Google results appearing in less than 1 second.
DDG's idea of "privacy" is interesting, but being (mostly) based on Microsoft Bing's search engine doesn't actually create a quantum leap in better search results.
What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?
You can't compete with that by building a better search engine. Even if you had the infrastructure, capital, and you offered Apple and Mozilla more money than they get from Google, Google could beat your offer.
The only way someone could beat Google, or even just compete with them, would be if they could make everyone understand that they can change their default search engine and give them a reason to do it and have Google screw up how they respond. I wouldn't bet on it happening.
I also think that's why it's very hard for competitors to compete. As soon as one is going to get credibly menacing they'll improve on the quality again to preserve their position.
I think that has happened with McDonald's for instance. A few years ago the quality was at it's lowest, the food definitely started to feel fake, air filled and not satisfying. Since the small local competition of hand crafted hamburger popped up a bit everywhere, and credible international competitors ramped up (5 guys, in n out...) they had to improve the quality. And my impression is that it's been a bit more than a year that the quality dramatically improved, and their hamburgers actually taste like food again.
I think it's simply the quality versus profits cursor and it can be adapted any time.
Beyond that, Google’s dominance isn’t going away due to their own “monopoly-like” positioning, their business relationships, the inertia of their massive public adoption, all of these applied within three or four other vital areas (ie: YouTube), their ability to pivot in response to anything novel that appeared to undermine their position, no apparent stories of Google executives hosting puppy kicking parties for the entire company to Satanic Panic everyone away from their products, the problem of promising companies being acquired because the owners (VC or founder or otherwise) are happy to be bought, the general “ick” factor of someone like Facebook attempting to enter the fray, some other things none of us have ever considered, random luck, and the initial conditions of the universe.
You have to have good results against the inconceivably vast amount of content out there, you have to be easy to use, you have to be free, and you have to be able to do all of this and more for the yeeeeeears it would take to wiggle into the space and expand while resisting the pressures above. Doable, wildly improbable.
There is an obvious problem here - companies who don't advertise with Google could easily find themselves blocked from appearing on the first pages of Google search results. Would anyone really be surprised to find that the internals of the Google search algorithm have a weight factor that assigns better scores to sites that advertise with Google? Hence requiring Google to expose the internals of the algorithm to public scrutiny really seems like the only way forward. Yes, those are 'trade secrets'. So are the contents of your favorite hot sauce, but government regulators require those contents to be displayed on the product label (for good reasons).
On top of all that, Google's also under pressure from governments and media corporations to push their information content (aka propaganda, influence, etc.) to the top of search results, burying anything like independent content in those areas (world events, domestic politics, etc.) far down. This is particularly obvious on Youtube incidentally, but Google has the same problem. Some of this can be avoided using the 'verbatim' option and some interesting word choices in your search string, but it's a fairly tedious exercise.
Again, exposing the internals of the Google search algorithm would be interesting here, as it seems clear certain 'authoritative sites' are assigned better scores in the search ranking - not because they have more backlinks or more accurate content, but merely because of governmental and media pressure.
https://www.quora.com/Is-it-possible-to-beat-Google/answer/M...
A few competitors such as you.com are in the works. I think there’s a lot of potential, but the ranking algorithm is going to need to be decentralized (aka user-configurable) in some way. This configuration would also need to be sharable, maybe something akin to an App Store, because not all users can be asked to be technical enough to configure everything.
There is an enormous amount of wasted value stemming from the disconnect between what Google thinks the user wants to see, and what the user actually wants to see.
If I had to guess why this happened, I’m betting that Google and others had a notion that AI and other advances could eventually approximate a user’s needs to a degree that would keep up with the demand. But the monopoly set in and there was no longer an incentive to innovate that far. I’m sure Google tried, but not to a degree that would have happened if the competition had really stepped it up, or even existed. However, I’m sure by now there’s less optimism that the above effort is even a solvable problem, or one worth solving. One-size-fits-all is extremely suboptimal, but that paradigm is Google at its core. Without a paradigm shift, it’s diminishing returns all the way.
What Google is good at, however, is monetizing what they do have. If you’re trying to challenge them in the monetization game, with the same business model competing for the same customers (advertisers), you most certainly will lose. Seriously, don’t do that. Get to unicorn status first.
Focus on the tech, make sure the business model isn’t competing with the giants, and say no to any buyout offer from Big Tech.
We assume people in general are dissatisfied with Google, because there are blog posts about it, and some people on HN agree, and because we have certain strongly held beliefs and technical knowledge that lead us to this conclusion. What we have not validated is whether people outside the tech community are as dissatisfied as we are. If they aren't willing to switch, then there won't be a notable competitor to Google.
You're going to need a quantum leap improvement over Google to unseat their positioning. It has to be very substantial to overcome all the various moats they have, not least of which is consumers being used to using Google, the brand awareness.
The next great search engine will emerge from a niche and conquer one segment after another from there. It won't be a massive general search engine that shows up one day (which is what the Google watchers have been waiting for forever - that new behemoth comprehensive competitor is never going to arrive fully formed). There's a decent possibility consumer Web search will be a later stage addition to said new niche competitor, consumer Web won't be its primary or initial target. They'll add on general consumer Web search as a "we might as well" offering once they conquer enough niches.
I realize I am a sample size of one, but I run this exercise about twice a year, and sadly always go back to Google.
As much as people claim otherwise, it's not that Google has no competition. People are trying! Google's product is legitimately good, despite the widespread perception of declining quality.
https://www.justice.gov/opa/pr/justice-department-sues-monop...
The only thing that would have a chance is if Apple developed a search engine and used it as the default instead of using Google. People are busy/ignorant and do not care and will eat whatever search results they are given.
But this might be a risk to Apple based on how many people use Gmail on their Apple phones.
It is not about quality anymore, it is only about market domination. The only thing that will bring about new search engines will be anti-trust legislation.
If it's not covered on HN, I'll append "from:reddit" instead (they're not always wise but as at least they're not getting paid per word written, and the upvote system gives me a sense of what real people think). If I want a broad survey of a topic, Wikipedia or some other wiki are usually great. If I want a really deep dive, I'll append "book", usually in conjunction with "from:news.ycombinator". Lastly, I sometimes add a time filter, e.g. "after:2021".
Clearly this is an awkward pattern. But it really works, and I'm having trouble thinking of something that would be meaningfully better. I suspect most people have found ways of using Google that work well enough for them. So I'm afraid Google Search will stick around for awhile.
I tried DDG for a couple of weeks and it was just awful. For almost all non-trivial searches i had to result to !g. I finally switched back after two weeks or so and suddenly I'm much happier with the quality of Google search results.
Obviously this is just one man's opinion.
I think the technical challenges are big enough to make this a really difficult task.
If you're willing to pay for search there is an alternative emerging: Kagi. It's not a Google competitor, though. It's a niche product for people (like me) who are willing to pay a significant amount of money for access to a search engine whose creators make money by providing value to their users rather than by providing their users to advertisers.
Secondly, the media would not allow it because many top media executives own Google shares. If a better Google competitor came along, media companies would do everything to suppress information about it so that nobody would know it exists.
The declining quality of Google results seems pretty clear, but only in relative terms. That is, Google now compared to Google in the past. In an absolute sense the service remains pretty great.
A confounding factor is that maybe the quality decline is in fact in internet content in general. That is, Google per se is as good as ever, it's just it's become much more difficult to find good results.
So, simply, the answer is perhaps that no matter the effort a competitor could put in they simply cannot outdo Google at their core competency. Perhaps for fundamental reasons, but even if not in some niche, surely then for sheer funding and scale reasons when trying to expand beyond that niche.
If a competitor to Google emerges, it'll be in some non obvious thing that is not core web search. Basically, when core web search becomes less important and Google can't pivot quickly enough to the new more important thing to compete.
Managing to grow enough in an unappreciated search niche while staying under the radar in order to better withstand that kind of eventual attention seems unlikely. Targeting a segment that Google has been burned on (like social search) may help, but that probably just draws attention from a different MANGAM.
Eventually, someone will be both smart and lucky enough to carve out some of the search space, but don't hold your breath, it is going to take so much luck it won't be soon, and may look accidental.
Remember, search is not just web search. It includes vast array of things such as maps, images, videos, knowledge datasets, discussion groups, real-time news and so on. Each of these segments is an enormous effort on its own requiring massive capital investments. Relevance algos these days are mostly driven by thousands of tweaks, ML models and legions of rules. It’s a complex beast that takes thousands of PhDs and years to perfect. Even than performance is pretty subpar than most expectations. So, what chance do you have as a new entrant in same game? The key is that you don’t want to play same game if you want to win.
A lot of companies have came and gone announcing themselves as Google competitor and trying to play same game. Things like DDG survive only because they can lower the COGS by offloading real work to other people like Bing which itself is fine example of what you might be able achieve if you only had a tiny fraction of Google’s budgets. If you consider dollar for dollar capital investment, Bing is actually quite good.
This is not to say there will never be a viable competitor for Google search. I just don’t think it can be through the traditional framework of index serving. No one really knows what other alternatives can arise in future. One very possible thing is language models. If we can figure out how to scale up serving of massive language models that effectively “memorizes” whole index in them, they can provide quality, capabilities and experience that cannot be matched by simple index serving. If this is viable route, I think we are still at least 3-5 years away.
Google owns one of the major mobile platforms, where it is the default search engine. It pays competing Browsers like Firefox to be the default and is so tied in public perception that "to google" is a verb.
Also Google is fast and mostly reliable.
Also Google search integrates with other services like Maps for localized search so that one is torn back to them easily.
And if you were to get close to it they have tons of money to fight you.
The interesting aspect is that outside the broader search domain their approach often enough doesn't work. They didn't get social networking, they didn't get messaging (except mail) ... so they way to beat them likely is to find new segments and occupy that space (like Zuckerberg wants to do with "Metaverse," whatever that shall be)
If we are talking about search then it will likely be something AI based probably operating under a different model than Google search. AI is a huge force probably akin to the Internet.
Today, it’s totally possible for a single person or a small team to build a domain specific search engine that indexes 10s or even 100s of million documents.
Building the index v0.1 that kind of working is not hard. But maintaining the index to handle countless edge cases is tedious and need non-stop investment (eg, infrastructure + paying salary for talents)
Also, a search engine is more than “keyword matching”. You need to do search result ranking. And it gets exponentially difficult to do as # of indexed documents increases.
1. Data about what the population is interested: More people use Google today, giving Google more data to train its AI with.
2. Server capacity: It's cheaper for Google to store 1 image than for me to store 1 image, because it stores many more images. Same for compute, such as training AI. Google can get one A100 GPU cheaper than I can.
3. Ads: If I had the exact same search quality and even the exact same number of users, Google will make more money from the same operation, because it has more advertisers.
They set up their business well!
Returning blue links is a thing of the past, as the Web of yesterday is long gone. Blue links always were about surfing i.e. following hyperlinks just for the sake of it since the main premise was most of them were of high quality and quickly proliferating.
All that is gone now and the links are a promotional thing how to get paid in one way or another. This is why Google results have been deteriorating, regardless of tens of trillions of archived pages on the Web. Google has lost the principal ranking signal years ago.
The next huge scale smart information system will be based on dense vectors (a few hundred dimensions) such as in AI but the key will be much bigger scale, of (tens of) billions of vectors. Contemporary AI works won datasets 4-5 orders of magnitude smaller, getting bogged down in gigantic transformer models such as GPT-3 with 175B+ parameters that take weeks and millions of dollars just to train. One might wonder what is innate knowledge of such a huge model, and it is not much as one can see for themselves as GPT-3 is now open (until Apr 1).
The future will be based on embeddings that are NOT contextualized i.e. no separate vectors for different senses in superpositions. Such systems will not be based on ads nor tracking as the resources required will be orders of magnitude less than what is currently required at Google.
Speed of access, an Internet browsing engine , contracts with non-GGL browsers, cloud data centers, maps/locations/navigation, fiber cables/internet pipelines, social networks and personal management software, AI/ML research, personal devices OS ...
Impossible is nothing, but any incumbent will likely take in capital linked to Google. GGL is dominant across the world's hundred and something countries. Excpect places where it is banned(China) or shunned(Russia), anywhere it is equivalent to a monopoly of internet services.
From a technical perspective, a challenger can rise up. From an economic standpoint, a challenger would not last long (maybe locally if favored by regulation). Even the most successfull others: Apple or Amazon would not dare take on Google's Ads or Search Engine. Microsoft tried with Windows Phones and Bing, then ended failing to dominate the market. Google is not singular business. It has so many tentacles in the tech sector that wave a web of interlocking tech products. Growth numbers in Search are boosted by integration in all of Google's own products and the rest of internet users and companies.
To answer your question is to figure out what advantage it has on any competitor starting from zero... Sadly, its network scale is too large to allow an equal challenger.
Setting aside all of the specifics related to building and running the engine itself, how do you make money doing it? Unless you are servicing a very specialized and lucrative vertical that doesn't mind paying, everyone else gets their search results for free so you generally can't charge users for it. This typically leaves advertising (which Google arguably owns the market for re: search) and/or selling user data (which is both unpopular with users and depending on jurisdiction, illegal). Figure out a new way to monetize search and you may have something...
This is a problem with today's world dominated by mega-tech companies: to compete in many areas of it you need to essentially be a monopolist in some other area so you can afford to compete in something like search as a loss leader at least until you get established. Look at how much Microsoft spent on Bing[1]... and they got how much of the market? Facebook wasn't initially profitable either, but they had a distinctly different approach to aggregating a pool of data Google couldn't search and users they couldn't monetize.
[1] https://money.cnn.com/2011/09/20/technology/microsoft_bing/i...
* Google (and all of FAANG) have a stupid high percentage of the worker pool.
* Google's biggest asset is that they are Google. Kind of like competing with Coke. Intrinsic quality of what they do is kind of irrelevant.
* General lack of value in most things internet. Google is getting a lot of value out of what they do, but they are having to put stupid levels of manpower into squeezing out that value.For instance how can you offer a better maps for PND if you can't elicit traffic data from the density and speed of android smartphones that are around of 80% of phones traveling around the world?
To defy Alphabet you do not need a better Google, you need a different kind of solution, for instance instead of competing in modern web crap to lock down users and surveil them you can propose classic desktop computing with decentralized tools. Try looking for instance at Jami, Retroshare, ZeroNet, you can integrate them in a suite and say "hey, instead of depending on Zoom, Meet or Teams, proprietary services with limits that might change, surveil, etc use this system, there is no SPOF, no service behind".
Some have tried something, for instance DeltaChat seems to be a WA clone, but it's actually a MUA, in that case it doesn't took off much because most people simply do not care, you have to know your public. Starting from CS and humanities courses [1] to attracts students and plant the seed of something new, being prepared to face years before a success simply because no empire born quickly and when some are against you it's even more complex.
The old adage: people have the power, but they do not know how to use it it's unfortunately very true, you can only bend people, use them, to elicit a slice of power :-)
[1] most CS students haven't enough skills to comprehend the world, they just explore few aspects ignoring the rest and that's why dictatorships like STEM, because they help to generate "ancient Greek's 'useless idiots' to be employed" instead of Citizens (disclaimer: I am an engineer, I learnt than personally in years)
There will be a dramatic change in how people ask questions and access information.
Maybe it will be personal assistants. I doubt it.
Perhaps these kinds of Neuralink implants? Could be...
Whoever nails it, will be one of the near-future trillionaires.
Also while the quality of search has declined it is nowhere near the point where casual users would be bothered enough to switch
If bing was successful at it, for example, it would cause the sentence "I tried to google it but ended up needing to Bing it" to make tangible sense to someone who has used both services.
Even if you create a better search engine, there is too much of a marketing gap to compete on that alone. "use bing because the results are better!" simply doesnt motivate enough people to switch off google.
Quite frankly, it is rare that google does not deliver expected results for the generic use case.
If you have a great idea and access to deep enough pockets to develop it, there's still the problem that much of the information people want to find are behind various walled gardens and not suitable for public search. Incrementally more helpful arrangement of the same publicly available data that Google already has, isn't worth the effort of challenging the behemoth
Google’s death will come not from some upstart search engine, but from many walled gardens that each have perfectly good topical search inside them. We’re already in a case where different kinds of information are checked at different destinations.
Google will keep trying valiantly to more aggressively turn into a knowledge base instead of a search engine, and ultimately will probably settle at being an interactive encyclopaedia.
The internet has moved past search engines. Now if you want information you need to know or figure out where to ask the question.
Can't just buy time on someone else's cloud network and hire a couple of coders to compete with that.
You could throw a lot of money at it but not compete on years of investment.
My brain is swiss-cheese these days, I wish I could remember the name of that one-man startup a decade ago that actually had a lightweight high-speed crawler and was making a serious attempt at competition. Was covered around here several times. But obviously they didn't succeed if I can't remember the name.
A competitive business model. You can create tech that is easier to use and more powerful than Google's services, but you won't last unless you actually make money.
I think this would allow a competitive market for indexers in different verticals to emerge.
Most probably you wouldn't want to use it for every small query. But in situations when you need high quality results in some domain, the p2p marketplace would be a good choice.
Probably the only hope is some type of peer to peer search engine, that actually catches on and works well.
Habits and ignorance. The casual web user is just fine with what Google has to offer.
And besides that: Google is convenient and everywhere. To get rid of it it requires not only the will but also the technological skill that the common user just does not have.
And that leads to the answer of your question: Create an alternative that is easy to use,ofcourse, that can be implemented by just "a click" and is has to be everywhere. On your phone, TV and computer.
Personally I wouldn’t mind a search engine that was focused only on code results. Not only results that that showed code, but code focused results - tutorials, samples, books.
For example, if I search for “c web server” in a private tab on on Google… well… I get exactly what I want. If I search for “water shader”… I mostly get what I want as well (youtube videos not code though).
Hum, I think you might have a hard time with this.
Building a general search engine better than Google is probably a non starter, but...
Could you build a better image search for graphic designers? Could you build a better search for specific document types, such as academic papers or legal precedence?
I think that's what we'll see - similar to how nothing replaced Craigslist, but a bunch of startups unbundled features in Craigslist and made companies around them.
I slightly better experience should get a larger portion of new users, but existing users do not say "Today I will develop a methodology to conduct a survey of internet search engines and select the best one for me."
At this point everyone has either chosen a search engine or had one chosen for them.
Which monopoly? Email? Search?
Regardless, the answer is time. It took Google a decade to win the search war completely. It will take a competitor time to dethrone Google.
Google "solved" the internet search engine problem. Other competitors are just replicating it poorly. There's nothing good for the consumer here.
https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...
TL;DR you need a lot of money and users (catch-22). Also, massive scale and massive noise.
You can also search HN for 'cliqz' for discussions about Cliqz shutting down.
Check it out, it's great!
I've been attaching url:reddit.com to my Google searches to bypass all the blog spam thats taking over Google.
Presearch decentralized search engine, powered by blockchain technology.
Bing isn't the perfect google replacement as it seems to have gone the google route of not returning hits for things you search for, but for things it thinks you wanted to search for.
Maybe the "site:reddit.com" hack works well enough for people who care about search results.
It's possible HN is an unreliable guide to Google's actual prospects.
A competitor would need to avoid being bought by one of the tech oligarchs. That's pretty hard to avoid.
If by some fluke, you find a breakout opportunity, you would need to somehow avoid getting crushed like Parler. The tech oligarchs will use any excuse to crush any competition.
That episode basically shows that to compete you would basically need to build your own internet first, and also your own banking system.
These are systematically entrenched monopolies of sorts.
Google is hiding important information on purpose... so don't do that.
With so many bots, a lot of sites run a whitelist process of user agents, so every new entrant is automatically blocked in those cases, and for the rest, any large scale crawling is quite likely to get blocked.
Anecdotally it seems a relatively new search engine entrant is using Chrome user agents and follows Googlebot robots.txt directives so are impossible to block. If true, that's an absurd way for a new player to enter the market and just annoys webmasters.
A recent anti-competitive report by the UK government estimated that it would take about $20bn dollars for a new entrant to compete on Google's level.
Then there's how Google pay Mozilla/Apple very large sums of money based on their ROI/search which no one else can compete with, partly due to the invasive surveillance capitalism that Google follows. That ultimately means that anyone who wants to compete has to pay browsers and operating systems to enter the market in a substantial way.
All that said, encouraging people to find alternative sources of information is most likely the best route to diversification with or without government(s) intervention.
By the very nature of solutions: a new company cannot beat google at general search. Google was and is search.
Like a tree that has grown up and shadows the entire land around it. There will be random small pockets of sunlight with smaller plants in it. But the big tree owns the area.
There can be a short term competition via extremely specialized search engines, but they will not rise to the same dominance. Google, in a way, is the entire idea of searching the web.
The next dominant generation has to be from a new paradigm, that makes web search obslete.
---
A relatable current example could be: youtube being dwarfed by tiktok. The nature of the new thing is such that it very naturally dethrones the old thing, without directly competing with the incumbent.
There will be specialized providers like vimeo, But at this point, youtube IS the idea of video on the internet.
---
Just note the context of this understanding: This understanding came from struggling to change the education system of my country. And then it seemed to apply everywhere I saw.
I'll round-off my reply with this quote:
“You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.”
– Buckminster Fuller
https://twitter.com/yegg/status/1501716484761997318
Not that there are many great alternatives, but the whole point of a SEARCH ENGINE is not censoring results. I understand ranking makes it searchable, but omitting views / results based on politics is self-reinforcing and makes anyone using the service blind to views and in a sense reality.
I think search can’t really be “improved” until it’s decentralized and uncensored. The quality decline IMO has to do with the social networks and “trusted news sources”, which I also think is a walled garden limiting accuracy of info.
See “trusted news initiative”: https://www.bbc.com/beyondfakenews/trusted-news-initiative/