Is there an opt-in search engine for the World Wide Web?

Question

Related to previous articles about Fediverse/Mastodon search[1][2]. Assuming that one comes to the conclusion that ethical search engines should be opt-in, and that there is no ethical difference between Fediverse and web search engines, I'm curious if there are any efforts out there to create an opt-in search engine for the web. Such a search engine would not index any pages, unless they are submitted directly, or have a machine-readable attribute (e.g. robots.txt, "yesindex") that explicitly allows indexing.I'm also open to objections to either the former or latter assumptions, though that is slightly off-topic.[1] https://news.ycombinator.com/item?id=34221851[2] https://news.ycombinator.com/item?id=34400143

verdverm · Accepted Answer

Most people want their content discoverable so defaulting to indexing would seem preferable as well. How would a search engine get people to opt in? Why would web admins take the time to add a fringe search engine crawl requirements to their site? Basically the search engine you describe would have a really hard time getting started