HACKER Q&A
📣 person2718

Which search engines do you use to do exhaustive research on the web?


I have to do extensive research on a number of potential variations of a product that I want to build and market. I will have to be searching for physical products, companies, research papers and news publications that refer to those things.

With the Google Search service being so dissapointingly degraded and returning so much noise and limited results it's almost impossible to try and dig in all that garbage it returns without losing your sanity.

Scholar still works well enough, so searching for papers won't be that hard, but I really need some advice from people who do this often (patent office workers? Market researchers? Inventors and entrepreneurs?) because this difficulty to discover new information relevant to a subject is making my life very difficult.


  👤 contingencies Accepted Answer ✓
For general technical and commercial background of the area I would recommend Library Genesis with a wide range of related search terms, including to be exhaustive those in other major languages. You can seed the search query set from relevant Wikipedia topics, book publishing information page listed keywords, dewey decimal categories, or other mature collaborative semantic databases. Using these established semantic databases helps to ensure you don't miss category specific jargon thus undermining your results.

However, arguably the most important element of hardware products from a research perspective is the marketplace state with respect to competition and demand patterns. For this area, I recommend: (1) https://patents.google.com/ (relevant results of which can also be used to seed/re-seed keywords for the above). The amount of information people put in patents is ridiculous, you can shortcut a lot of wasted effort by learning from their mistakes. (2) Existing consumer product databases such as eBay, Amazon, or Etsy. (3) Searching Taobao in Chinese for both the product category (machine translate adequate) and the major functional components. This reveals a true state of costs and supply chain outside of any tightly held specialist manufacturing lines, which is often deeply informative with respect to product design decisions and explaining marketplace state. (4) Researching specific competitors: funding, timeline, team, location, announced marketing and business strategy. Job ads will also give you huge insight in to their skill gaps and technical stack / strategy. (5) Using google trends to identify search volume changes over time within your target market(s).

Hope that helps.


👤 kirubakaran
I'm working on this! https://histre.com/

Histre aims to help with the whole "knowledge funnel", if you will. My approach for the web search part of it is to heavily penalize (to varying degrees) websites with ads, tracking, referral links etc.

The web search part isn't done yet. You don't have an email address in your HN profile. So if you'd either drop me an email or signup to Histre, I'll let you know when this is ready.

An additional note re Histre: I find that all the knowledge base apps out there are "write only". Search sucks in all of them. What's the point of creating and maintaining KBs if you can't use it later? So I'm solving search first. Another way Histre stands out (I believe) is that, it strives to solve the upkeep problem, by making it trivial to keep your knowledge bases up to date.


👤 caleb-allen
Google Scholar mostly.

Sometimes I'll find myself using search from a publisher or library when seeking specific physical document.

That said, I've had the desire for a more robust document search engine many times. A bit ironic..


👤 vstuart
searx? https://persagen.com/2020/02/02/searx.html

I've been using it for a couple of weeks and I am quite happy with it.

Per your comments, I think anything that does a metasearch (not relying on a single search engine -- Google ...) [Dogpile? ...] will offer benefits.



👤 textread
Search Hacker News: https://hn.algolia.com/