How do you start researching for a project idea?

Some things that help me a lot:

- Go walk outside

- Consume media / go to events with topics that are VERY different than my field of expertise

- Do any kind of art

- Play like a kid

- Last but not least, my favorite: Have a lot of bad ideas. https://seths.blog/2018/06/the-two-simple-secrets-to-good-id...

If I was really truly stuck for an idea and could not get anything to pop into mind, here's something I might try:

Apply the "take an old idea and reinvent it in light of newer discoveries, new technology, or in application to a new problem" approach.

To do that, read about old ideas in some area of interest to you, that didn't catch on, or were abandoned for whatever reason. Use Google, Wikipedia, Sci-hub, old print magazines, old newspapers, whatever it takes. Look at things through a lens of "Could I resurrect this idea with a new twist that wasn't available when it was first proposed?" Do that for enough "things" there's a chance it'll spark something.

Extra hint: if you're looking for a startup idea specifically, Google for one of the multiple lists of "startup ideas that never work" that are out there. Of course it should go without saying to be very cautious about jumping into a project where many other people have failed with the same idea: there's probably a good reason for their failure. But IF you can find a new angle or hook that's "just right" maybe you can succeed. Or the whole exercise may actually bring a completely different idea to mind. Try to be open minded with this stuff.

Also, find the two video series[1][2] that Alan Kay did on "Inventing the Future" and watch those. He shares a lot of great ideas in those.

[1]: https://www.youtube.com/watch?v=id1WShzzMCQ

[2]: https://www.youtube.com/watch?v=1e8VZlPBx_0

I keep a lot of notebooks of general problems I run into and ways I think problems can be solved. I sketch out a bunch of ideas in a notebook and let it sit. I might do some research to see if it's been done before, and if so how, and if that solves the problem. If I keep coming back to the idea to add things, or see where it can be used, it goes in my queue of "things to do".

I prototype with basic materials (post-it notes, notecards, paper, etc.) to see if the idea is fleshed enough and has legs. I do this because a lot of my "failed projects" were just really not well-defined enough to proceed. It also helps build out the idea and helps illustrate deep-ends you should avoid in a prototype, and what an MVP might look like. Simple material prototypes help demonstrate the sorts of capabilities your program will need.

Once a project receives enough attention, I consider actually coding it up. I boil down the idea to the core of what's to be done, and try that. If it fails, maybe it wasn't a good idea, so try alternative approaches if I have any sketched out or any lessons learned from the initial attempt. I try to build up knowledge around the area of the project in-case it deserves a return in a few months or years.

If a project does actually work out, then I justify further development based on effort vs. what I need from the program.

The closest thing to innovation I do is scope expansion. I like to take things that exist, and look for overlaps. If 2 products are 99% the same, even if they conceptually have totally different purposes, that's an opportunity to apply the DRY principle

I also enjoy sci-fi shows and often think "Ok that should really exist by now". Sometimes you see something that really is possible with current tech, but it's mostly just tedious UI work and nobody has actually made it practical yet.

Right now I'm working on open source CCTV software. I am also working on some basic VJ type functionality and adding QR support to an automation system. These are in fact the same project.

Non tech projects are different, but with tech I mostly focus on what existing things are out there.

Then I look for anything vaugely similar and see if it could ve extended to cover the use case.

If I'm absolutely sure a new app is needed, I look at what standards are used in slightly similar things, what tools exist to use those, and go from there.

Basically all my projects have just been FOSS things that already exist in the commercial space. I'm not sure I've ever really done anything innovative. I just use more software tricks to work around and hardware and put more effort into UI.

But ultimately, everything I do has been done a bazillion times. I just kind of pick up where the state of the art left off and add some nice to have features.

I really hate custom code and things I have to maintain by myself, so my first goal is always to make my project fit the larger ecosystem with as little originality as possible.

On a related note, old popular science type magazines have tech from 50 years ago that somehow still doesn't exist. If you want a real world changing project, I see zero reason a consumer level raman spectrometer couldn't be made.

I have too many project ideas. What I do is look on forums on popular products and look for needs and wants of the user base.

For example: Look to see what people complain about with shopify and see if there a product you can make to fix those issues. If you do it well, even if it's a niche solution. You can build on it or let it make you an extra $100 - $1000 a month.

Rinse and repeat.

I personally think that scientific papers offer a lot of good ideas. They will often talk about what useful applications that certain mathematical concepts have and the state of the art of what can be done with a certain technology, you will be amazed how many applications there are for something as simple as linear regression. A lot of the times they will outright publish open source libraries that have achieved state of the art performance in certain specific applications and can be integrated in your own product demo.

Also, watch some lectures (and do problems if you really want to understand the material) from MIT open courseware on challenging subjects like Linear Algebra or Statistical Inferencing. If you can combine fields from web crawling/page ranking and linear algebra (like Google did), you could have a really great idea.

Alan kay has a brilliant [0] powerful idea about ideas :

- Complete the sentence "It would be stupid if in 20 years we didn't have ..." (flying cars ? direct brain interface ? biological self-healing computers ? pick one)

- Now imagine this 'obvious' invention and bring it back into the present (IIRC he was trying to invent the iPad and came up with the Apple II - and succeeded. IIRC he's involved in Object Oriented Programming, Apple II, TCP-IP and other revolutionary inventions)

[0] https://www.ted.com/talks/alan_kay_a_powerful_idea_about_ide...

Coincidentally, I've been thinking about this as well recently. Especially your second major question:

> If you completely don't have an idea, but want to work on one, how do you look for ideas?

(But also the one before.)

My line of thinking was there oughta be a community online where people can pitch their ideas and get opinions on it or look for collaborators, also cross-discipline. Like "hey I have this cool idea for an, idk, gardening app but I don't know how to code, wanna help?" or "I'm working on this thing, any ideas to make it better", etc.

From what I've seen, there are some forums like that on Reddit but they look more focused on momentary thoughts and whacky ideas rather than projects people are actually attempting. And it doesn't help that Reddit is mostly focused on recent posts (like HN or Twitter), so your post would only really be "active" for a few days.

GitHub could work like that in theory. You can push an empty repository with your idea in a README and people can use discussions to, well, discuss it. Tags could help others discover ideas to work on. But I haven't really seen anyone use GitHub like that, especially not people who aren't involved with IT in any way.

I guess what I would like to have is some kind of "Product Hunt but there is no product yet" (and with more useful discussions). Is anyone aware of anything like that? Or is this perhaps a niche that's not filled yet?

I have endless ideas so I can’t address the second part. But for the first part, To me the most important book for this ever written is “The Mom Test”. It is the only search book written with methodical, repeatable, reliable steps to do this research. I warn you, implementing those steps will be damaging to your ego but may also make you wealthy.

Not great advice, probably, but an answer to your question: I stubbornly try to rebuild the same thing ("a search engine for food"), over and over again, and each time learn slightly more and have better (software) tools and experience available to call upon.

https://www.reciperadar.com/ is the current version of the idea, and I'm fairly happy about how it's developing. Previous iterations have included a Java-based web application (using recursive common table expressions in PostgreSQL) and dabbling with a Prolog-based web service implementation.

The current containerized architecture (which is ridiculous, I know) provides for a migration path towards Rust or golang for more resource-efficient microservices, if & when that's ever useful (or enjoyable to work on).

I usually tend to think over what I (personally) would expect the project to have / be. Nothing more, nothing less. That keeps me motivated.

I started m4b-tool[1] for using it myself - and noticed, that the most features, I wanted it to have where not available in existing, free solutions. So I started to note things down, I would like to have for a utility to merge / create audio books.

Now the project is a bit bloated and I learned so much on the way to its current state, that I'm thinking over rewriting it (not in RUST, but in C# ;) but it was a joyful journey with a lot of new ideas and experiences.

Pro tip: Early shipping is very good for geetting early feedback and new ideas. Don't be afraid to publish something unpolished.

[1] https://github.com/sandreas/m4b-tool/

I would say never actually look for problems. Certainly, never start with the solution and work back to find the problem. Train yourself to stop and think when you encounter a difficulty. Learn to ask yourself if there can be a better (faster, cheaper, automated) way. If you have an answer, THAT is your idea

Keep a list of “why can’t I X?”

For example, “why can’t I hail a taxi from my phone?”. At some point, all the ingredients were there but it didn’t exist. Uber answered the question.

Things that you wish existed, especially if they are a combination of existing ingredients.

Then you have to ask “so why hasn’t someone already done it really well?”

There's a nice iterative process called Customer Development from the Lean Startup methodology. Use this to validate your initial assumptions, to fine out the Business Model (Value Proposition and the target audience) and generally understand pain points of your potential customers (how much does the problem currently cost them, how they solve it currently, what are the possible alternatives, etc.).

Hard to give a complete advice im just a few sentences, but basically this framework gave me the understanding necessary to quickly build out (validate) business ideas before anything is really developed. To oversimplify it - try selling before building it. That will give you a very good understanding of the potential market size, the actual needs of people and so on.

My coming up with idea process:

I have most of my good ideas when I have good mental stimulation. A diet of well written books is the gold standard. Talking to interesting people also helps.

My research process:

Fermi estimation.

Like when I started working on my search engine, I had a raspberry pi 4 with 8 Gb of RAM. I estimated that a webpage was 10 Kb. From that I figured a raspberry pi ought to be able to hold an index containing about 800 000 documents.

This is of course ignoring the fact that the index information is smaller than the document, and that an index requires metadata and operating system space. But like, ballpark wise it's not bad. I got a pi to index about 500k documents.

Then I did some basic research on data structures, both on what's typically used and just independent thinking.

When I had a sketch of an idea, I just go to work.

Not exactly answering your question, but I’ve realized that sometimes, “solutions in search of problems” can take up to decades before people realize they are actually solutions to actual problems.

On the flip side, tech people tend to place importance on using technology to solve problems that did not even exist a few years ago. Think most SaaS startups.

Whereas, I’ve now realized, many of the problems most worth solving, also affected people 500 years ago. There is a lot of noise today in our information society, but the persistence of those problems is true signal.

So if you want to work on an idea that may stand the test of time, perhaps start looking at the problems that withstood the test of time. These are also hard problems that will be very humbling.

I've been running a problem validation platform for over 2 years just for this: https://needgap.com .

People post their problems, For which they haven't found a solution (or) when existing solution doesn't fulfill their needs i.e. need-gap.

You can check if any of the problems resonate with you and if it does you can choose to apply your skills to build a side-project which address that problem. The best part is you can get the feedback from those who want what you're building while building it.

Every few years Hacker News has a discussion on research ideas. If anyone can find links to past HN idea-lists, that would be great!

"Factor Tables" (AI), "Dynamic Relational", and a stateful GUI markup standard are my pet suggestions. They are based on practical needs I observe, not pie-in-sky. (Desktops aren't going away, so let's stop making them a 2nd class citizen per UI's. Mobile is more "sexy" right now, but desktops get the real work done. Office Space's Milton probably does the real work.)

I've been using the following to pick up a new hobby (wood working [surprise]) and start learning about potential business ideas (data/ML space and pet industry):

I'm sure there are topics you're curious about or sound interesting to you. Maybe there's a hobby you wanted to try out? Spend a little bit of time watching YouTube, listening to podcasts, reading blogs or joining discord/slack communities for that topic.

This doesn't have to be some huge investment. Spend an hour or two looking at a topic. This can be in 5-minute increments or a 2-hour block, w/e.

Are you still interested? Do you want to learn more about the topic? Do you want to try your hand at it? Go for it. Go as deep as you want on it, actually make your own version of w/e you're interested in.

Now go talk to people in various communities online. There are so many slack and discord groups out there, you'll find something you connect with. Ask questions. Engage in conversations. If you're feeling up to it, reach out to people to setup video calls. Some meetups are starting to offer in-person events, if you're comfortable doing that.

Still interested, still want to turn it into something more? Check out The Mom Test by Rob Fitzpatrick. It's fantastic guide on how to actually do customer development. Doesn't have to be about making something into a business. He's got great tactical advice on how to get info from people in a regular conversation and learn about stuff.

Books are a great starting place. google scholar for searching the research landscape. scihub for getting details.

as for fishing for ideas, meditation, doodling, keeping a journal

I don't believe I have ever 'looked' for ideas. As far as I can recall, I build stuff to solve a problem I have. There are 2 classes though

1. I have a problem but I think it's just a small thing or very unique to me.

2. I have a problem and I see that other people have the same problem but they have simply learnt to live with it. Or I have this problem and it's driving me crazy ;)

For stuff in group 2, I first try to quickly build something (no matter how hacky) and then as time goes on, I try to expand it/get others to use it (this is how I got my current project). This in turn leads to more feedback which either helps me refine or abandon trying to commercialize it.

I also check out forums for the area that I'm working on and also look at bug trackers if it's public. Those are 2 places to get information on user requirements/pains. For example, my current project is on Google App Engine so I check the Google Groups forum for Google App Engine regularly

My ideas are mostly things I need/want myself. So the first step is to find something that solves it. If it exists, great!

If it doesn't or if there's some pain points, I google about the topic and find out the important things about it. Depending on the field, this varies from blogs, youtube videos, research papers and press releases.

Napkin math

Is it mathematically possible? Physically possible? Economically possible? Politically possible? Etc?

Usually ideas arise out of frustration. I'm trying to accomplish X, but the tools to accomplish X are clunky, slow, ill-suited, etc. So I start to think I could build a solution for accomplishing X that others might find useful as well.

No idea

I tried an Ask HN with my idea, but nobody cared.

Another time, I booked Facebook ads for people to subscribe to a pre launch email list, but nobody cared.

So mostly I just build stuff that me our my friends want. Or where a prospective customer comes to me with their project idea.

A post (not my own) I find worth revisiting in this very subject from time to time, on the various approaches to product ideation:

https://jakobgreenfeld.com/gut

Think about the biggest problem (pain point) in your own life. Solve that.

Find out first if you can get a market for it. It is better to have a product with competition than a monopoly on something nobody needs.

What's even harder is product-market fit. How do you even begin testing the market hypothesis?

Here’s a simple approach— If you don’t have any ideas, read more books on interesting ideas.

If you have an interesting idea, then write—Write for yourself, to yourself.

And when you’re ready, create a prototype or mock-up.

Without more information, simple is best.