HACKER Q&A
📣 andrepd

Help finding 1 example of non-trivial task that ChatGPT can do


It's very simple: I've played for several hours with ChatGPT and yet, despite all the hype, I have not been able to find one non-trivial task that it can actually accomplish. I want to understand the power and the limitations of LLMs, so I'd be grateful if someone could point me to any example of a non-trivial question that ChatGPT can actually answer.

For the purposes of this question, non-trivial means roughly "well-defined question whose answer is not found in <1min of googling". So no non-well-defined tasks ("write X in the style of Y") and no copy-paste tasks (for example, programming tasks which have a ready-made answer in stackoverflow).

Here is a (non-exhaustive) list of such tasks I've tried, with no success.

- Write a program that solves the TOV equations

- Write a program that decodes a Posit

- Write a program that scores a trick in a simple card game

- Simulate a python terminal (fucked up immediately when I asked to slice a string or to bit-shift an int).

- Tell me the melody of a given Mozart piano sonata

- Prove a basic fact about metric spaces (diameter of open ball is twice the radius)

- Prove a basic fact about quantum gates

- Identify a chord given the pitch classes

- Write a poem with a well-defined rhyme

- Write a poem with a well-defined metric

- Tell me who developed an (obscure) software

- Give me a short biography of an (obscure) singer

- Summarise a research paper

- Translate a joke

- Simulate a Zork-like game

- Summarise the Europa League final 2013

- Summarise the Counterstrike Boston Major final 2018

In every single one of these it failed. Now it's great that it can answer the question "how to I make an http request in js", but that's a google search, not really impressive in my book. As you can see, even some questions that are google searches it failed.

Suggestions and discussion about LLMs welcome!

Also note I only have access to GPT 3.5, not 4.


  👤 sashank_1509 Accepted Answer ✓
We’ll I’m in DL research and a good chunk of my day to day work is creating visualizations so that we can verify if the network is working well. Recently I was asked to create my visualizations with a toggle button. So that I can quickly toggle on and off different aspects of the visualization (I visualize point clouds heavily). I was using plotly and had no idea if such API was available. Google search didn’t work that well, I go through the docs and I see an interface for buttons but it doesn’t work in my case because I’m using Scatter3D which is a non standard plotting interface inside plotly. Normally I would now look at the source code on git, or go through every argument in detail to understand how to add buttons to toggle visibility, and it would take me an hour in total to have a good visualization available. Instead I just asked GPT4 and it spit out code that worked on just copy pasting it. It knew the correct API, everything, I was thoroughly impressed. Very possible that this was asked in stack overflow and GPT just spit that out, but even that way it’s a better search engine than Google.

👤 Zanfa
With GPT 4, I was able to give it a JSON blob and asked it to generate a JSON schema definition to describe it. Then verified the generated schema, found a few edge cases in the dataset where it wouldn’t validate, asked ChatGPT to add those exceptions to the schema under certain conditions. Then found out that the library I wanted to use only supported JSON schema draft 6, so I asked ChatGPT to rewrite it to accommodate that and verified it still works. Then kept asking it to add misc validations as I needed and that was it.

Took me maybe 10-15 minutes total, where it would have taken me hours to brush up on the multiple versions of JSON schema, since I’ve never really written it. Reading it is trivial though, so you can review the final output quickly.


👤 vood
Use GPT-4, buy subscribing to OpenAI platform. You'll see stark differences. Also mandatory to watch this video https://www.youtube.com/watch?v=qbIk7-JPB2c

👤 ultra_nick
Who cares?

ChatGPT4 does trivial things faster and cheaper than humans. That's useful enough to replace jobs.

Researchers release a smarter AI every year. Just be patient and they'll have one that can replace us within the decade.


👤 auntad
As a software engineer that spends a lot of time in marketing, I didn't find GPT 3.5 useful for anything besides writing marketing copy faster.

I have found GPT-4, on the other hand, to be excellent at writing complicated SQL queries, one-off scripts in just about any language, expanding and fleshing out content ideas from just a seed of insight, and debugging cryptic error messages. Among some other one-off use cases, but these I use on a recurring basis.


👤 sharemywin
Take two lists of data(where you need to map them semantically) and map one to the other then generate the code. field level mapping where the descriptions for each code are in the tables or in the data name(check, cash, charge) mapped to CHK, CSH, CC.

you have a list of 300 products and you want to create a table of categories.

crud code. give it a list of fields and a table name and you can generate a create statement(with it guessing types), html form, db access code.

looking to do a simple project but with a twist and create a tutorial in a language you don't know very well.

the thing is isn't not a knowledge base. can you build a knowledge base and use it as a front end. that's where the power lies.

it will also do a pretty good job of generating realistic field level test data


👤 readyplayernull
I use it thru poe.com mostly to write gamedev utilities like math functions, editor plugins and shaders. It gets it right about 50% of the time, but can also provide a starting point and even solved several errors it made with console feedback. Also created a huge design document with quests, mechanics, dialogs and so on which is pretty non-trivial unless you are a very skilled and creative designer. I'm implementing this game.

Current LLM's are broken shovels, their creators sell them for others to dig the gold.


👤 sosodev
"well-defined question whose answer is not found in <1min of googling"

Isn't this most things? Maybe my search skills have gotten bad but it seems like it takes several minutes to filter out spam and scroll through SEO filler. ChatGPT immediately gives me the answer with no bullshit.

Also, how do you define failure? I've asked ChatGPT+GPT-4 to write poetry and have been quite satisfied with its response.


👤 alangibson
It's also been my experience that it fails at anything I could actually need it for.

1) tried to use it to generate some complex Typescript types that I couldn't remember how to do. Result was invalid Typescript.

2) tried to get it to write a product description of a CNC plasma table with IHS and THC. Result was entirely factually inaccurate because it didn't know the difference between THC and IHS.


👤 Mezzie
Two examples:

- Creating cover letters if given decent parameters

- Asking for customized fashion, makeup, or self-presentation advice

It's not great at answering questions that are rooted in fact unless you are very specific and willing to break the question/task into much smaller chunks. It shines when it comes to essentially aggregating public opinion.


👤 PaulHoule
Many of the people who are succeeding at these things have a knack for seeing the emperor's clothes.

👤 istjohn
Copy and paste a random Nature article into it in an area that you're not familiar with. It will summarize it in plain language. Ask questions about parts you don't understand or want to know more. GPT-4 is much better than GPT 3.5.

👤 pella
> Suggestions and discussion about LLMs welcome!

https://www.emergentmind.com/ ("The latest AI news, curated & explained by GPT-4" )


👤 Fatnino
Top few answers here are along the lines of what I was going to write.

The way I see it, it's main strength right now is turning an hour of googling into 5 minutes of chatgpt + Google.


👤 zitterbewegung
Have it write a simple crud app on python or fastapi endpoint using Huggingface .