For the purposes of this question, non-trivial means roughly "well-defined question whose answer is not found in <1min of googling". So no non-well-defined tasks ("write X in the style of Y") and no copy-paste tasks (for example, programming tasks which have a ready-made answer in stackoverflow).
Here is a (non-exhaustive) list of such tasks I've tried, with no success.
- Write a program that solves the TOV equations
- Write a program that decodes a Posit
- Write a program that scores a trick in a simple card game
- Simulate a python terminal (fucked up immediately when I asked to slice a string or to bit-shift an int).
- Tell me the melody of a given Mozart piano sonata
- Prove a basic fact about metric spaces (diameter of open ball is twice the radius)
- Prove a basic fact about quantum gates
- Identify a chord given the pitch classes
- Write a poem with a well-defined rhyme
- Write a poem with a well-defined metric
- Tell me who developed an (obscure) software
- Give me a short biography of an (obscure) singer
- Summarise a research paper
- Translate a joke
- Simulate a Zork-like game
- Summarise the Europa League final 2013
- Summarise the Counterstrike Boston Major final 2018
In every single one of these it failed. Now it's great that it can answer the question "how to I make an http request in js", but that's a google search, not really impressive in my book. As you can see, even some questions that are google searches it failed.
Suggestions and discussion about LLMs welcome!
Also note I only have access to GPT 3.5, not 4.
Took me maybe 10-15 minutes total, where it would have taken me hours to brush up on the multiple versions of JSON schema, since I’ve never really written it. Reading it is trivial though, so you can review the final output quickly.
ChatGPT4 does trivial things faster and cheaper than humans. That's useful enough to replace jobs.
Researchers release a smarter AI every year. Just be patient and they'll have one that can replace us within the decade.
I have found GPT-4, on the other hand, to be excellent at writing complicated SQL queries, one-off scripts in just about any language, expanding and fleshing out content ideas from just a seed of insight, and debugging cryptic error messages. Among some other one-off use cases, but these I use on a recurring basis.
you have a list of 300 products and you want to create a table of categories.
crud code. give it a list of fields and a table name and you can generate a create statement(with it guessing types), html form, db access code.
looking to do a simple project but with a twist and create a tutorial in a language you don't know very well.
the thing is isn't not a knowledge base. can you build a knowledge base and use it as a front end. that's where the power lies.
it will also do a pretty good job of generating realistic field level test data
Current LLM's are broken shovels, their creators sell them for others to dig the gold.
Isn't this most things? Maybe my search skills have gotten bad but it seems like it takes several minutes to filter out spam and scroll through SEO filler. ChatGPT immediately gives me the answer with no bullshit.
Also, how do you define failure? I've asked ChatGPT+GPT-4 to write poetry and have been quite satisfied with its response.
1) tried to use it to generate some complex Typescript types that I couldn't remember how to do. Result was invalid Typescript.
2) tried to get it to write a product description of a CNC plasma table with IHS and THC. Result was entirely factually inaccurate because it didn't know the difference between THC and IHS.
- Creating cover letters if given decent parameters
- Asking for customized fashion, makeup, or self-presentation advice
It's not great at answering questions that are rooted in fact unless you are very specific and willing to break the question/task into much smaller chunks. It shines when it comes to essentially aggregating public opinion.
https://www.emergentmind.com/ ("The latest AI news, curated & explained by GPT-4" )
The way I see it, it's main strength right now is turning an hour of googling into 5 minutes of chatgpt + Google.