Can ChatGPT generate fully functional code?

Someone on /r/rust is currently marveling that it spit out a working bitmap allocator: https://reddit.com/r/rust/comments/zgkuq6/

ChatGPT is sensitive to how you prompt it, and it's designed to work as a dialogue where you ask further questions and it can refine its answer.

It can already write reasonable working code, although you shouldn't expect it to spit out entire working programs when the prompt is Gimme Teh Codez without further details. That being said, AI capabilities only get better with time, so I think it'd be shortsighted to dismiss its potential because of its current limitations.

If the pace keeps up, companies with access to large language models in a few years could have a large advantage on legacy programming, the same way programmers in high-level languages have a massive productivity advantage on 1970's low-level assembly programmers. You give up a little more control, but you save time.

Take a look at this: https://www.dropbox.com/s/gq2hkp7w37c5y6q/ai.png?dl=0

I've been using it for small blocks of code or specific functions - stuff I'd have to look up documentation or Stackoverflow for. I see it as a productivity booster, not a replacement.

You can't tell it to "create a CRUD app", but you can ask it to create a nodejs function that gets all the records from a mongodb database.

Yes constantly get this. It’s quite annoying and I’d love to know why. Doesn’t seem consistent or based on char limits.

Anyway, when this happens just use the prompt : “continue”. From this it’ll generally do a good job of continuing where it left off, but some refactoring is often required. As it doesn’t always exactly begin where it ended and leaves out vital code or in complete event handlers and functions for example.

From an initial prompt, you may need to continue several times Ron get to complete code you’re after.

Of course, you can throw the resultant incomplete buggy code back in as a new prompt and request analysis and bug fixes. Which it does a good job of resolving.

Of course, all of this subject to the complexity of your perk or/request

The code I got for "write a small python program to play tic tac toe" was complete (though I had to write "continue" to get all the output.

It made a playable game, that was close to the output that it suggested would occur, but it did have a few mistakes - it missed 2 win conditions (rows and one of the diagonals), and the output didn't quite align fully (i.e. "" instead of " " being printed and no lines for board edges, it also doubled up the diagonal checks oddly.

  # Check if there is a winner, and if so, print a message indicating who won
  if board[0][0] == board[1][1] == board[2][2] and board[0][0] != "":
    print(f"Player {board[0][0]} wins")
  elif board[0][2] == board[1][1] == board[2][0] and board[0][2] != "":
    print(f"Player {board[0][2]} wins")
  else:
    # If there is no winner, the game is a draw
    print("It's a draw!")

It took me a while to find the right prompt to fix this as it seems to confuse the code checking for game over, which checks for a winner as one of the conditions, for the code that checks for the winner at the end of the game.

One of my prompts led to the output being made the program be refactored into a function run in a loop rather than a one shot game. and then I tried:

>the play_game function prints the wrong output when there is a winner in the columns or rows

and got the desired fix.

I like the stateful nature of this.

It all depends on what you mean by generate fully functional code. In my experience, it is able to generate code snippets that are useful but very rarely can be simply copy & pasted into your work. The experience is actually quite similar to Stack Overflow.

YEs, but in small blocks and with specific asks. I asked it to generate code based on an existing project of mine, which generates analytics from social media activity. It does a good job of producing function bodies from prompts equivalent to a short comment you might use to document your function in the IDE.

For your example, I'd ask it to 'generate code, using Python, that accepts input and updates a display to allow a user to play an interactive game of TicTacToe with the computer.' I'm sure I could do better with some experimentation, but that will have to wait for some other day.

First of all, don't use ChatGPT for that. Use full GPT-3, specifically the code models (ie. code-davinci-002).

ChatGPT is a subset & tweak of the full GPT-3 model to make a good chatbot, and it's actually less capable than the full model.

It's strange to me that people are going crazy over ChatGPT when GPT-3 has been out for months now.

More info on the difference between ChatGPT and GPT-3: https://dev.to/ben/the-difference-between-chatgpt-and-gpt-3-...

I've seen some experiments done with using ChatGPT to produce entire websites or programs

https://github.com/pdparchitect/ChatGPT-JobsForBots and https://github.com/pdparchitect/ChatGPT-Pong were two of them.

The author shows all the steps used to generate the programs.

I believe ChatGPT has some kind of response token limit, which is why longer code snippets commonly get cut off. I've seen some people recommend to enter something like "Continue your previous message starting from the last token", which sometimes works, but even when it does, it often breaks formatting, making it difficult to read the resulting code without copying it somewhere else. What I found to work better is instructing it to output a single function at a time, this usually results in short enough code that it can output all of it in a single message. Something like:

  I want you to send a single function in each response, I will say "Next" when I want you to send the next function.

I was able to get it to generate decently lengthy programs using this (on the order of a few hundred lines) that were syntactically and logically correct without any modification. This may work for some languages better than others, though I imagine you can find some way to logically split up the program for most.

It sure can, esp good at simpler things like ui, and popular frameworks etc. The only issue is when there's breaking changes between upgrades, like maybe the framework is at 9.0 but the model was trained on 7.0, so there's some things that don't work just right.

So far it's created some pretty amazing tailwindcss layouts for me, and a flutter app to interface with chatGPT that I want to add more features like saving, sharing, etc.. In 3 hours with zero flutter experience I can now at least do as much as the website does. That's pretty amazing.

Edit: YMMV -- imho you do need to be mid-senior level in tech/programming skills to know and find the best prompts to get it to go your way, there's some trial and error and your ide will usually let you know about errors/etc and you can ask them to fix it and gpt does a good job of correcting things.

I gave it a .txt file that i wanted to parse its data into .csv columns. It actually got most of it right but after spending hours prompting it, it never really got everything right.

Its worst nightmare was that there were some nested values in the .txt. What amazed me the most is that pasting the error code from colab into the chat, it recognized the mistake and generated a fix.

That being said, most of the times it would try to fix a logical mistake by overengineering a solution to an error, or it would identify the mistake correctly, explain in natural language what needs to be done (impressive), but then give me the code unchanged.

Try a simple calculator. It will give you the complete code for that. I think there's a set limit on output text so that's why your tic-tac-toe was incomplete. I've asked for the calc in Rust and Python and I got a working app for both. It was simple but the fact that it gave me what I asked for is amazing.

I bet that you could break-up your app into functions and have ChatGPT write each function to get to a bigger program.

It's limited now but I can definitely see the possibilities.

It won't be free forever so we'll have to weight the usefulness vs cost eventually.

It's ok for small blocks or code, maybe a function or two, even a whole class, but beyond that, it's pretty pathetic. Even the small blocks of code it outputs only compile a few times and have a few errors. At this point, I would be wary of using it in Production systems.

Having said that, I usually compare it with Junior programmers, more like someone just learning a new programming language. You can expect that level of performance from ChatGPT!

If you know roughly what you want and just don't want to dig through documentation to find the exact correct incantation, it's very good at that.

It's kinda crazy how fast my opinion went from "haha neat, the robot can write poems" to "this is an invaluable tool and I am begging you to let me pay for stable access". I use it multiple times per day, it's easily saved me hours already.

There is a character limit on a single response. You can ask things like 'Continue the previous answer' to get it to generate more.

When it’s incomplete I ask it to refactor to smaller methods, and if it crops the output again I ask it to show the full text of a method one by one to get the full output. Tedious but it works.

From what I've seen so far you can tell it it's incomplete, tell it what errors you're getting etc, and it will agree and correct itself until you have a working application.

I feel like the more interesting question is whether ChatGPT is better at generating political speeches instead?

Seems like there would be far more content to train on and less restrictive criteria.

Tim Corey has some thoughts on that https://www.youtube.com/watch?v=FEsahkX7ZF8

Somewhat related — I’m curious has someone built a ChatGPT based Linux command line helper? Ideally should be runnable via CLI

It does many mistakes that are only obvious if you know what you're looking at.

type "continue" & it will continue where you left off.

it has a token cap so it stopped once it filled them.

i really hate all this. codepilot and stuff. it takes away the joy of writing code. esp clean code.

no it's mostly object oriented

Golang results are amazing, they compile most of the time with some lib-specific issues from time to time.

You can ask to explain code, write tests, refactor for perfomance or readability etc.

Prompt:

  Write a small application in javascript that logs in to news.ycombinator.com with credentials passed on the command line and then posts the following message:

  >Fellow members,
  >
  >I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer.
  >
  >One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - "give me the python code for TicTacToe Game" and the code produced was incomplete.
  >
  >Have you all encountered the same or my prompt is inappropriate?

Output:

Here is an example of a small JavaScript application that logs in to news.ycombinator.com using credentials passed on the command line and then posts the message you provided:

  // First, we need to import the `request` module, which will allow us to make HTTP requests to the Hacker News API
  const request = require('request');

  // Next, we need to parse the username and password from the command line arguments
  const username = process.argv[2];
  const password = process.argv[3];

  if (!username || !password) {
    // If the username or password is not provided, print an error message and exit the program
    console.log("Error: Please provide a username and password as command line arguments");
    return;
  }

  // Now we can use the `request` module to make a POST request to the Hacker News login API endpoint
  // In the request, we need to provide the username and password as part of the request body
  request.post({
    url: 'https://hacker-news.firebaseio.com/v0/login.json',
    body: JSON.stringify({
      "username": username,
      "password": password
    })
  }, (error, response, body) => {
    if (error) {
      // If there is an error, print it to the console
      console.log(error);
      return;
    }

    // If the login was successful, the API will return the user's authentication token
    // We need to save this token so that we can use it in subsequent requests
    const authToken = JSON.parse(body).token;

    // Now that we have the authentication token, we can use it to post a new message on Hacker News
    // To do this, we need to make a POST request to the `item` endpoint, providing the authentication token as part of the request headers
    request.post({
      url: 'https://hacker-news.firebaseio.com/v0/item.json',
      headers: {
        "Authorization": `Bearer ${authToken}`
      },
      body: JSON.stringify({
        "title": "Best Prompts for ChatGPT",
        "text": "Fellow members, I'm curious to know what are some of the best prompts to use on ChatGPT to understand its full-fledged capability as a newbie developer. One of the things I noticed, is that it produces the requested code in the cleanest manner but it's not complete. For instance, I tried the prompt - \"give me the python code for TicTacToe Game\" and the code produced was incomplete. Have you all encountered the same or my prompt is inappropriate?"
      })
    }, (error, response, body) => {
      if (error) {
        // If there is an error, print it to the console
        console.log(error);
        return;
      }

      // If the request was successful, the API will return the ID of the newly-created post
      const postId = JSON.parse(body).id;
      console.log(`Successfully posted message with ID ${postId}`);
    });
  });

To use this code, you would need to save it to a file (e.g. hacker-news-post.js) and then run it from the command line, providing your username and password as arguments: