Those with success using GPT-4 for programming – what are you doing?

SQL queries. I am poor at writing SQL queries with bunches of joins. I just show it the table definitions and tell it what I want. Sometimes it doesn't get it quite right (which is why you need some knowledge of what you are asking of it) and I point it out and it fixes it.

Regular Expressions. I hate doing regexps. It is excellent at them.

Wiki articles for an encyclopedia I'm developing. It is great at this, but occasionally I catch it hallucinating articles out of whole cloth because it has access to zero information about what I asked it about, so it just goes from the article title and imagines what it must be about.

I know where to use it right, and I've found my output has not only doubled since I started using it, I am enjoying coding even more than ever because it has got rid of the worst drudgery that would cause me to switch from my IDE to my browser and bring up HN to avoid working.

Coding with LLMs was easier after understanding their limitations.

1. They don't know what they are saying until they have said it.

2. Your inputs and its outputs help make the next message.

3. LLMs are not suited for information retrieval like databases and search engines.

LLMs excel at reasoning and predicting subsequent text based on given context. Their strength lies in their ability to generate relevant and cohesive responses.

To optimize results, outline clear rules, strategies, or ideas for the LLM to follow. This helps the model craft, revise, or build upon the established context.

Starting with a precise query and introducing rules or constraints incrementally can help steer the model's output in the desired direction.

Avoid zero-shot queries as these can lead to the model generating unexpected or unrelated responses.

Be cautious while seeking pre-calculated or non-derived answers. Some instruction-tuned models might output incorrect solutions, as they are trained to respond to certain queries without proper context or information.

also, this is my biggest gripe no fault of ours of course: don't seek pre-calculated or non-derived answers. I've seen some of the demonstration data that people are using to train instruction-tuned models and are being taught to respond by making up answers to solutions it shouldn't try to compute. Btw, the output is wrong.

{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." },

https://github.com/sahil280114/codealpaca/commit/0d265112c70...

I use it to brainstorm and prototype approaches to a problem. First, I'll ask it to give me an overview of the problem domain; this gives the LLM context. Then, I describe the problem and ask it to generate solutions, along with pros/cons of each approach. This is iterative: you might ask it questions, modify its suggestions, periodically summarize. After that, you can either ask it to give you code for a prototype or build it yourself.

These models are good for ideation, scaffolding, and prototypes. It's currently clumsy to fully build an app with an LLM, but they are quite useful for certain tasks.

GPT-4 is amazing for me for writing Python, Go, and Kubernetes YAML.

I have designed and implemented 8 operators with between 90-100% of the code auto generated.

I use it to generate mermaid diagrams that implement the first 3 layers of the C4 model, sometimes needing some editing or guidance to modify it, then have it generate the code.

I generate diagrams with high temperature, and code with low.

That’s my experience. I have a coworker using ChatGPT Plus (gpt-4), and they fail to get anything working. It’s not zero effort, but how I think generally aligns with the model I think.

I love having a partner to talk to about thoughts and design ideas while on walks using the ChatGPT app w/ transcription.

I have never felt more productive.

I use it as a personal tutor. It is really good at it, because it can respond to clarifying follow up questions and it is usually right. For pure development or peer review, it is not there yet. I asked it recently to help me decipher some real life SQL performance issues and it did not really help. Actually, it turned out to be a waste of time. The classic Postgres query analyzer and some 3rd party query visualization tools got the job done.

It's useful for rubberducking.

https://en.wikipedia.org/wiki/Rubber_duck_debugging

https://RTEdge.net was polished up in no time, thanks to its vast knowledge. If you're knowledgeable in the field you ask questions about, it becomes a pure time save, as you can either just tell or test it.

It's general world knowledge is also vastly impressive though.

It can draw a 2D representation of a keyboard key

  _____________
  |           |
  | Keycaps   |
  |___________|
  |           |
  | Key Switch|
  |___________|
  |           |
  | Circuit   |
  | Board     |
  |___________|
  |           |
  | Base      |
  |___________|

explain what happens at each layer when I press the key but more impressively also what happens if I spill cola over it:

  _____________
  |           |
  | Keycaps   | --> Coca-Cola makes the keys sticky
  |___________|
  |           |
  | Key Switch| --> The switch could become sticky, preventing it from registering key presses properly
  |___________|
  |           |
  | Circuit   | --> The liquid could cause a short circuit, damaging the keyboard 
  | Board     |
  |___________|
  |           |
  | Base      | --> Coca-Cola pools at the bottom, potentially leaking into the circuit board from below
  |___________|

You hit the nail on its head right there. It’s all based on a doomed technology that is not going to give us the benefits it promises. I do think future models will improve on this, but there’s likely very little room (read: data) left to make that happen.

Developing symbolic reasoning further would likely be a much better use of researchers’ time, even if it takes longer. But the incentives in the short term just aren’t there, sadly.

I'm the creator of rqlite[1], an open-source distributed database written in Go.

It's saving me time (sometimes 2x speed up on certain, well-specified, tasks), and I enjoy using it. I wrote a blog post with some details on how it has helped me code the database: https://www.philipotoole.com/what-did-gpt-4-find-wrong-with-...

That said, the most recent release of GPT-4 seems a little more buggy[2].

[1] https://www.rqlite.io

[2] https://news.ycombinator.com/item?id=35970711

1. It can do some reformatting tasks faster than I can do them by hand. Example: Inline FuncA into FuncB .

2. For more complicated tasks it requires good prompting. Example: Tell me three ways to fix this error, then pick the best way and implement it. . Without the "step-by-step" approach it almost never works.

3. It's pretty good at writing microbenchmarks for C++. They always compile, but require some editing. I use the same prompting approach as (2.) for generating microbenchmarks.

4. It's pretty useful for explaining things to me that I then validate later via Google. Example (I had previously tried and failed to Google the answer): The default IEEE rounding mode is called "round to nearest, ties to even". However all large floating point numbers are even. So how is it decided whether 3,000,003 (which is not representable in fp32) becomes 3,000,002 or 3,000,004?.

5. It can explain assembly code. I dump plain objdump -S output into it.

The main limitation seems to be UI. chat.openai.com is horrible for editing large prompts. I wrote some scripts myself to support file-based history, command substitution etc.

I had to optimize some Python code to reduce its memory usage. After trying all ideas I could think of, I thought about rewriting it in a different language. Copied and pasted the code into ChatGPT 4. Tried Rust at first, but there were too many compilation errors. Then I tried Go and it worked perfectly. For the next couple of weeks, I used it to improve the Go code, as I've never used Go. It gave me great answers, I think maybe once or twice the code didn't compile (I used it dozens of times per day).

I'm now using the optimized Go code in production.

I did this experiment (a game) to see what's up and what's down around all this: https://github.com/romland/llemmings.

While there is some GPT4 in there, it's mostly ChatGPT and a small handful of LLaMA solutions.

That project is a contrived scenario and not realistic, but I wanted to experiment with _exactly_ what you are talking about.

Very often I could have done things a lot faster myself, but there is one aspect that was actually helpful, and I did not foresee it. When inspiration gets a bit low and you're not in the "zone"; throwing something into an LLM will very often give me a push to keep at it. Even if what is coming up is mostly grunt work.

The other day I threw together a script to show the commits in a reverse order and filter out (most of) the human commits (glue) over at https://llemmings.com/

It has saved me some time writing queries in some obscure DSLs. Results were slightly off but close enough for me to run with it. Replaced my first pass of reading through the docs, but not really helpful beyond that.

I probably could have spent more time framing the query to get better results.

I've been using it to generate code that follows certain pattern.

For example, I saved 2 hours. I told it to generate a GraphQL resolver after inferring a Zod schema.

It followed the code conventions from other file.

It generated it beautifully.

Every time there's boilerplate, to ChatGPT it goes.

Writing in a language that I don't know that well, but I know enough. Okay, my javascript isn't the strongest, so I'd prolly have to spend 30-45 minutes just coming up to speed again on my basic AJAX and modern syntax, or BAM write a schema of my idea and get GPT to get my idea on paper with halfway decent style, syntax. I can take it from there.

One of the first things I tried (which I thought would be harder) was “Write a Python program that accepts a YouTube url and a start and stop time in the form MM:SS and a file name and it extracts the video between those times and converts it to an animated gif. Also provide a list of packages to be installed.”

The resulting code worked. Took an interaction or two to add usage info and tweak things, but it’s a neat little utility. Due to the libraries used, it was also simpler than I expected and would have been easy to write if I knew about those libs, so I also learned something.

It’s just one or two steps away from just saying “Go to this YouTube URL and extract the video between 3:20 and 3:27 into an animated GIF named ‘CatAndRaccoon.gif’” and having it write, debug, and execute the code.

Also had it hallucinate absolute garbage from time to time. The worst offender, that consistently shows up: Claiming there are certain library features, when there are not.

Easily the best rubber ducky though. Copy pasting big blocks of my own code and asking about it gives new perspectives to problems I am stuck on. Huge time saver in that regard.

I have found it completely fucking worthless for anything I have thrown at it.

Computing how to dilute a concentrate into a fruit punch, how to create and render a template in golang, how to parse an int in golang, what makes a chord progression a "lydian" chord progression... it goes on and on.

I was writing code for a Raspberry Pico and a beginner at it. The Pico is hooked up to a little display with no documentation other than a few horrible convoluted and broken-english examples with hundred line functions, and I had no idea what to do to just display some things on this little screen.

So I simply dumped the code examples into ChatGPT and said "Given these examples that display text and boxes on a screen, can we write a simpler interface and straightforward small functions for the display code?"

And it was done.

This wasn't code I wanted to mess with, I really just wanted to build my application rather than spend any time messing with the code to interact with this proprietary display. It was fantastic!

I let it write the boring stuff. Needed a python class to take an example dict, and generate a class that creates all the properties that map to the keys in said dict (I have a constraint preventing me from using something like json_dataclass or similar). While it's churning that out, I can focus on other things.

Also found its great for regex, I'm pretty good with writing these, but recently came across a pretty complex one in our codebase that wasnt commented. Pasted into GPT4 and asked it to explain it, it broke down each bit in detail, and in the end even generated an example string that would match it.

AI has replaced 90% of my SO searches. I now don't contribute anything back and have to trust unvetted code that tends to have defects

Coincidentally, SO seems to be worse. But then again, I only use it 10% of the time so I might be wrong

I documented my process of building an email-auto responder with ChatGPT, not even GPT-4: https://dearai.substack.com/p/coding-gabrielle-six-chatgpt-s...

I only have a little coding background, and ChatGPT didn't make me an expert. But the collaborative Human+AI process did allow me to complete a project end-to-end, including figuring out where to host it and how to do that.

I found that it helped me with 6 "superpowers": 1. Choosing between options (e.g., AWS vs. GCP vs. Zapier) 2. Walk me through it (e.g., how to set up a Firestore database) 3. Text-to-code (including simple nuisance calculations and code-to-code changes) 4. Help me out! (i.e., fixing broken code based on error messages) 5. Teach me (e.g., learning the difference between let, const, var, etc...) 6. Check my code (e.g., it caught errors before I even ran the code)

Check out the post for more details if you'd like!

There's another post on building a website from scratch where I also tried Replit's Ghostwriter. Yes, I faced a lot of frustrations in the process, but going from "I can try struggle through this on my own" to "I actually have some help here that's always available and usually right" is amazing IMO.

Writing small python or bash utility one-off scripts to do various things.

For example, a utility to use the bitbucket api to dump all of the environment variables configured for a pipeline.

This one was using Bing AI chat:

I had a bunch of PDF files with coded file names, like ABC-123-abc-999-001.pdf, where each section of the file name was meaningful. Inside the PDF were several form fields. I needed to insert records in a database for each of the 97 files. Easy, but tedious.

I prompted it with a description of the file name breakdown, where the text to grab was in the PDF, and then asked for a Python program to find the files (in subdirectories), extract the PDF text, and write a text file with the SQL Insert statements for all the files.

It took two or three minor iterations, but less time than it would have taken me to write from scratch because my Python is rusty. Regular expressions, PDF processing libs, file system traversal, and SQL generation, and it all compiled and worked from the start (the iterations were to tweak a few things I didn’t specify).

This is the kind of thing that I think is perfect for these tools. With the new tools that let the LLM compile and execute code, it will be cool (and potentially dangerous).

Adobe automations are written in a truly bizarre language that, among other things, starts counting at 1 instead of 0. ChatGPT cheerfully obliges.

IME GPT-4 is bad at doing things and great at looking things up for you. Rather than trying to get it to do things, I ask it how I should do it (and argue with it about any hallucinations that come out—a process that often teaches me a lot!) and then I go do the task myself, copying whatever I need from the conversation but not relying on it slavishly.

It’s basically a fuzzy inverted index for public docs and code. “What’s the normal way of doing X”-type queries work best, with quality falling off quickly as complexity increases. Stuff like “what is a ‘git log’ command that only shows commits containing a particular snippet in the diff, limited to merge commits on master”, for example.

For more complex tasks, a trick I’ve seen work well is “give me an outline for how to do large task X” followed by “let’s go through each step in the above outline. For each one, I’d like a description of how it should be solved, including example code. Let’s start with the first step.” But that trick is not totally reliable and has its own complexity limit.

Top tips:

* If you can write it from memory, go ahead and do so. Do not consult GPT-4

* If you know what to do but need to look a few things up - put your best effort into GPT-4. It will flesh it out

* If you're using a library that is new, you can copy paste the library examples into GPT-4 and then describe what you want to do. It will give a great starting point

I'm great with backend software but not so great with frontend, so I had GPT4 write me a simple React app to test out some endpoint. Works really well. After the code was working, I pasted the app into it and asked it to make it "more visually appealing" and it did.

I use it to get more programming done, and take care of all the things that comes in the way:

A big client asked us to fill around 200 questions in an excel sheets, regarding our company security.

Then he asked for a cybersecurity standard document (like a big thing around 50 pages)

I took the previous excel sheet, removed noise, anonimized it. The n pass it to gpt3.5 to summarize for each security category (access management, code source security,...) the bulletpoints answering how do we implement that

To finish, I passed the bulletpoints for each category to gpt4, to write a nice bullshit document which sound more professional than me

Autopatching vulnerabilities on my network.

I have a set of tools which build prompts describing the environment and the output of vulnerability scans. The tool then requests a shell script to disable/fix/update the vulnerability. The script is submitted as a PR which has actions that run integration tests. Human intervention is sometimes needed, but the focus is on better engineering of prompts (and by proxy tooling).

Describing the environment relies heavily on my CMDB (Combodo’s iTop) so this is not a one-size-fits-all approach and this is functioning entirely in my personal lab of ~100 servers. That said, ChatGPT has given me the best results compared to locally run LLMs

I've been using it to refactor Clojure codes and port Scheme examples into Clojure.

Maybe because Lisps have very low syntax, I find I can just copy off function/macro definitions as context and GPT3.5 can rewrite the code into something I can use... And by testing in a REPL (in :dev) I can instantly see which code has hallucinations and which work perfectly.

Tbh I find it hallucinates mainly when dealing with mutable states (e.g. atoms) or with pipelines of complex maps transformed by multimethods -

Just using small data bits and normal pure functions make GPT3.5 work perfectly because it doesn't need to take into account code outside the 2048-4096 tokens it's thinking right now?

I'm launching a financial planning business right now but I was recently using it to write xpath queries to use with "xmllint --format".

I then used it to generate a more robust Python script processing these large XML files instead.

I like using it to generate scaffolding and for debugging but I haven't had to touch a legacy C++ code base for a few years.

I’m using it to do the mundane tasks of unit testing and (some) documentation. I find that the code it spits out isn’t perfect but getting some boiler plate and fixing it up is pretty fast compared to writing from scratch.

I’ve used this enough that I wrapped some cli glue around it and wrote https://github.com/radoshi/llm-code

I’ve used this mostly to write Python and bash, with some Makefiles and Dockerfiles thrown in.

GPT-4 is better, albeit slower, than 3.5-turbo. HTH!

Text categorization for podcasts.

The prompt asks for specific aspects from a podcast - people, dates, locations - score them by relevance and count the occurrences. Now, GPT can't count for a damn, but it is a useful proxy. The relevance score is pretty good.

There are existing services that can do this, but with GPT and the API I don't need to read a manual and I define exactly what format I want back.

This is what is exciting - GPT will format its responses to _my_ requirements, not the other way around.

OpenSearch (or ElasticSearch) query building. I was new to the technology and their syntax took a while to wrap my head around. Instead I'd just tell ChatGPT my document format and then ask for specific data in natural language.

Fair warning, the queries were not always perfect on first try, but it was a lot easier than parsing replies to somewhat similar questions on stack overflow. Now I mostly write my own queries but it really helped me get started.

I'm using it for single file programs and building command line utilities, it excels at building apps from scratch, but struggles to follow code it didn't write.

I've had so much success that I built my own command line utility (https://github.com/0xmmo/codemancer) to use in the VSCode terminal and my side projects are now ~70% written by LLM.

When I finish writing a method, function or larger chunk of code that works well but could be simplified or optimized, I ask it do it and it's surprisingly good at it.

Sometimes I ask it to write complete classes for me. My longest prompt was full page long and it wrote a county-city autocomplete from a database: the back end in Go, the front end in plain vanilla JS with a non-jQuery based lightweight library (I asked it).

Charts on charts on charts

I've converted from an excel wiz to python, but making charts was always the bane of my existence in python, until GPT. And I personally use 3.5 more than 4 because of the speed, but I used 4 when I need something critical or I know I need it to balance multiple thoughts.

Given a 2021 cutoff, I'm not surprised that rust and Nix are too new for it to do well with. I've been using it to avoid actually learning what manifest.json looks like for Chrome extensions.

One thing that helps is telling it it's wrong or missing a case or whatever. It'll type out a fix (assuming it understands, which it frequently does) faster than I can.

It's great at log pocessing (generating commands for awk, sed, complex grep regexps, shell scripts to combine it all). Anything where I'm not an expert but need something very basic done quickly (e.g. the bulk of my day job is in C++, but I frequently need little bits of Python and ChatGPT is often the quickest way to get the answer I need).

Ha, yeah, GPT* sucks for nixos. I think this is due to

* NixOS development being pretty high paced, [with certain developments (like flakes) post-dating the original GPT cutoff still?]

* Documentation being slow to show up on the internet.

* Just in general lower popularity (than windows or ubuntu or etc) leading to less data for ChatGPT to pick up on.

I've used it a lot for experimenting with new frameworks I haven't used. I recently used it to do a project in Deno with Oak and wasm-dom. It got me 90% of the way with a very accurate bullet point list of what it needed to do. It makes me think about the function of my code a lot more than the exact code representation of it.

I know how to program. I understand lisp. What I have problems with are all the built in functions and variables in emacs-lisp. It has dramatically improved the readability of my emacs configuration and also taught me about some new ways to do stuff in emacs (and when to stop because I can’t do something).

I've used it to SPARQL query to explore wikdata data a bit. I'm about at the breakeven point where hallucinations mean it would be worth it to just learn the language, but I think it clearly saved me time to write some simple queries without knowing the language at all

Today I used it to rewrite a function in llama.cpp that compiled in gcc9 but not gcc7 (an avx2 intrinsic wasn’t available… idk specifics, I barely know C/C++). First attempt was wrong, so I had it write test code to debug. Using the test output it fixed the code afterward.

I use it as a sound board to think out. Also it's helpful in certain programming tasks that I am not familiar with but popular in certain circles. It's great to get up to speed in knowledge without reading tons of articles or wikipedia.

It works well for non-programming tasks, writing the bullshit career development I'm forced to write and crap like this. It's absolutely useless for software development though.

I just use Amazon CodeWhisperer as a nice autocomplete but that's it.

I find that it's great at creating a starting point for react components, using material ui components. Generally, I find it more helpful to review the code it produces than to produce it myself on the first pass.

Optimizing SQL queries. Give GPT-4 a Postgresql EXPLAIN output together with the query and it gives very nice results (you do need to iterate a bit). Has steered us in a better direction quite a few times now.

Here's roughly how my conversation with GPT4 went

Me: I want to make an svg editor, give me some suggestions. I mainly want something with mobile support.

GPT4: gives me some options

I look over the options and choose fabricjs

Me: Start by loading an svg at a predefined url

GPT4:

Me: Ok now implement a save feature, send the json to this url ...

GPT4:

Me: The text is loaded as an image, I want the text to be editable

GPT4:

Me: I'd like to add some google fonts to the text editor

GPT4:

Me: The fonts aren't loading, I think we need to load the fonts first before initializing the canvas.

GPT4:

Me: Ok add a undo/redo feature

GPT4:

Me: Let's add some clickable buttons instead of hotkeys, here is the html..

GPT4:

I probably could have done this myself, but frankly it would have taken me a long time to figure out the fabricjs api. It probably saved me at least a week making this thing.

here's the live app: https://tinyurl.com/2dhh58cn

and the code: https://tinyurl.com/2tu4xrtn

you can tell the GPT generated sections by the (overly) verbose comments