AI-generated spam pull requests?

Relatedly, the Rails codebase recently received a (clearly marked) AI-generated pull request: https://github.com/rails/rails/pull/47708

At a glance, it looks like it's been mostly well-received and has not yet been immediately closed as spam.

I wrote more about what actually happened here: https://navendu.me/posts/ai-generated-spam-prs/

It can help set some context to the discussions.

TLDR:

Recently, a person has been using AI tools to generate code and open pull requests to open source projects I contribute to.

The code is entirely wrong and doesn’t work, and it is evident that the person making these pull requests doesn’t understand the code.

The person also copied explanations (which was an obvious giveaway as it sounded like a typical response) into the pull request and attempted to explain the code and answer questions from the reviewers.

We were polite and when it didn’t work, reported the person to GitHub.

I don’t want to shame the person publicly. But I want to make other open source maintainers aware that this is a thing and prevent them from wasting time and effort chasing such people down.

> We do not know what the end goal of the person was but we confronted the person and closed the pull requests.

Maybe this was a naïve attempt at inflating their GitHub numbers? Some people use those as a credibility measure when applying to jobs or getting clients.

We've seen a guy create thousands of spam NPM packages for is-even, is-odd, is-red, etc. I don't doubt we'll see many people try to "contribute" to many projects via AI to make a name for himself.

I got one too https://github.com/tonsky/FiraCode/pull/1518

I've said this before, but in the very near future AI will a consume your Jira board, write code, compile it, test it, then raise a PR all without you. In fact, it might not even bother with a PR because the output will be so good it will be comparable to a human trying to critique the moves of stockfish in Chess.

Devs who think they're "optimising their workflow" with AI are in lala land. Understand that soon you're not going to be needed to prompt ChatGPT to do your work for you. You're not going to be a 10x engineer, you're going to be an unemployed one. You're current workflow of copying ACs into ChatGPT then copying the output into VSCode will also be automated soon, obviously.

Learn hard skills now as it will buy you a few years. We have all been given a death sentence and most people are still yet to understand this. It's unlikely that in the future we're building humans will be needed, let alone our inefficient labour.

As a developer, it's important to pay attention how we are using these tools to co-exist, not to be replaced by them. In addition to using these tools, problem solving skills and creativity are going to be the most important parts of software development jobs in the future. Keep your problem solving skills sharp with ChatGPT help and guidance folks: https://github.com/Liopun/leet-chatgpt-extension

This is a good example of bad use of AI. If your developer wrote code and tried to open a PR without talking to anyone, without testing, without an observability plan, without even running the code to make sure there are no errors, that would be crazy!

Good AI systems will do all the above.

Sorry to plug, but if you're a developer interested in building on top of langchain and building similar tech, please email me (in my profile). I'm a senior developer looking to collab.

Don't other projects have automated tests that check that the PR will actually compile/work? And if the tests fail, the PR is rejected.

If the AI PR covered the bases within one standard deviation of your other (human) PRs, would you care?

Formerly "hand wavy" questions about humanity, cognition, awareness are now showing up right in front of us. They are transforming into things like (a) is this PR worth my time? (b) does it introduce legal / license risk? (c) what principles were considered during its creation and so on.

It is the new reality though. Soon AI generated PRs will look human enough to require a good amount of time of the repo maintainers to review.