I have a good reason to believe that one (if not multiple) of our contractors is stealing company time by using ChatGPT or some other LLM, not even running the generated code or spot-checking it, sitting on it for days or even weeks to make it look like they are working and then tossing it over the fence for our internal seniors to review it so that it gets fixed. I honestly wouldn't even be suprised if they are using the code review feedback as part of a new prompt while "iterating" the code.
I started to get suspicious when I was getting small merge requests with code that was obviously broken at a glance. It was clear to me at the time that they didn't even execute the code once because it would obviously just crash instantly. More and more I started noticing the patterns of LLM generated code in style and comments, as well as seemingly random changes to completely unrelated portions of code being included in PRs unrelated to their given tickets. Most recently, I received a PR that was so obviously and egregiously broken and full of LLM-isms there is just zero chance this person even tried to make it work as it had a blatant infinite crash-loop in it. I doubt this person even has a local setup of the codebase on their machine.
Now, I'm not completely opposed to developers using LLM assistants like copilot or whatever and I have used them myself for time to time; this is on a whole other level though. My issue is that this person is essentially stealing company time by using the LLM to write their code and sitting on it, while also "stealing" company time away from QA and other developers who need to review, test and ultimately fix it. Notwithstanding any bugs that are innocuously injected by the LLM, or the problems with intellectual property with sending our code to whatever LLM is being used.
My kneejerk reaction is to fire them for cause immediately and nullify their contract, also refuse to pay them for any work they've done this cycle as it is not fit for purpose. Frankly if this person was in the country, I would be contacting the police for fraud charges but I have no idea how I would even start taking action against a foreign outsourcing agency, other than just refusing to use them in the future. We have multiple developers from the same agency and I have noticed suspicious code from them before but nowhere near as bad as the main example in this post but now I feel like I cannot trust any of them at all.
Am I overreacting? Has anyone had to deal with this and what did you do? How do you protect yourselves from this happening in the future?
Can you issue fixed price RFPs instead of issuing T&M (I guess it would just be ‘T’ here) contracts?
Whenever I hire construction subcontractors (which is frequently) I ask for a fixed price bid with a clear work scope and hold their feet to the fire if they forgot something that they should’ve included.
For small jobs (under $25k) I will occasionally negotiate a T&M NTE (not to exceed) price while working with contractors I trust for jobs that have scope uncertainty (and a bucket of money to cover the worst case scenario)
It’s entirely possible you have a contract where you get x number of bodies per hour for $y an hour. I would have a hard time trusting and managing a subcontractor without a fixed deliverable/scope of work with a definite end.
Negotiating penalties/non-payment for non-working code/pull requests into your contract could help protect against fraud. It might be difficult/impossible to get a contractor to agree to terms like that.
I don’t know if time tracking sheets would help as those can be doctored.
In any case, I’d fire your current contractor and hire a new one, assuming you have the power to do so.