There is a difference between the tool increasing the productivity of a developer vs. using the tool to pass an interview while not having the know-how. ChatGPT is especially good at knowledge questions and such that compare approaches X with Y. This means that it's necessary to approach interviewing differently in order to extract more signal from the interview session.
How are you planning to adjust the ways you interview candidates for Software Engineers?
One way is to test for experience and dive deeper into past projects of a candidate and let them present how they solved problems, why and what alternatives they considered. This puts those under NDA at a disadvantage as they can't speak about their past projects.
This output is an example: the generator made up an option for Macro.escape and completely misunderstood what Macro.to_string does:
https://elixirforum.com/t/chatgpt-generated-code-for-eval-st...