I've tried a little with ChatGPT, but context length is an issue and it's a huge PITA trying to give it awareness of tons of small modules. I have copilot, but, to date, I've only really found myself using it as a glorified comment writer.
Does this exist yet? If so, what's a good approach? If not, is the limitation due to context length or is it fundamentally an issue that LLMs will struggle to intelligently discuss?
1) Assume that LLMs work like improvisational actors in that they are in the mode of saying, "Yes, and...". By that I mean that they will mostly agree with what you write without providing a real critique of your ideas unless so prompted. If you don't ask critical questions the AI will happily entertain crazy ideas. I bet I could get it to happily act as if it believed the moon is made of cheese with sufficient prompting effort.
2) Use multiple session to explore and discuss separate topics. Use your "main context" with purpose.
3) Ask a series of "warm up" questions. I've found that the quality of reasoning differs immensely whether the AI has been discussing a topic. Asking a question, then asking the AI to continue (several times), then asking it to reconsider it's responses seems to create the required knowledge graph network that is required for a pretty good response.
4) Some LLM's will eventually either stop working or start to slow down as their context becomes filled. I've found the best way around this is to ask one AI to generate concise prompts with minimal formatting to help another AI get up to speed. Don't let it get too meta though. Two AI's will happily "fluff" each other's ego and get nothing done over a series of dozens prompts.
5) Use "Less Meta." as a command when an AI talks about how to do something instead of actually doing it.
6) Pay attention when an AI says it cannot do something. It might start lying to you if you continue to push it.