What open source AI projects do you wish existed?

Question

I'm thinking things like Llama.cpp.What do you wish existed?My big one is an open source RLHF InstructGPT style GPT alternative. I've tried most of the other big models, and they're not really useable for anything that I'd use GPT for, due to the lack of InstructGPT type training.

anigbrowl · Accepted Answer

A multimodal language assistant (MS has hinted the forthcoming GPT4 might deliver some of this). I'd love to have something with about the smarts of ChatGPT (which I find more than acceptable/trainable) but the ability to deal with dynamic buffers.
Imagine being able to use it for code generation, but then to highlight a line or two and say 'I dont like this bit, because...', integrate with error messages/debuggers, and so on. I think we'd rapidly see a shift towards node-based low-code architecture/specification graphs (and graphs of structured types) coupled with modular code blocs in Your Favorite Language. Over time, development interfaces might shift toward a combination of math formulae and pseudocode as found in scientific papers.
Similarly, imagine a persistent conversational interface to image generation, where you could highlight regions or objects in a generated image (or depth map, or voxel model, or...) and have two-way conversations about aesthetic preferences, rendering considerations, instruction ('distorted feet look back, because it makes humans think of the pain involved in an ankle injury...we need to treat this as a general constraint.')
I think we're barreling toward a technological future where successful AIrtisans™ simultaneously work as clients and teachers, transferring their unique perspectives and modes of thinking to explore problem spaces in new ways. To me, the current offerings are great but also feel like the state of 8-bit home computing - you can play a text adventure game, or an exciting 2d graphic game, or a fun music composition toy, or a programmable calculator/educational game. They're all great in their own way, but they're also pretty unimodal and if you have the technical/domain knowledge, it's not hard to pick out where the 'rails' are. I think once we are able to combine different input modes and run run good-not-great assistants locally on affordable hardware (as in, they don't necessarily get things right first time, but have to be trained and hand-held), then we'll see astonishing breakthroughs.
I'll spare you my overly-opinionated thoughts on AGI.

ofalkaed · Answer

https://github.com/pedrozath/coltrane Has gotten me thinking about AI for teaching music theory, but I really don't think it is there yet, music theory is far to fuzzy for AI right now. I do think there are niches where it could be quite great, it could absolutely do well in teaching much of counterpoint and classical forms, perhaps even the basics of harmony but it is hard to disentangle harmony from the fuzzy areas and I could see AI doing more damage than good there.

version_five · Answer

There is a project along the lines you mention but it's still under developmenthttps://github.com/LAION-AI/Open-Assistant

What open source AI projects do you wish existed?

I'm thinking things like Llama.cpp.
What do you wish existed?
My big one is an open source RLHF InstructGPT style GPT alternative. I've tried most of the other big models, and they're not really useable for anything that I'd use GPT for, due to the lack of InstructGPT type training.

There is a project along the lines you mention but it's still under development
https://github.com/LAION-AI/Open-Assistant