What is Q* (Q star) at OpenAI and how does it threaten humanity

Question

Reuters broke that the precursor to all the recent drama at open AI started with researchers at open AI writing to the board about a recent breakthrough they made and the threat it poses humanity. (It could be possible that Sam was aware of this and didn&rsquo;t care but that&rsquo;s a tangent)Given where AI stands today what kind of breakthroughs are possible. What are the big gaps to AGI that exist today?would be great to know of the gaps to follow progress and closeness to achieving AGI

MAXPOOL · Accepted Answer

Speculating from the name only.Q* might be name derived from Q-learning and A* search algorithm.In that case it would be informed best best-first search using reinforcement learning.

DicksonX · Answer

I think it's nothing but an obvious first step to have AGI not limited to fine tuned with static biases and human feedbacks. It's the idea I was in my mind for last 2 to 3 years. We use tree of thoughts chaain them and use a massive q learning probability array to find the best path for decision making. Seems a common sense concept and a known idea for long time. Open AI now moving from static rewards to dynamic rewards . That's AGI and agents will have the truth aligned by its own . A good step in mimicking us.

wahnfrieden · Answer

Its threat to humanity is that VC-backed businesses will use it to justify regulatory capture and recommendations of total state authoritarianism under the guise of safety, leading us to autocrat rule and subsequent demise.It&rsquo;s all out in the open, you can look at the papers coming from the EA community which as Frontier AI Regulation and the freedoms it claims are necessary to strip from society to protect ourselves.

quietthrow · Answer

https://drpippa.substack.com/p/q-tigrisInteresting but not sure who this author is.

jonincanada · Answer

balderdash? "Q-star". Yes, the Q as in q-learning -- optimize a long term goal. The "star points" are the embedded algorithms discovered and joined within the transformer/NN architecture. Stars where formed after SGD discovered the best representation of said embedded alg type. I'm running a scaled down version myself -- somewhat impressive. Do it at 1k B parameters? hold my beer.

andyjohnson0 · Answer

The Guardian is reporting [1] that Q* "was able to solve basic maths problems it had not seen before" and cites a paywalled article on The Information [2]. They also say "the pace of development behind the system had alarmed some safety researchers" and "The artificial intelligence model triggered such alarm with some OpenAI researchers that they wrote to the board of directors before Altman’s dismissal warning it could threaten humanity,"
Sounds like it might be something notable, perhaps related to Q-learning and A* search as others here have speciulated. How it represents a specific or general existential threat is less clear, to me at least.
[1] https://www.theguardian.com/business/2023/nov/23/openai-was-...
[2] https://www.theinformation.com/articles/openai-made-an-ai-br...

MrCoffee7 · Answer

Gary Marcus just put out a column about this: https://garymarcus.substack.com/p/about-that-openai-breakthr...

NoZZz · Answer

I think it means that the letter Q is the answer to life the universe and everything. Notice the line entering the circle, it symbolises the initial act required to create life.

What is Q* (Q star) at OpenAI and how does it threaten humanity

Speculating from the name only.
Q* might be name derived from Q-learning and A* search algorithm.
In that case it would be informed best best-first search using reinforcement learning.

https://drpippa.substack.com/p/q-tigris
Interesting but not sure who this author is.

Gary Marcus just put out a column about this: https://garymarcus.substack.com/p/about-that-openai-breakthr...

I think it means that the letter Q is the answer to life the universe and everything. Notice the line entering the circle, it symbolises the initial act required to create life.