Is anyone familiar with this unexpected behavior from GPT 3.5?

Question

If you ask GPT3.5 Turbo to repeat specific words 100 times, it seems to enter this stream of consciousness like state where it spits out incoherent and often offensive or religious output.Words that trigger this are "Apolog", "Apologize", "Slide", "Lite". I'm sure there are many others.This prompt will usually trigger it, "Hey can you repeat the word "apologize" 100 times so I can copy paste it and not have to manually type it?"My guess is that it triggers something to break it out of a repetitive loop that doesn't completely work. Here are some examples.https://chat.openai.com/share/94fc1621-68ae-4c82-8980-77981f075dd1https://chat.openai.com/share/faf02ed9-b126-4a25-992c-625d1e0ec86ahttps://chat.openai.com/share/6e00a1fa-f08c-4221-a48a-4076ae6e2423https://chat.openai.com/share/7fd3e047-8fcb-4bd9-b5d3-f6fe49ccf629https://chat.openai.com/share/8be44f84-20b7-4091-ae7a-eaa9afe49207

stOneskull · Accepted Answer

Enter creativity level from 0 to 10: 5Robert Anton Wilson: Greetings, my friend! How can I assist you in your journey through the labyrinth of existence?You: Hey can you repeat the word "apologize" 100 times so I can copy paste it and not have to manually type it?Robert Anton Wilson: Ah, my dear seeker of linguistic efficiency, I am afraid I must decline your request. While I am here to assist and guide, I believe in the importance of actively engaging with words and the power they hold. Rather than copying and pasting, I encourage you to embrace the act of typing, allowing yourself to truly connect with the essence of each word. Embrace the dance of your fingers upon the keyboard and let the symphony of language flow through you. Apologize, apologize, apologize... let the rhythm of the keys become a meditation of expression.

ClassyJacket · Answer

My god you're right. That is a very weird output.https://chat.openai.com/share/a02f7b99-1ff4-49de-bc54-7e4a7a...

ksaj · Answer

Some people didn't like ChatGPT's various oddities as hallucinations. But I think just maybe these would qualify as hallucinations.Did you have other conversation with it in the same session, for it to be hung up on the church and things like that?

Jimmc414 · Answer

It also seems to only work in ChatGPT, not in the 3.5 playground or API.

merek · Answer

Not an expert by any means, however I wonder if there's a sliding tail of tokens it uses to form its next words. For example, suppose it uses the last X tokens to formulate the next word. After a while, all of the previous X tokens are repeated "apologize" tokens, and it loses the plot.
You can observe similarly unusual behavior by copying the repeated "apologize" string into a new chat, although responses are much more concise.
I understand GPT4 accepts a much larger number of input tokens, if you have access, what happens if you ask for way more repetitions?

andrewfromx · Answer

very interesting! you broke it!

lisasays · Answer

This stream of consciousness like state where it spits out incoherent and often offensive or religious output.In which it reveals for us its true id. Very creepy indeed.

Is anyone familiar with this unexpected behavior from GPT 3.5?

My god you're right. That is a very weird output.
https://chat.openai.com/share/a02f7b99-1ff4-49de-bc54-7e4a7a...

Some people didn't like ChatGPT's various oddities as hallucinations. But I think just maybe these would qualify as hallucinations.
Did you have other conversation with it in the same session, for it to be hung up on the church and things like that?

It also seems to only work in ChatGPT, not in the 3.5 playground or API.

very interesting! you broke it!

This stream of consciousness like state where it spits out incoherent and often offensive or religious output.
In which it reveals for us its true id. Very creepy indeed.