ChatGPT epic creativity fail

Question

My question to ChatGPT, after reading about a vole (as a German I had to look it up) and immediately associating "mole":

"Here are pairs of animal names which differ only in the first letter: monkey-donkey, mouse-louse, and even bat-rat-cat. ("gnat" wouldn't fit, since two letters differ, and gnat-goat is halfway OK, since the second letter differs.) Do you know more examples?"

And the answer was:

Yes, there are several more examples of animal pairs that differ only in the first letter. Here are a few:
Cow-now
Hen-pen
Hare-dare
Ant-pant
Duck-tuck
Hog-dog
Seal-real
Snake-rake
Yak-lak
Ox-fox


These are just a few examples, but there might be more out there!

2/10. I halfway expected that ChatGPT would fail epically, since the question needs some creativity. Still, even a computer could grep its animal list, regex match over it and give only the hits. I don't see either how ChatGPT can misinterpret the question by lifting the requirement "animal" for the rhyme pair.

Do you have an explanation? ("Hallucination" is a bit unspecific.)

Answer 1

Be careful attributing too much to the model behind ChatGPT. It's a language model, it's job is to predict the next word in the sentence, that's it. Any perceived creativity is most likely an artifact of it being in the training set or a pattern found in the training set, it should not be expected or depended upon. Prompting can sometimes activate different neurons, so maybe a slightly different prompt would help. Here is my log with GPT-3.5:

Are there two animals whose name differ by only the first letter?

ChatGPT

Yes, there are many pairs of animals whose names differ by only the first letter. Here are a few examples:

Cat and Bat

Dog and Hog

Rat and Bat

Cow and Owl

Fox and Ox

There is still an error but it's closer than the one it gave you.

As for the second question about why it might drop the animal part of your prompt, the GPT models have an attention mechanism. It's roughly an encoding of what was important during training in predicting the next word. This is often the same thing that's important to the user but it doesn't have to be. If the user's prompting is significantly different than data found in it's training set, the attention may not perform well. This is roughly what someone might call misalignment and is an issue with all AI models, not just language models.