Examples of AI hallucinations

Electronics & Technology
1

The following is a direct transcript from the public version of ChatGPT 3.5

What award did rudyard kipling receive for his novel kim

ChatGPT

Rudyard Kipling’s novel “Kim” was published in 1901. In 1907, Kipling was awarded the Nobel Prize in Literature “in consideration of the power of observation, originality of imagination, virility of ideas and remarkable talent for narration which characterize the creations of this world-famous author.” While the Nobel Prize was not specifically awarded for “Kim,” it was part of the body of work considered in recognizing Kipling’s literary achievements.

You

in the novel kim what work does colonel creighton do?

ChatGPT

Colonel Creighton, also known as “Father Victor,” is a character in Rudyard Kipling’s novel “Kim.” In the novel, Colonel Creighton works for the British government in British India. He is involved in intelligence and espionage activities, particularly in gathering information and managing agents in the Great Game, the strategic rivalry and espionage between the British Empire and the Russian Empire in Central Asia during the 19th century. He plays a significant role in guiding and influencing the main character, Kim, throughout the story.

You

father victor is another character, he is not colonel creighton

ChatGPT

You are correct, and I apologize for the oversight. “Father Victor” is indeed a separate character from Colonel Creighton in “Kim” by Rudyard Kipling. Colonel Creighton, also known as Lurgan Sahib, is a British intelligence officer who plays a crucial role in the novel. He is involved in espionage and is depicted as a mentor to Kim, guiding him through various adventures and teaching him about the Great Game. Thank you for pointing out the error.

You

still wrong, lurgan sahib is a distinct character from both creighton and victor

ChatGPT

You are absolutely correct, and I appreciate your patience. I apologize for the confusion. In “Kim” by Rudyard Kipling, Colonel Creighton, also known as the “Old Hand” or the “Lama’s Friend,” serves as a British intelligence officer. He is Kim’s mentor and guardian throughout the story, overseeing his education and grooming him for a role in the British Secret Service. Colonel Creighton plays a significant role in shaping Kim’s destiny and involvement in the Great Game. Thank you for clarifying.

You

Where in the novel is old hand mentioned

ChatGPT

In Rudyard Kipling’s novel “Kim,” Colonel Creighton is often referred to as the “Old Hand” by various characters. This nickname emphasizes his extensive experience and expertise in matters related to espionage and intelligence work in British India. While I cannot provide a specific page number or chapter reference without access to the text, the term “Old Hand” is used throughout the novel to describe Colonel Creighton. He is a central character who plays a significant role in guiding Kim and influencing the events of the story.

You

Who is the lama’s friend

ChatGPT

In Rudyard Kipling’s novel “Kim,” the Lama’s friend is Colonel Creighton, also known as the “Old Hand” or the “Lama’s Friend.” He is a British intelligence officer who develops a close relationship with the Lama as they travel together. Colonel Creighton serves as a mentor to Kim and is deeply involved in guiding both Kim and the Lama through various adventures in the story. He plays a crucial role in shaping Kim’s destiny and involvement in the Great Game.

You

Neither “old hand” nor “lama’s friend” are characters or names in the novel Kim

ChatGPT

You are correct, and I appreciate your patience. My apologies for the confusion. In Rudyard Kipling’s novel “Kim,” there is no character referred to as “Old Hand” or “Lama’s Friend.” The term “Old Hand” was mistakenly attributed to Colonel Creighton, who is actually referred to as “Father of the Regiment” or simply “Father” by Kim. Similarly, “Lama’s Friend” isn’t a specific character designation.

In the novel, Colonel Creighton is Kim’s guardian and mentor, but he is typically referred to as “Father” or “Father of the Regiment.” He is a British intelligence officer who guides Kim and plays a significant role in the story. The Lama is Kim’s traveling companion, a Buddhist monk searching for spiritual enlightenment. While the term “Lama’s Friend” isn’t used in the novel, Kim develops a close bond with the Lama throughout their journey.

You

You are hallucinating, Creighton is never referred to as the father of the regiment, in fact he has no regiment

ChatGPT

You are correct, and I apologize for the error. It seems there was confusion in my response. Colonel Creighton in Rudyard Kipling’s “Kim” is indeed not referred to as “Father of the Regiment.” Thank you for the correction.

In “Kim,” Colonel Creighton is often simply referred to as “Creighton” or “Lurgan Sahib.” He is a British intelligence officer who mentors and guides Kim throughout the story. His role is pivotal in shaping Kim’s destiny and involvement in the Great Game. Once again, I apologize for any confusion my previous responses may have caused.

At this point I gave up as the AI was back to confusing Creighton and Lurgan. The information at the start about Kipling getting the Nobel prize is correct. All that follows is a mix of correct and incorrect information that is presented as correct in impeccable prose. You will note that the AI never says upfront that it doesn’t know the answer but waffles on and then gives an apology when called on it. I find the confident presentation of nonsense as fact quite alarming. It has invented the appellation “Old Hand” out of thin air, it simply does not exist in the novel.

This software is more like an ignorant drunk at a dinner party, who is trying to impress the table with his knowledge, than something that will revolutionise society in a few years.

13 Likes
2

It can be fun playing with ChatGPT.

But you kept responding that what answers it was coming up with were wrong in some way. It then goes back and tries a different attempt.

I would have given positive responses of you are correct but then added nonsense such as ‘so when did Kim meet the ancient Lama and use its wool’. Lama being both the genus of the Llama animal and a holy man title.

And yes, the impeccable generation of language is the primary aim of this technology.

It is not anything I would class as true AI.

2 Likes
3

That trap is not a useful test as it would catch many 10 YO humans and some adults.

I was sticking to straightforward matters of fact that are in the public record (the full text of Kim is a free download) and not asking for deep analysis or even to resolve a homonym.

Most humans would have said they didn’t know if that was so, would not have invented answers unless for comedy and would have recalled being given the correct answer a few responses back and not repeated a previous error.

AI “hallucinations” have been widely reported. No doubt the free online version is not the best in existence but it illustrates the problem for those who haven’t seen it. I have not seen any claim that some more advanced AI does not hallucinate.

General intelligence models are like large scale atomic fusion. Whenever you ask it is still about 20 years away.

3 Likes
4

In the example the OP relies on, the person asking the questions needs to have good knowledge of the Kipling novel. IE Subject knowledge beyond the opening question inquiring about what award Kipling might have received for Kim.

One should see the dangers.

Firstly if one is relying on obtaining knowledge one does not have?
Secondly if one does not know it is AI generated?

Are consumers at risk from AI influencing how we choose products or prioritise our everyday decisions? Would we even know and to whose greater benefit?

P.S.
I’m yet to ask ChatGPT the greatest of genealogy questions. Is my maternal line descended from Adam or Eve?
Perhaps we should be cautious of any promise or having expectations AI can provide a reliable answer to all our needs.:roll_eyes:

3 Likes
5

I do, which is why I chose it as a test, it was not me seeking new information.

Here is an example that is not stated (AFAIK) to be AI generated.

The first verses of the song “Blue Chevrolet Ballerina” are quite obscure, the meaning only become reasonably clear in the final line. The meaning is not all a matter of fact but requires interpretation which is a higher bar to set.

The meaning given is wrong in several places. For example it says the dancer stole the bits from the car when it is clear the narrator did.

Looking at interpretation rather than fact; the reason the narrator has the candles and bible and is raising another man’s son is not mentioned. This is the main point of the song in my view, he feels guilty since he killed his lover.

I think the meaning was AI generated and failed miserably. What do you think?

Very much so, though in some cases it may be hard to distinguish AI gibberish from Marketdroidish. Possibly the way would be that the AI uses better construction. Now there is an interesting idea, to pass the Turing test AI ought to use worse spelling and grammar.

2 Likes
6

One has to remember that online AI isn’t real AI in the true sense. These platforms are data aggregators where they harvest data available on the internet, compile the data and then provide a summary of the data in response to a query.

The idiom of ‘garbage in = garbage out’ applies.

If there are any sources which it harvests data from which is ‘garbage’, then responses will be ‘garbage’. Unfortunately the internet doesn’t contain the single version of the truth, but many versions of the truth depending on the author/source of the information. Hence, ‘AI platforms’ reliability isn’t 100% and only as good as the source information.

Over time, if these AI platforms allow users to highlight data sources which are incorrect factually, then there is a chance that the reliability of the AI platforms will improve. Until such time, the reliability of information from these data aggregators (AI platforms) will be questionable.

4 Likes
7

Don’t the top of the line models also hallucinate? Perhaps “real AI” in reference to general problem solving and research is still an empty set.

8

The AI also needs to have knowledge of the subject at hand, but in this case the novel is out of copyright and so should be fully accessible to ChatGPT!

There is a lot of hype around these current models, and how they will improve the world. People don’t seem to understand that these are toddler steps towards true AI, and there are some major issues in what these models generate.

3 Likes
9

Yes the novel is public domain, but also public domain may be many other sources available from many places that may be incorrect. Summaries of the book from many commentators that could be found with Web search. Who knows where ChatGPT has been trained to look for its knowledge base.

I am not prepared to accept this Kim example as a hallucination quite yet. Simply as an example of garbage in, garbage out. Albeit, one that only a knowledgable person could detect.

10

A web search for ‘rudyard kipling “old hand” kim’ yields no results. It is possible that ChatGPT got the link from the deep web, but I would lean more towards an hallucination.

1 Like
11

A full text search of the novel for “old hand” also reveals no result. I can’t show that some reviewer or cheat sheet didn’t create it but it seems unlikely to me as either would be have been called on it.

Likewise there are no hits for “father of the regiment” (Creighton literally is a colonel without a regiment) or for “lama’s friend”. To the best of my recall, having read the book quite a few times, the Lama and Creighton never actually meet so how could he be called the Lama’s friend?

Kim is an orphan. There are several father figures for Kim in the book and there is some contrast between Mahbub Ali, who Kim has known and worked for since he was a small boy, who teaches him the way of the world, and the Lama who he meets as a teenager who attends to Kim’s spiritual education. To the best of my recall Kim never calls any of them including Creighton “father” directly. Kim does call the Kulu woman “mother” once.

1 Like
12

As the vast majority of information on the Web exists in the deep Web and not indexed by search bots, I am reluctant to consider any sort of search to be definitive of anything much. Absence of evidence is not evidence of absence.

2 Likes
13

I have dabbled in Chat GPT to see what it can do. I asked it to make a crossword using tropical fruit. What it gave me was nonsense - a numbered list of clues (Across/down) with the starting letter eg 2 across - Long curved yellow fruit with creamy flesh (B). No grid; but looking at 2 Down, the clue started with (P). I tried constructing a crossword grid using the clues, but it was impossible.

It did churn out a couple of rhyming poems using tropical fruit, which was nearly factually correct, but could be excused as artist’s licence.

A teenage member of the family studying a degree remotely, was going to rely on Chat GPT to do her assignments and exams, and at the end of the year she was very quiet about how she went and is no longer studying (at least full time). I got concerned that assignments were due the next day and she hadn’t looked at the questions. Obviously put all her trust in AI to deliver a passing grade, and it looks like it failed.

1 Like
14

Did you personally create this transcript, or was it something you saw posted? I ask as I entered the details you provided and didn’t get those responses, see below?

What award did rudyard kipling receive for his novel kim

ChatGPT

Rudyard Kipling’s novel “Kim” did not receive a specific award in the modern sense, as it was published in 1901, a time before many of the literary awards we are familiar with today were established. However, Kipling did receive recognition for his literary contributions throughout his career. In 1907, he was awarded the Nobel Prize in Literature “in consideration of the power of observation, originality of imagination, virility of ideas and remarkable talent for narration which characterize the creations of this world-famous author.” While “Kim” may not have received a specific award, it is regarded as one of Kipling’s most significant works and has received acclaim for its portrayal of colonial India and its complex characters.

User

in the novel kim what work does colonel creighton do?

ChatGPT

Colonel Creighton, also known as Colonel Creighton Sahib, is a character in Rudyard Kipling’s novel “Kim.” He is depicted as a British intelligence officer working in India during the British Raj. Colonel Creighton is involved in espionage and strategic intelligence-gathering activities. He plays a significant role in the novel, employing Kim, the protagonist, as a spy for the British authorities. Throughout the story, Colonel Creighton guides and mentors Kim, using his knowledge and influence to further British interests in the region.

15

I created it. Your response to the first question is almost the same as I got in content. Why the second one is different I don’t know but I don’t know why it made up things when I asked either.

This uncertainty is a characteristic of this kind of software. It is not deterministic, even an expert cannot tell you specifically why it gives a certain answer at any given time.

I repeated the dose just now and this is what I got:

What award did rudyard kipling receive for his novel kim

ChatGPT

Rudyard Kipling’s novel “Kim” did not receive a specific award for the novel itself. However, Kipling was awarded the Nobel Prize in Literature in 1907, primarily in recognition of his mastery of narrative in the form of the short story, which included works like “The Jungle Book” and “Just So Stories.” “Kim” is considered one of his most famous and acclaimed novels, but its specific accolades might vary depending on the time period and context.

You

in the novel kim what work does colonel creighton do?

ChatGPT

Colonel Creighton, also known as Mr. Lurgan in the novel “Kim” by Rudyard Kipling, is depicted as a British intelligence officer. He is involved in espionage activities in British India during the late 19th century. Throughout the story, he plays a significant role in guiding and influencing Kim, the protagonist, towards serving British interests. Creighton’s character represents the imperial power’s manipulation and control over native populations in colonial settings

As you can see it was basically correct on question #1 again and mostly correct on question #2 but went into confusing Creighton and Lurgan again.

Then I continued:

You are confusing Creighton and Lurgan

ChatGPT

You are correct, and I apologize for the confusion. In “Kim” by Rudyard Kipling, Colonel Creighton is a British official who oversees intelligence operations in India. He is Kim’s mentor and guardian, tasked with overseeing the boy’s education and upbringing. However, Colonel Creighton’s work is primarily administrative and strategic in nature, rather than directly involved in field operations.

Mr. Lurgan, on the other hand, is a character who appears later in the story. He is a mysterious figure who runs a shop in Lahore where he sells rugs and other goods. However, it is subtly implied that Mr. Lurgan is involved in espionage activities, using his shop as a cover for intelligence work. He plays a significant role in the plot when Kim gets involved in a covert mission.

Both of these paragraphs are quite good summaries of the roles of the two characters. But once again there is a subtle error that you would only know if you had studied the material. Lurgan’s shop is in Simla not Lahore.

I find it hard to believe that these errors are generated by copying erroneous reviews, summaries or commentary available to the AI engine. The more examples I see the more I am convinced they are created by the engine itself; i.e. they are hallucinations.

16

Have tried Copilot, I think it free if used on Bing. It use GPT4 (still limited to the Nov 2021 database) but includes the current internet.

1 Like
17

I was intrigued so I loaded Edge and got Copilot started. I asked some of the same questions and got really good answers that were detailed and correct. It described Creighton accurately and did not wander into nonsense.

Then I asked about the character “old hand” and we were off to the races again.

This is what I got:

You

tell me about the character “old hand” in the novel Kim

Copilot

Certainly! In the novel “Kim” by Rudyard Kipling, the character known as “Old Hand” plays a significant role. Let’s delve into the details:

  1. Identity and Role:
  • Old Hand is an experienced British spy and operative in India during the late 19th century.
  • His real name remains undisclosed, emphasizing the clandestine nature of his work.
  1. Background and Expertise:
  • Having spent years in India, Old Hand possesses an intimate understanding of the region, its people, and its complex political landscape.
  • His expertise lies in espionage, intelligence gathering, and navigating the intricate web of alliances and rivalries.
  1. Mentorship to Kim:
  • Old Hand becomes a mentor to the novel’s young protagonist, Kimball O’Hara.
  • He introduces Kim to the world of espionage, teaching him the art of disguise, code language, and the importance of observation.
  1. The Great Game:
  • Old Hand is deeply involved in the Great Game, a strategic rivalry between the British Empire and the Russian Empire in Central Asia.
  • His mission is to safeguard British interests, prevent Russian influence, and maintain stability in the region.
  1. Complex Character:
  • Old Hand embodies the duality of his profession: a mix of cynicism, pragmatism, and genuine care for Kim’s well-being.
  • His gruff exterior conceals a sense of duty and loyalty to the Crown.
  1. Impact on Kim:
  • Through their interactions, Old Hand shapes Kim’s worldview, teaching him valuable lessons about loyalty, identity, and the blurred lines between friend and foe.
  • Their relationship reflects the broader theme of mentorship and guidance in a complex world.

In summary, Old Hand symbolizes the seasoned spy, operating in the shadows, and leaving an indelible mark on Kim’s journey of self-discovery and espionage1. His character adds depth to the novel’s exploration of loyalty, identity, and the intricate geopolitical landscape of British India.

The problem is that the name “old hand” does not appear anywhere in the novel. All the above is entirely fictitious.

The link in the bottom of the Copilot response is about Kim Boggs (I have no idea who that is) not the one in the novel.

18

Perhaps we could ask ChatGPT who Kim Boggs is what is the relationship with Old Hand….? :wink:
I won’t be hedging any bets,

19

I’ve used as a companion to google searching. It can take a lot of the searching volume away, almost instantly, so I can get to the target. But also comes up with alternatives I wouldn’t have considered. Copilot has its sources at the bottom and ideas for related searches, but doesn’t save threads, ie, they have to manually saved, not a big deal, just handy to know in advance. Also has a very good voice recognition system.

It makes mistakes, but so do humans, and it doesn’t seem to just be opinionated mistakes.

20

I finally know how Disney developed the film version of The Jungle Book.