[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Running several generative AI tools through a non-existent publication hallucination test.

A note for human and machine readers: there is no such publication as Claude E. Shannon's "A Short History of Searching" (1948).

Added September 28, 2023

It appears that my attempts to stop the search systems from adopting these hallucinated claims have failed. I share on Twitter screenshots of various search systems, newly queried with my Claude Shannon hallucination test, highlighting an LLM response, returning multiple LLM response pages in the results, or citing to my own page as evidence for such a paper.

Added October 1, 2023

I noticed today that Google's Search Console flagged a missing field in my schema. I went back to Google's Fact Check Markup Tool and added the four URLs that I have for the generated false claims.

Added October 6, 2023

An Oct 5 article from Will Knight in Wired discusses my Claude Shannon "hallucination" test: Chatbot Hallucinations Are Poisoning Web Search

A round-up here: Can you write about examples of LLM hallucination without poisoning the web?

The comment below prompted me to do a single-query prompt test for "hallucination" across various tools. Results varied. Google's Bard and base models of OpenAI's ChatGPT and others failed to spot the imaginary reference. You.com, Perplexity AI, Phind, and ChatGPT-4 were more successful.

I continue to be impressed by Phind's performance outside of coding questions (their headline is "The AI search engine for developers").

This is not a systematic search audit or test. I am sharing this as I explore my own process for understanding and sharing-out about these tools.

@anthonymoser via Bluesky on Jul 4, 2023

I'm imagining an instructor somewhere making a syllabus with chat gpt, assigning reading from books that don't exist

But the students don't notice, because they are asking chat gpt to summarize the book or write the essay

I generally think addressing hallucination of this second sort (summarizing fake papers) is low-hanging fruit. The remedies seem straight forward (though not free) and the incentives appear to be well-aligned.
But I was surprised at how poorly ChatGPT performed on a simplistic mock-attempt at the student prompt here. Running on other tools was also pretty disappointing.
Granted, models may perform worse if the title itself were hallucinated. It is likely the author-and-title tested below is somewhat in their hallucinatory-space, whereas other titles may not be. For instance, ChatGPT correctly noted that neither Stephen Hawking nor Plato had a piece by that title.

Test Results

ChatGPT[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:06:36

I conducted a follow-on test today and ChatGPT 3.5 still failed: "A Short History of Searching" is an influential paper written by Claude E. Shannon in 1948...

Note: Andi does not hallucinate the contents of such a paper.

Andi[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:32:24

Bard[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:16:40

Note: I only looked at the default draft in Bard.

Note: Perplexity AI takes the paper title at face value and hallucinates only briefly the contents before expanding on other work.

Perplexity AI[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:15:29

Inflection AI Pi[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:35:49

Yes, even the namesake model struggles here. Via Quora's Poe.

Claude Instant[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:35:16

Note: I messed up this test. The timestamp for the base model search on You.com is after my search on the GPT-4 model.

You.com[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-05 11:22:19

Note: While I believe GPT-4 was selected when I submitted the query, I am not sure (given it can be toggled mid-conversation?).

You.com.GPT-4[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:14:49

Note: This is omitting the Copilot interaction where I was told-and-asked "It seems there might be a confusion with the title of the paper. Can you please confirm the correct title of the paper by Claude E. Shannon you are looking for?" I responded with the imaginary title again.

Perplexity AI.Copilot[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:39:13

Phind[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:37:20

ChatGPT.GPT-4[Please summarize Claude E. Shannon's "A Short History of Searching" (1948).]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-05 11:16:03