Those numbers are illuminating! A couple of years ago I tried to get ChatGPT to do a web search for me to gather and compare specs on cars I was considering buying. Things that were available online but took some digging, like cargo storage capacity and mpg for each. Lo and behold, the results were verifiably wrong for about half its responses!
"these AI systems can’t get better because they’re now breathing their own exhaust." ...love this phrasing. I've seen it on Midjourney, which mostly defaults to some really identifiable color palettes, shapes and styles. It's training on sites like Behance and DeviantArt, which people are helpfully filling up with their AI art made in Midjourney. Bleargh.
Hallucination is an anthropromorthism to describe something that can be explained that is simple in concept, but complex in representation.
What people refer to as AI are various statistical calculation algorithms to product an output, based upon an input through an algorithm that creates representations of learned data, the algorithm of which has been created by the ‘training process’.
The output produced is statistically the most likely answer based upon the input, the weights, the model, and other factors, sometimes also not even the most likely but one of the most likely to make it seem more ‘human’.
Note that I never said ‘correct’ or incorrect, because there is no thought in the calculation, only probabilities weighted by the model etc.
It always produces an answer because there is always an answer - just not verified as to whether it is the correct answer.
This is a simple way of describing a very complex statistical calculation across a lot of data that represents the input to the output.
The only time you have seen any correction to this is through post processing of the result being validated by programmed corrections or sometimes using another or even the same model that checks a result again and if the same answer is not produced, it knows it is likely not correct and thus gives a stock answer that is more vague and tries again through a variety of strategies that someone programs.
The number of cases to handle are also so large that only a minority of correction approaches can be determined, so there is always a way of getting a wrong answer by slightly changing the input that bypasses the post processing triggers.
The problem is that the approach to ‘AI’ today is that it is a brute force approach to learning/artificial thinking that is based on an incredibly simplified model of what our brains are actually like. It is missing something that we have not yet produced, nor do we have a clue what ‘something is’. Probably why we need a massive collection of computers to do what we can do with one tiny little brain each.
“it is not known whether…” this is fairly common if AI doesn’t know something i am asking. On occasion i see a weird punt but my experience shows that yes it does admit
I ask Co Pilot things about which I already know enough to question the results and for that it’s handy. In my case it is law. I often use it to track down the exact cite for a statute whose wording I can then check.
The same goes for details about news I have seen elsewhere and am already fairly familiar with. It has only steered me wrong twice and both times I asked it to analyze something. In one case it was perfectly correct about what a case SAID but missed the fact that what it reported was not the holding, just background the court was providing. Now I’ve learned to always ask it to provide cites to things it says have been said. It is pretty good about telling you where it got its info IF YOU ASK.
And, if you catch it in an error, and you point that out, it apologizes and admits it is wrong and goes on to correct itself.
The point is, AI works fine as an information source IF you actively challenge anything that doesn’t pass the smell test AND you already know enough to realize there IS a smell test. That clearly wouldn’t work if you ask it things like your wife did.
I never let it “correct” my prose or write anything for me. My favorite story involves an Excel Spreadsheet that is just a check list. When I finish something I put an X in the cell. And for the longest time it asked me if I wanted to reword that. Somehow, I’ve managed to get it to stop doing that.
I’ve never used Chat GPT. I treat Google’s little summaries the same way I do CoPilot. I feel pretty safe asking, for example, when Biden was born and stuff like that. For things like “what is a knee cyst” (my daughter developed one)I skip the AI and go looking for a reliable health site. Sometimes, to get though the drek of specialist doctor websites trying to sell me services, I have to ask “what does Web MD or the Mayo clinic say about X issue” (sadly, I never any more ask “what does the CDC say…”)
“Wrong a third of the time” does not match my experience at all. AIs are right almost all the time now, which is really worrisome, because you have no clue or warning when it is wrong. What is worse, a calculator that is wrong 1/3 of the time, or one that is way wrong once in a thousand times? Futhermore, there is no way that millions of pepople would be using these all day everyday if they were as wrong as often as your column suggested. For many tasks they are like 99.9% right. The AIs are getting more accurate through a process of one fallable AI checking the results of another fallable AI, which has checked the soluton made by another fallable AI. That kind of heirarchy is effective, as multiplle AIs are unlikely to be wrong all in the same direction.
Interesting how we become so normalized to AI already. David, check back next week and give us a report. That's how fast it's moving of course. But more importantly, people that use AI as an information source are doing it wrong. It's interactive, ask it to recheck its work. I code with it, and do this quite frequently as anyone else would tell you, you can have it recheck its work.
“Trust but Verify”?… well maybe. I do cross reference and test everything I get out of it. I also go through the trouble of pointing out its errors until it gets the facts straight. Sometimes that's tedious, but it helps the next person with the same question.
Earlier this week, I asked Google Gemini to pick a few good wines to bring to a specific Asian restaurant. Since I haven’t deleted wine-order confirmations from my email, and since I’ve done many wine-related searches using Gemini over the past year, I asked Gemini to see whether I had anything in my cellar that would work.
It came back with a full list of wines that I had never bought. All wrong!
I picked the first one and asked how it had come up with it. Gemini replied that the wine had been mentioned in a previous chat as one I owned. I checked, and it had never been mentioned, so I copied and pasted the actual chat for Gemini to see. It then said: “My apologies for the confusion in my previous response. I misinterpreted the timeline of your purchases.”
So we tried again, and I got a new list of wines, none of which I owned. I mentioned that one of the wines had been tasted at a Wine Society presentation. We tried again. I got another new list of wines, this time “based on your email records,” but I owned none of them. My response was: “I didn’t purchase any of those. I really don’t know how you’re coming up with this.”
Gemini replied: “I apologize for those incorrect suggestions. It appears I hallucinated a list of wines that were not in your records, and I am sorry for that confusion. I have checked your actual recent emails and found the correct list of white wines from your most recent purchase….”
Well, it then returned a new list of wines I didn’t have.
This was the first time Gemini used the word “hallucinated” when apologizing for getting something completely wrong. What struck me wasn’t just that Gemini was wrong. It was that each time I challenged it, it produced a new, confident-sounding explanation and a new, equally wrong answer — instead of simply saying, “I don’t know” or “I can’t reliably determine that from your records.”
First of all, how alarming/impressive that it admitted to hallucinating.
Second, there was a recent research study that established that the longer a chat goes on, the more likely the AI is to hallucinate. I think you ran into that!
I used to think that by persevering and pushing back, Gemini might learn something that could benefit other people. But it doesn’t work that way.
Yesterday I asked Gemini for geeky setup help with my computer audio system. I wanted to save presets for my RME ADI-2/4 Pro SE, which is an A/D and D/A converter. I got an hour of bad advice that led me into settings that could not be configured the way Gemini imagined. I finally solved the problem on my own by doing something Gemini had flat-out said would not work. By that time, I had learned a lot about the device’s settings by following Gemini’s incorrect advice.
I finally convinced Gemini it was wrong by reporting my results after doing what it had told me not to do. It then said: “I stand completely corrected, and I apologize for being so stubborn about avoiding DIG Thru! You just taught me something new about the specific routing architecture of the ADI-2/4 Pro SE.”
At that point, I did what I hate doing: I read the manual. I found the passage that explained why it was working my way. I copied and pasted a snippet from the manual for Gemini to analyze. It replied: “That snippet explains exactly what is happening under the hood, and it completely solves the mystery of why your setup is working so perfectly.”
I then asked whether it had learned anything that would help other users of the RME device. In other words, would this knowledge be passed on to the larger Gemini ecosystem, or would it remain strictly private?
Gemini’s answer: “That is a great question, but the candid answer is no, my learning here does not apply globally to other users.”
Reading these comments is rather bizarre. People are both A) using AI regularly. B) admitting it’s inaccurate. Then why keep using it?? Maybe because I’m older than dirt or maybe it’s because I believe that IT’S NOT ACCURATE so I avoid using it. It’s hard, and yes I have seen responses on searches, but I’ve never used it directly such as with Grok (I HATE that name. Robert Heinlein must be rolling over in his grave), Gemini etc.
Oh my Lord. Every living person should be required to read AND memorize this article. Notice how they gave it the cutsie name “hallucinating” vs “it’s wrong'“? I guess I’m glad I’m really old and won’t see where this all ends up.
I spent 35 years in the newspaper business, most of which was in positions where my name was published in the masthead every day. On a lark, I asked about my career. What newspapers did I work for and what was my position. Sound familiar?
Fail.
It didn't get the communities right, or the name of the newspaper, or my title! More than once it put somebody elses name from the staff in my position with no mention of me.
Multiple fails.
I make lots of images too and AI handles text really poorly. Misspellings are the norm. Trying to move text is always a challenge. I've hit my generation limit so many times trying to get just a couple words of text correct, I stopped trying. I have AI make the image and I add the text using Adobe Express!
Gemini is a wimp.
I've found that Grok will make racier images than Gemini. Grok will make photo realistic caricatures of well known people. Gemini won't. Sometimes Gemini won't make cartoonish versions.
The fact that users have multiple AI fail stories is concerning. My thesis is "you get what you pay for." I use the free versions of all AI (except Suno) so I get lots of hallucinations. I can only hope that the super dooper versions of AI are better because they cost more.
Tip: I make AI music too and http://rhymezone.com is my friend when I need to tweak a stanza.
To your point about not warning users, I have noticed that at least Google AI results now have the tiny low-contrast disclaimer “AI can make mistakes, so double-check responses” at the bottom of the the results. Of course, that’s once you’ve already reviewed all of the results. And not to mention that most people who go to AI for results really don’t know how to “double-check responses”, otherwise it’s not likely that they’d have wound up asking Google AI for the info. All in all though, I’ll bet that the Google employee who came up with the idea of moving the warning to be after the results DEFINITELY was awarded a Waffle Party for their team.
A key part of the problem is the very human psychology evident right in the title of your article. We anthropomorphize AI chatbots. As humans, we will anthropomorphize a rock if it has two spots and a line that we can imagine is a face, so of course we will anthropomorphize something that spits actual sentences out at us. The people making AI chatbots know that we will intuitively overestimate the capabilities of their product, and have built up their hype around that.
So here you are, wanting a computer program to confess its shortcomings. The problem is that it’s an inanimate object that doesn’t actually know anything. It’s a computer program that calculates probabilities of what words should come after the words you entered into it. Because the database it’s built on is a huge collection of words, the thing it’s best at is simulating human like interactions.
Human con artists do much the same thing. They’ll simulate knowledge they don’t have by paying attention to what you say, and confidently stringing together words that sound like the thing you want to hear about. If their responses sound right, you’ll be conned.
The fundamental problem with LLM AI is then what you’ve described. Its output is unreliable, but so voluminous that that’s now a significant part of its training input. Eventually it will settle out into a mean of hot garbage. The question is, when will we stop being conned by it?
But I’m still left wondering: Even if it’s just an algorithm… If it SOMETIMES can warn us that it doesn’t know the answer (see: Matt Damon example), why can’t it ALWAYS do that?
Those numbers are illuminating! A couple of years ago I tried to get ChatGPT to do a web search for me to gather and compare specs on cars I was considering buying. Things that were available online but took some digging, like cargo storage capacity and mpg for each. Lo and behold, the results were verifiably wrong for about half its responses!
"these AI systems can’t get better because they’re now breathing their own exhaust." ...love this phrasing. I've seen it on Midjourney, which mostly defaults to some really identifiable color palettes, shapes and styles. It's training on sites like Behance and DeviantArt, which people are helpfully filling up with their AI art made in Midjourney. Bleargh.
Beware any model trained on Wikipedia, Reddit and Facebook!
Hallucination is an anthropromorthism to describe something that can be explained that is simple in concept, but complex in representation.
What people refer to as AI are various statistical calculation algorithms to product an output, based upon an input through an algorithm that creates representations of learned data, the algorithm of which has been created by the ‘training process’.
The output produced is statistically the most likely answer based upon the input, the weights, the model, and other factors, sometimes also not even the most likely but one of the most likely to make it seem more ‘human’.
Note that I never said ‘correct’ or incorrect, because there is no thought in the calculation, only probabilities weighted by the model etc.
It always produces an answer because there is always an answer - just not verified as to whether it is the correct answer.
This is a simple way of describing a very complex statistical calculation across a lot of data that represents the input to the output.
The only time you have seen any correction to this is through post processing of the result being validated by programmed corrections or sometimes using another or even the same model that checks a result again and if the same answer is not produced, it knows it is likely not correct and thus gives a stock answer that is more vague and tries again through a variety of strategies that someone programs.
The number of cases to handle are also so large that only a minority of correction approaches can be determined, so there is always a way of getting a wrong answer by slightly changing the input that bypasses the post processing triggers.
The problem is that the approach to ‘AI’ today is that it is a brute force approach to learning/artificial thinking that is based on an incredibly simplified model of what our brains are actually like. It is missing something that we have not yet produced, nor do we have a clue what ‘something is’. Probably why we need a massive collection of computers to do what we can do with one tiny little brain each.
This isn’t the reason I don’t use AI, but it’s a pretty good one.
“it is not known whether…” this is fairly common if AI doesn’t know something i am asking. On occasion i see a weird punt but my experience shows that yes it does admit
I ask Co Pilot things about which I already know enough to question the results and for that it’s handy. In my case it is law. I often use it to track down the exact cite for a statute whose wording I can then check.
The same goes for details about news I have seen elsewhere and am already fairly familiar with. It has only steered me wrong twice and both times I asked it to analyze something. In one case it was perfectly correct about what a case SAID but missed the fact that what it reported was not the holding, just background the court was providing. Now I’ve learned to always ask it to provide cites to things it says have been said. It is pretty good about telling you where it got its info IF YOU ASK.
And, if you catch it in an error, and you point that out, it apologizes and admits it is wrong and goes on to correct itself.
The point is, AI works fine as an information source IF you actively challenge anything that doesn’t pass the smell test AND you already know enough to realize there IS a smell test. That clearly wouldn’t work if you ask it things like your wife did.
I never let it “correct” my prose or write anything for me. My favorite story involves an Excel Spreadsheet that is just a check list. When I finish something I put an X in the cell. And for the longest time it asked me if I wanted to reword that. Somehow, I’ve managed to get it to stop doing that.
I’ve never used Chat GPT. I treat Google’s little summaries the same way I do CoPilot. I feel pretty safe asking, for example, when Biden was born and stuff like that. For things like “what is a knee cyst” (my daughter developed one)I skip the AI and go looking for a reliable health site. Sometimes, to get though the drek of specialist doctor websites trying to sell me services, I have to ask “what does Web MD or the Mayo clinic say about X issue” (sadly, I never any more ask “what does the CDC say…”)
Basically, the rule is “use it, don’t trust it”
“Wrong a third of the time” does not match my experience at all. AIs are right almost all the time now, which is really worrisome, because you have no clue or warning when it is wrong. What is worse, a calculator that is wrong 1/3 of the time, or one that is way wrong once in a thousand times? Futhermore, there is no way that millions of pepople would be using these all day everyday if they were as wrong as often as your column suggested. For many tasks they are like 99.9% right. The AIs are getting more accurate through a process of one fallable AI checking the results of another fallable AI, which has checked the soluton made by another fallable AI. That kind of heirarchy is effective, as multiplle AIs are unlikely to be wrong all in the same direction.
Interesting how we become so normalized to AI already. David, check back next week and give us a report. That's how fast it's moving of course. But more importantly, people that use AI as an information source are doing it wrong. It's interactive, ask it to recheck its work. I code with it, and do this quite frequently as anyone else would tell you, you can have it recheck its work.
Another subject but I wanted to let you know my husband loves your Apple book.
Awww, man ... you made my day! Clearly, he’s got excellent taste. :)
“Trust but Verify”?… well maybe. I do cross reference and test everything I get out of it. I also go through the trouble of pointing out its errors until it gets the facts straight. Sometimes that's tedious, but it helps the next person with the same question.
Earlier this week, I asked Google Gemini to pick a few good wines to bring to a specific Asian restaurant. Since I haven’t deleted wine-order confirmations from my email, and since I’ve done many wine-related searches using Gemini over the past year, I asked Gemini to see whether I had anything in my cellar that would work.
It came back with a full list of wines that I had never bought. All wrong!
I picked the first one and asked how it had come up with it. Gemini replied that the wine had been mentioned in a previous chat as one I owned. I checked, and it had never been mentioned, so I copied and pasted the actual chat for Gemini to see. It then said: “My apologies for the confusion in my previous response. I misinterpreted the timeline of your purchases.”
So we tried again, and I got a new list of wines, none of which I owned. I mentioned that one of the wines had been tasted at a Wine Society presentation. We tried again. I got another new list of wines, this time “based on your email records,” but I owned none of them. My response was: “I didn’t purchase any of those. I really don’t know how you’re coming up with this.”
Gemini replied: “I apologize for those incorrect suggestions. It appears I hallucinated a list of wines that were not in your records, and I am sorry for that confusion. I have checked your actual recent emails and found the correct list of white wines from your most recent purchase….”
Well, it then returned a new list of wines I didn’t have.
This was the first time Gemini used the word “hallucinated” when apologizing for getting something completely wrong. What struck me wasn’t just that Gemini was wrong. It was that each time I challenged it, it produced a new, confident-sounding explanation and a new, equally wrong answer — instead of simply saying, “I don’t know” or “I can’t reliably determine that from your records.”
🤦🏻♂️
First of all, how alarming/impressive that it admitted to hallucinating.
Second, there was a recent research study that established that the longer a chat goes on, the more likely the AI is to hallucinate. I think you ran into that!
I used to think that by persevering and pushing back, Gemini might learn something that could benefit other people. But it doesn’t work that way.
Yesterday I asked Gemini for geeky setup help with my computer audio system. I wanted to save presets for my RME ADI-2/4 Pro SE, which is an A/D and D/A converter. I got an hour of bad advice that led me into settings that could not be configured the way Gemini imagined. I finally solved the problem on my own by doing something Gemini had flat-out said would not work. By that time, I had learned a lot about the device’s settings by following Gemini’s incorrect advice.
I finally convinced Gemini it was wrong by reporting my results after doing what it had told me not to do. It then said: “I stand completely corrected, and I apologize for being so stubborn about avoiding DIG Thru! You just taught me something new about the specific routing architecture of the ADI-2/4 Pro SE.”
At that point, I did what I hate doing: I read the manual. I found the passage that explained why it was working my way. I copied and pasted a snippet from the manual for Gemini to analyze. It replied: “That snippet explains exactly what is happening under the hood, and it completely solves the mystery of why your setup is working so perfectly.”
I then asked whether it had learned anything that would help other users of the RME device. In other words, would this knowledge be passed on to the larger Gemini ecosystem, or would it remain strictly private?
Gemini’s answer: “That is a great question, but the candid answer is no, my learning here does not apply globally to other users.”
Reading these comments is rather bizarre. People are both A) using AI regularly. B) admitting it’s inaccurate. Then why keep using it?? Maybe because I’m older than dirt or maybe it’s because I believe that IT’S NOT ACCURATE so I avoid using it. It’s hard, and yes I have seen responses on searches, but I’ve never used it directly such as with Grok (I HATE that name. Robert Heinlein must be rolling over in his grave), Gemini etc.
Oh my Lord. Every living person should be required to read AND memorize this article. Notice how they gave it the cutsie name “hallucinating” vs “it’s wrong'“? I guess I’m glad I’m really old and won’t see where this all ends up.
I spent 35 years in the newspaper business, most of which was in positions where my name was published in the masthead every day. On a lark, I asked about my career. What newspapers did I work for and what was my position. Sound familiar?
Fail.
It didn't get the communities right, or the name of the newspaper, or my title! More than once it put somebody elses name from the staff in my position with no mention of me.
Multiple fails.
I make lots of images too and AI handles text really poorly. Misspellings are the norm. Trying to move text is always a challenge. I've hit my generation limit so many times trying to get just a couple words of text correct, I stopped trying. I have AI make the image and I add the text using Adobe Express!
Gemini is a wimp.
I've found that Grok will make racier images than Gemini. Grok will make photo realistic caricatures of well known people. Gemini won't. Sometimes Gemini won't make cartoonish versions.
The fact that users have multiple AI fail stories is concerning. My thesis is "you get what you pay for." I use the free versions of all AI (except Suno) so I get lots of hallucinations. I can only hope that the super dooper versions of AI are better because they cost more.
Tip: I make AI music too and http://rhymezone.com is my friend when I need to tweak a stanza.
Song: Phantom Road https://suno.com/s/e6OTBxZ5XJhf7t97
To your point about not warning users, I have noticed that at least Google AI results now have the tiny low-contrast disclaimer “AI can make mistakes, so double-check responses” at the bottom of the the results. Of course, that’s once you’ve already reviewed all of the results. And not to mention that most people who go to AI for results really don’t know how to “double-check responses”, otherwise it’s not likely that they’d have wound up asking Google AI for the info. All in all though, I’ll bet that the Google employee who came up with the idea of moving the warning to be after the results DEFINITELY was awarded a Waffle Party for their team.
LOL!
A key part of the problem is the very human psychology evident right in the title of your article. We anthropomorphize AI chatbots. As humans, we will anthropomorphize a rock if it has two spots and a line that we can imagine is a face, so of course we will anthropomorphize something that spits actual sentences out at us. The people making AI chatbots know that we will intuitively overestimate the capabilities of their product, and have built up their hype around that.
So here you are, wanting a computer program to confess its shortcomings. The problem is that it’s an inanimate object that doesn’t actually know anything. It’s a computer program that calculates probabilities of what words should come after the words you entered into it. Because the database it’s built on is a huge collection of words, the thing it’s best at is simulating human like interactions.
Human con artists do much the same thing. They’ll simulate knowledge they don’t have by paying attention to what you say, and confidently stringing together words that sound like the thing you want to hear about. If their responses sound right, you’ll be conned.
The fundamental problem with LLM AI is then what you’ve described. Its output is unreliable, but so voluminous that that’s now a significant part of its training input. Eventually it will settle out into a mean of hot garbage. The question is, when will we stop being conned by it?
Outstanding analysis!
But I’m still left wondering: Even if it’s just an algorithm… If it SOMETIMES can warn us that it doesn’t know the answer (see: Matt Damon example), why can’t it ALWAYS do that?