Jump to content

Stochastic parrot: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
Fixed error to remove the page from Category:CS1 maint: PMC format. Alter: title, template type, pmc. Add: bibcode, pages, arxiv, volume, series, chapter, authors 1-1. Removed URL that duplicated identifier. Removed access-date with no URL. Parameter name changes. | Use this tool. Report bugs. | #UCB_Gadget
Line 19: Line 19:
In July 2021, the [[Alan Turing Institute]] hosted a keynote and panel discussion on the paper.<ref>{{Cite AV media |url=https://www.youtube.com/watch?v=N5c2X8vhfBE |title=On the dangers of stochastic parrots: Can language models be too big? 🦜 |language=en |access-date=2024-04-03 |via=www.youtube.com}}</ref> {{as of|May 2023}}, the paper has been cited in 1,529 publications.<ref>{{cite web |title=Bender: On the Dangers of Stochastic Parrots |website=[[Google Scholar]] |url=https://scholar.google.com/scholar?cluster=415069420329958137 |access-date=2023-05-12}}</ref> The term has been used in publications in the fields of law,<ref>{{cite journal |last=Arnaudo |first=Luca |title=Artificial Intelligence, Capabilities, Liabilities: Interactions in the Shadows of Regulation, Antitrust – And Family Law |date=April 20, 2023 |website=SSRN |doi=10.2139/ssrn.4424363|s2cid=258636427 }}</ref> grammar,<ref>{{cite journal |title=In the Cage with the Stochastic Parrot |first=Pete |last=Bleackley |author2=BLOOM |year=2023 |journal=Speculative Grammarian |volume=CXCII |number=3 |url=https://specgram.com/CXCII.3/07.bloom.cage.html |access-date=2023-05-13}}</ref> narrative,<ref>{{cite journal |last=Gáti |first=Daniella |title=Theorizing Mathematical Narrative through Machine Learning. |journal=[[Journal of Narrative Theory]] |volume=53 |number=1 |year=2023 |pages=139–165 |publisher=Project MUSE |doi=10.1353/jnt.2023.0003|s2cid=257207529 }}</ref> and [[humanities]].<ref>{{cite journal |last=Rees |first=Tobias |title=Non-Human Words: On GPT-3 as a Philosophical Laboratory |journal=[[Daedalus (journal)|Daedalus]] |volume=151 |number=2 |year=2022 |pages=168–82 |doi=10.1162/daed_a_01908 |jstor=48662034|s2cid=248377889 |doi-access=free }}</ref> The authors continue to maintain their concerns about the dangers of [[chatbot]]s based on large language models, such as [[GPT-4]].<ref>{{Cite web |first=Sharon |last=Goldman |date=March 20, 2023 |title=With GPT-4, dangers of 'Stochastic Parrots' remain, say researchers. No wonder OpenAI CEO is a 'bit scared' |url=https://venturebeat.com/ai/with-gpt-4-dangers-of-stochastic-parrots-remain-say-researchers-no-wonder-openai-ceo-is-a-bit-scared-the-ai-beat/ |access-date=2023-05-09 |website=VentureBeat |language=en-US}}</ref>
In July 2021, the [[Alan Turing Institute]] hosted a keynote and panel discussion on the paper.<ref>{{Cite AV media |url=https://www.youtube.com/watch?v=N5c2X8vhfBE |title=On the dangers of stochastic parrots: Can language models be too big? 🦜 |language=en |access-date=2024-04-03 |via=www.youtube.com}}</ref> {{as of|May 2023}}, the paper has been cited in 1,529 publications.<ref>{{cite web |title=Bender: On the Dangers of Stochastic Parrots |website=[[Google Scholar]] |url=https://scholar.google.com/scholar?cluster=415069420329958137 |access-date=2023-05-12}}</ref> The term has been used in publications in the fields of law,<ref>{{cite journal |last=Arnaudo |first=Luca |title=Artificial Intelligence, Capabilities, Liabilities: Interactions in the Shadows of Regulation, Antitrust – And Family Law |date=April 20, 2023 |website=SSRN |doi=10.2139/ssrn.4424363|s2cid=258636427 }}</ref> grammar,<ref>{{cite journal |title=In the Cage with the Stochastic Parrot |first=Pete |last=Bleackley |author2=BLOOM |year=2023 |journal=Speculative Grammarian |volume=CXCII |number=3 |url=https://specgram.com/CXCII.3/07.bloom.cage.html |access-date=2023-05-13}}</ref> narrative,<ref>{{cite journal |last=Gáti |first=Daniella |title=Theorizing Mathematical Narrative through Machine Learning. |journal=[[Journal of Narrative Theory]] |volume=53 |number=1 |year=2023 |pages=139–165 |publisher=Project MUSE |doi=10.1353/jnt.2023.0003|s2cid=257207529 }}</ref> and [[humanities]].<ref>{{cite journal |last=Rees |first=Tobias |title=Non-Human Words: On GPT-3 as a Philosophical Laboratory |journal=[[Daedalus (journal)|Daedalus]] |volume=151 |number=2 |year=2022 |pages=168–82 |doi=10.1162/daed_a_01908 |jstor=48662034|s2cid=248377889 |doi-access=free }}</ref> The authors continue to maintain their concerns about the dangers of [[chatbot]]s based on large language models, such as [[GPT-4]].<ref>{{Cite web |first=Sharon |last=Goldman |date=March 20, 2023 |title=With GPT-4, dangers of 'Stochastic Parrots' remain, say researchers. No wonder OpenAI CEO is a 'bit scared' |url=https://venturebeat.com/ai/with-gpt-4-dangers-of-stochastic-parrots-remain-say-researchers-no-wonder-openai-ceo-is-a-bit-scared-the-ai-beat/ |access-date=2023-05-09 |website=VentureBeat |language=en-US}}</ref>


The term coined by [[Emily M. Bender|Bender]] et. al. is a neologism used by AI skeptics to insult to refer to machines' lack of understanding of the meaning of their outputs and is sometimes interpreted as a "slur against AI."<ref name=":52" /> The neologism's use expanded further when [[Sam Altman]], CEO of [[OpenAI|Open AI]], used the term ironically when he tweeted, "i am a stochastic parrot and so r u."<ref name=":52" /> The term was then designated to be the 2023 AI-related Word of the Year for the [[American Dialect Society]], beating out the words "ChatGPT" and "LLM."<ref name=":52" /><ref>{{Cite news |last=Corbin |first=Sam |date=2024-01-15 |title=Among Linguists, the Word of the Year Is More of a Vibe |url=https://www.nytimes.com/2024/01/15/crosswords/linguistics-word-of-the-year.html |access-date=2024-04-01 |work=The New York Times |language=en-US |issn=0362-4331}}</ref>
The term coined by [[Emily M. Bender|Bender]] et. al. is a neologism used by AI skeptics to refer to machines' lack of understanding of the meaning of their outputs and is sometimes interpreted as a "slur against AI."<ref name=":52" /> The neologism's use expanded further when [[Sam Altman]], CEO of [[OpenAI|Open AI]], used the term ironically when he tweeted, "i am a stochastic parrot and so r u."<ref name=":52" /> The term was then designated to be the 2023 AI-related Word of the Year for the [[American Dialect Society]], beating out the words "ChatGPT" and "LLM."<ref name=":52" /><ref>{{Cite news |last=Corbin |first=Sam |date=2024-01-15 |title=Among Linguists, the Word of the Year Is More of a Vibe |url=https://www.nytimes.com/2024/01/15/crosswords/linguistics-word-of-the-year.html |access-date=2024-04-01 |work=The New York Times |language=en-US |issn=0362-4331}}</ref>


The phrase is often referenced by some researchers to describe LLMs as pattern matchers that can generate plausible human-like text through their vast amount of training data. However, other researchers argue that LLMs are, in fact, able to understand language. While others argue they lack true understanding or the ability to reason, merely “parroting”<ref name=":62" /> in a stochastic fashion.
The phrase is often referenced by some researchers to describe LLMs as pattern matchers that can generate plausible human-like text through their vast amount of training data. However, other researchers argue that LLMs are, in fact, able to understand language. While others argue they lack true understanding or the ability to reason, merely “parroting”<ref name=":62" /> in a stochastic fashion.

Revision as of 17:46, 5 April 2024

In machine learning, the term stochastic parrot is a metaphor to describe the theory that large language models, though able to generate plausible language, do not understand the meaning of the language they process.[1][2] The term stochastic parrot was coined by University of Washington researcher Emily M. Bender in her paper "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜" co-authored by Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell.[1][3][2][4][5] There is currently no consensus on whether LLMs are stochastic parrots, with 51% of researchers believing that AI can understand language, while 49% disagree[6], but people have continued to use this term to describe such systems. [1][3][7][8]

Origin and definition

The term was first used in the paper "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜" by Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (using the pseudonym "Shmargaret Shmitchell").[5] The paper covered the risks of very large language models, regarding their environmental and financial costs, inscrutability leading to unknown dangerous biases, the inability of the models to understand the concepts underlying what they learn, and the potential for using them to deceive people.[9] The paper and subsequent events resulted in Gebru and Mitchell losing their jobs at Google, and a subsequent protest by Google employees.[10][11]

According to linguist Ben Zimmer, the word “stochastic” derives from the ancient Greek word “stokhastikos” meaning “based on guesswork,” or “randomly determined.”[3] The word "parrot" refers to the idea that large language models(LLMs) that merely repeat words without understanding their meaning.[3]

In their paper, Bender et al. argue that LLMs are probabilistically linking words and sentences together without considering meaning. Therefore, they are labeled to be mere "stochastic parrots."[5]

According to Lindholm, et. al., the analogy highlights two vital limitations:[1]

  • LLMs are limited by the data they are trained by and are simply stochastically repeating contents of datasets.
  • Because they are simply making outputs based on training data, LLMs do not understand if they are saying something incorrect or inappropriate.

Lindholm, et. al. note that because poor quality datasets and other limitations, a learning machine might produce results that are "dangerously wrong".[1]

Subsequent usage

In July 2021, the Alan Turing Institute hosted a keynote and panel discussion on the paper.[12] As of May 2023, the paper has been cited in 1,529 publications.[13] The term has been used in publications in the fields of law,[14] grammar,[15] narrative,[16] and humanities.[17] The authors continue to maintain their concerns about the dangers of chatbots based on large language models, such as GPT-4.[18]

The term coined by Bender et. al. is a neologism used by AI skeptics to refer to machines' lack of understanding of the meaning of their outputs and is sometimes interpreted as a "slur against AI."[3] The neologism's use expanded further when Sam Altman, CEO of Open AI, used the term ironically when he tweeted, "i am a stochastic parrot and so r u."[3] The term was then designated to be the 2023 AI-related Word of the Year for the American Dialect Society, beating out the words "ChatGPT" and "LLM."[3][19]

The phrase is often referenced by some researchers to describe LLMs as pattern matchers that can generate plausible human-like text through their vast amount of training data. However, other researchers argue that LLMs are, in fact, able to understand language. While others argue they lack true understanding or the ability to reason, merely “parroting”[7] in a stochastic fashion.

Debate

Some LLMs, such as ChatGPT, have become capable of interacting with users in convincingly human-like conversations.[7] The development of these new systems has deepened the discussion of the extent to which LLMs are simply “parroting.”

In the mind of a human being, words and language correspond to things one has experienced.[8] For LLMs, words correspond only to other words and patterns of usage fed into their training data.[20][21][5] Proponents of the idea of stochastic parrots thus conclude that LLMs are incapable of actually understanding language.[20][5]

The tendency of LLMs to pass off fake information as fact is held as support.[8] Called hallucinations, LLMs will occasionally synthesize information that matches some pattern, but not reality.[20][21][8] That LLMs can’t distinguish fact and fiction leads to the claim that they can’t understand language at all.[20][8]  Further, LLMs often fail to decipher complex or ambiguous grammar cases that rely on understanding the meaning of language..[20][21] As an example, borrowing from Saba et. al., is the prompt:[20]

“The wet newspaper that fell down off the table is my favorite newspaper. But now that my favorite newspaper fired the editor I might not like reading it anymore. Can I replace ‘my favorite newspaper’ by ‘the wet newspaper that fell down off the table’ in the second sentence?”

LLMs respond to this in the affirmative, not understanding that the meaning of the newspaper is different in these two contexts.[20] Based on these failures, some AI professionals conclude they are no more than stochastic parrots.[20][8][5]

However, there is support for the claim that LLMs are more. LLMs do pass many tests for understanding well, such as Super General Language Understanding Evaluation (SuperGLUE).[21][22]  Tests such as these and the smoothness of many LLM responses help as many as 51% of AI professionals believe they can truly understand language with enough data, according to 2022 survey.[21]

Another technique which has been applied to show this is termed "mechanistic interpretability". The idea is to reverse-engineer a large language model by discovering symbolic algorithms that approximate the inference performed by LLM. One example is Othello-GPT, where a small transformer is trained to predict legal Othello moves. It is found that there is a linear representation of Othello board, and modifying the representation changes the predicted legal Othello moves in the correct way.[23][24] In another example, a small Transformer is trained on Karel programs.

Similar to the Othello-GPT example, there is a linear representation of Karel program semantics, and modifying the representation changes output in the correct way. The model also generates correct programs that are on average shorter than those in the training set.[25]

However, when tests created to test people for language comprehension are used to test LLMs, they sometimes result in false positives caused by spurious correlations within text data.[26] Models have shown examples of shortcut learning, which is when a system makes unrelated correlations within data instead of using human-like understanding.[27] One such experiment tested Google’s BERT LLM using the argument reasoning comprehension task. They asked it to choose between 2 statements, which is more consistent with an argument. Below is an example of one of these prompts:[21][28]

"Argument: Felons should be allowed to vote. A person who stole a car at 17 should not be barred from being a full citizen for life.

Statement A: Grand theft auto is a felony.

Statement B: Grand theft auto is not a felony."

Researchers found that specific words such as “not” hint the model towards the correct answer, allowing near-perfect scores when included but resulting in random selection when hint words were removed.[21][28] This problem, and the known difficulties defining intelligence, causes some to argue all benchmarks that find understanding in LLMs are flawed, allowing shortcuts to fake understanding.

Without a reliable benchmark, researchers have found difficulties differentiating models between stochastic parrots and entities capable of understanding. When experimenting on ChatGPT-3, one scientist argued that the model was in between true human-like understanding and being a stochastic parrot.[7] He found that the model was coherent and informative when attempting to predict future events based on the information in the prompt.[7] ChatGPT-3 was frequently able to parse subtextual information from text prompts as well. However, the model frequently failed when tasked with logic and reasoning, especially when these prompts involved spatial awareness.[7] The model’s varying quality of responses indicates that LLM models may have a form of “understanding” in certain categories of tasks while acting as a stochastic parrot in others.[7]

See also

References

  1. ^ a b c d e Lindholm, Andreas (2022). Machine learning: a first course for engineers and scientists. Cambridge, UK ; New York, NY: Cambridge University Press. ISBN 978-1-108-84360-7.
  2. ^ a b Uddin, Muhammad Saad (April 20, 2023). "Stochastic Parrots: A Novel Look at Large Language Models and Their Limitations". Towards AI. Retrieved 2023-05-12.
  3. ^ a b c d e f g Zimmer, Ben. "'Stochastic Parrot': A Name for AI That Sounds a Bit Less Intelligent". WSJ. Retrieved 2024-04-01.
  4. ^ Weil, Elizabeth (March 1, 2023). "You Are Not a Parrot". New York. Retrieved 2023-05-12.
  5. ^ a b c d e f Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (2021-03-03). "On the Dangers of Stochastic Parrots: Can Language Models be Too Big? 🦜". Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. ACM. pp. 610–623. doi:10.1145/3442188.3445922. ISBN 978-1-4503-8309-7.
  6. ^ Michael, Julian; Holtzman, Ari; Parrish, Alicia; Mueller, Aaron; Wang, Alex; Chen, Angelica; Madaan, Divyam; Nangia, Nikita; Pang, Richard Yuanzhe (2022-08-26), What Do NLP Researchers Believe? Results of the NLP Community Metasurvey, arXiv:2208.12852
  7. ^ a b c d e f g Arkoudas, Konstantine (2023-08-21). "ChatGPT is no Stochastic Parrot. But it also Claims that 1 is Greater than 1". Philosophy & Technology. 36 (3): 54. doi:10.1007/s13347-023-00619-6. ISSN 2210-5441.
  8. ^ a b c d e f Fayyad, Usama M. (2023-05-26). "From Stochastic Parrots to Intelligent Assistants—The Secrets of Data and Human Interventions". IEEE Intelligent Systems. 38 (3): 63–67. doi:10.1109/MIS.2023.3268723. ISSN 1541-1672.
  9. ^ Haoarchive, Karen (4 December 2020). "We read the paper that forced Timnit Gebru out of Google. Here's what it says". MIT Technology Review. Archived from the original on 6 October 2021. Retrieved 19 January 2022.
  10. ^ Lyons, Kim (5 December 2020). "Timnit Gebru's actual paper may explain why Google ejected her". The Verge.
  11. ^ Taylor, Paul (2021-02-12). "Stochastic Parrots". London Review of Books. Retrieved 2023-05-09.
  12. ^ On the dangers of stochastic parrots: Can language models be too big? 🦜. Retrieved 2024-04-03 – via www.youtube.com.
  13. ^ "Bender: On the Dangers of Stochastic Parrots". Google Scholar. Retrieved 2023-05-12.
  14. ^ Arnaudo, Luca (April 20, 2023). "Artificial Intelligence, Capabilities, Liabilities: Interactions in the Shadows of Regulation, Antitrust – And Family Law". SSRN. doi:10.2139/ssrn.4424363. S2CID 258636427.
  15. ^ Bleackley, Pete; BLOOM (2023). "In the Cage with the Stochastic Parrot". Speculative Grammarian. CXCII (3). Retrieved 2023-05-13.
  16. ^ Gáti, Daniella (2023). "Theorizing Mathematical Narrative through Machine Learning". Journal of Narrative Theory. 53 (1). Project MUSE: 139–165. doi:10.1353/jnt.2023.0003. S2CID 257207529.
  17. ^ Rees, Tobias (2022). "Non-Human Words: On GPT-3 as a Philosophical Laboratory". Daedalus. 151 (2): 168–82. doi:10.1162/daed_a_01908. JSTOR 48662034. S2CID 248377889.
  18. ^ Goldman, Sharon (March 20, 2023). "With GPT-4, dangers of 'Stochastic Parrots' remain, say researchers. No wonder OpenAI CEO is a 'bit scared'". VentureBeat. Retrieved 2023-05-09.
  19. ^ Corbin, Sam (2024-01-15). "Among Linguists, the Word of the Year Is More of a Vibe". The New York Times. ISSN 0362-4331. Retrieved 2024-04-01.
  20. ^ a b c d e f g h Saba, Walid S. (2023). "Stochastic LLMS do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMS". In Almeida, João Paulo A.; Borbinha, José; Guizzardi, Giancarlo; Link, Sebastian; Zdravkovic, Jelena (eds.). Conceptual Modeling. Lecture Notes in Computer Science. Vol. 14320. Cham: Springer Nature Switzerland. pp. 3–19. arXiv:2309.05918. doi:10.1007/978-3-031-47262-6_1. ISBN 978-3-031-47262-6.
  21. ^ a b c d e f g Mitchell, Melanie; Krakauer, David C. (2023-03-28). "The debate over understanding in AI's large language models". Proceedings of the National Academy of Sciences. 120 (13): e2215907120. arXiv:2210.13966. Bibcode:2023PNAS..12015907M. doi:10.1073/pnas.2215907120. ISSN 0027-8424. PMC 10068812. PMID 36943882.
  22. ^ Wang, Alex; Pruksachatkun, Yada; Nangia, Nikita; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2019-05-02). "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". arXiv.org. arXiv:1905.00537. Retrieved 2024-04-04.
  23. ^ Li, Kenneth; Hopkins, Aspen K.; Bau, David; Viégas, Fernanda; Pfister, Hanspeter; Wattenberg, Martin (2023-02-27), Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, arXiv:2210.13382, retrieved 2024-04-04
  24. ^ Li, Kenneth (2023-01-21). "Large Language Model: world models or surface statistics?". The Gradient. Retrieved 2024-04-04.
  25. ^ Jin, Charles; Rinard, Martin (2023-05-24), Evidence of Meaning in Language Models Trained on Programs, arXiv:2305.11169, retrieved 2024-04-04
  26. ^ Choudhury, Sagnik Ray; Rogers, Anna; Augenstein, Isabelle (2022-09-15), Machine Reading, Fast and Slow: When Do Models "Understand" Language?, arXiv:2209.07430, retrieved 2024-04-04
  27. ^ Geirhos, Robert; Jacobsen, Jörn-Henrik; Michaelis, Claudio; Zemel, Richard; Brendel, Wieland; Bethge, Matthias; Wichmann, Felix A. (2020-11-10). "Shortcut learning in deep neural networks". Nature Machine Intelligence. 2 (11): 665–673. arXiv:2004.07780. doi:10.1038/s42256-020-00257-z. ISSN 2522-5839.
  28. ^ a b Niven, Timothy; Kao, Hung-Yu (2019-09-16), Probing Neural Network Comprehension of Natural Language Arguments, arXiv:1907.07355, retrieved 2024-04-04

Further reading

External links

Retrieved from "https://en.wikipedia.org/w/index.php?title=Stochastic_parrot&oldid=1217412916"