ELI5: ChatGPT is a Giant Plinko Game

Photo By: sumofus

ChatGPT is like a giant Plinko game.

Plinko is a game where you put a coin in at the top. The coin falls down the board, hitting pins that influence it on the way down. At the bottom are slots, with various points or prizes. Depending on where you put your coin in at the top, you have different chances of which slot it will land it at the bottom. It's not random, some slots will be more likely than others depending on where you start.

Now, rather than a coin going in at the top, assume it's your question to ChatGPT. At the bottom, rather than prizes, assume it's all possible words. ChatGPT looks at your question, and determines based on the words you used where to put your question in at the top. Then, the pins influence your question on the "way down" depending on what the words are. Rather than landing in on a single word, ChatGPT outputs a list of the most likely words to come next. ChatGPT then picks a word from the most likely words and shows it on your screen.

After a word is generated, ChatGPT makes a new coin (your question, and the start of its response), and puts it back into the top of the Plinko board to generate the next word. This continues until ChatGPT has a response it deems a sufficient length.

This is how ChatGPT "remembers" things from your conversation. It's actually submitting as much of your chat history as it can handle, so that it can generate relevant output.

But how does a Plinko board know how to make really human like answers?

Using around 570 GB of text from the internet, the ChatGPT model (Plinko board), was trained. Parts of the text were input into the top of the Plinko board, and then the output was analyzed. Since the trainer knows what the next word is it can adjust the pins until the coin starts to predictably fall where expected. After the initial training, humans interact with the board, and can give feedback on if generated content is good or bad (which can be subjective). The trainer is its own computer program, but it still takes a long time, and is expensive to do for such a large dataset.

Does ChatGPT remember things? Is it sentient?

Since ChatGPT gives very realistic answers, some believe it could be sentient. If you believe this, you'd have to assume that a Plinko board is also sentient. IMO, for a model to be sentient or conscious it would need to be constantly running, have feedback loops, memory, and the ability to modify its internal state. ChatGPT does not modify its internal state, have a memory, and only runs (with no feedback loops) when text is input.

Deep Dive

If you want a really deep dive into how these models work, this article by Stephen Wolfram does a great job explaining many of the parts that make up something like ChatGPT.

Comments

Post a Comment

Popular posts from this blog

Phoenix LiveView: Async Assign Pattern

Image
I've been using LiveView for about two years now. It's a great framework that makes snappy and responsive pages. One anti-pattern I see fairly often is loading a lot of data in the initial page render. For the un-initiated, the mount/3 function is called twice.  Once for the initial 'dead' render, and again after the socket is connected.   Many times, for the sake of simple straight forward code, not much is done differently between these two renders. I haven't found any references (I'm sure they exist) on a best practice for managing the following flow: Set sensible, lightweight default values on initial render. Kick off one or more async processes to make longer running function calls. Receive the values in the LiveView, and update the assigns. As always in Elixir, the tools are powerful, and theres many ways to accomplish this. Spawn a linked, or unlinked process. Start a supervised, or unsupervised Task. Make a call to a GenServer, and have it send a mess
Read more

Write Admin Tools From Day One

Image
UPDATE:  I gave a talk expanding on this blog post at EMPEX NYC 2023 . The Problem Writing useful features for your users is key to a successful product.  It makes sense then that you should maximize your time writing features for those users.   This approach is very effective at the beginning of a product's life, but over time, you may find you are spending more time maintaining the product that developing it. Some of the biggest development time sucks can be: Tracking down the cause of unexpected behavior. Fixing data. Answering user, or team member questions. Running maintenance scripts, or other ad-hoc tasks. The "Oh #&@$" Solution These small support tasks start to add up, and there is no easy way to offload the work to product, or support.  You and your development team reach a breaking point where you must start prioritizing administrative tools. Unfortunately, you will slow down your new feature development even more in the short term.  It will take time for t
Read more

Custom Kaffy Styling

Image
I recently gave a talk at Empex NYC ( link to talk ) on why you should be writing admin tools from day one .  In my talk, one thing I highly suggest is to make it very obvious what environment you are in when providing a web admin UI for your team.   We've been using Kaffy at SwayDM since the beginning.  At the time, you couldn't easily overwrite the styling.  I maintain a private fork of Kaffy for changes I've made, which included the ability to change the background color. I recently merged the past few minor Kaffy releases back into my fork, and saw that Kaffy now has the ability to provide Extensions which allow you to inject custom html, css and javascript.  I spent a little time making a custom CSS file to change the color of the Kaffy nav, and add a title which notes which environment you are in. For each environment, I have a static file for CSS styling overrides. kaffy-ext-local.css kaffy-ext-staging.css kaffy-ext-production.css defmodule SwayDMWeb.Admin.Kaffy
Read more