Why is it called ChatGPT?

To see what changed, look at what kept breaking

To understand what made ChatGPT special, it helps to rewind a bit and look at what came before it, and where those older systems kept failing.

Years earlier, scientists built early AI language systems called RNNs (Recurrent Neural Networks). Think of an RNN as reading a book with a tiny flashlight in a dark room. It can only see one word at a time, and it barely remembers what it saw a few pages back.

Try this yourself. Read this short story:

“I grew up in India. After completing my bachelor's degree, I moved to the United States. Today I work as an AI architect. Which country do I currently live in?”

Easy, right? You said the United States right away, even though it was mentioned way back at the start. An RNN really struggles with this. The further back something is, the more it fades away, almost like a memory slipping out of reach. Scientists gave this a name: the long-term dependency problem. In plain words, it just means old information gets forgotten too fast.

So scientists tried to fix it. In 1997, they built something called an LSTM (Long Short-Term Memory network). Instead of forgetting everything, an LSTM is a bit smarter. It decides what's worth remembering and what it can let go, kind of like taking notes instead of trying to memorize every single word.

It was a real step forward, and for a long time, LSTMs were the best tool around. But they still had one big problem: they read one word at a time, just like RNNs did. That made them slow to train, and they still couldn't hold onto very much information at once.

What if it just looked back, instead of trying to remember?

In 2014, a group of researchers had a simpler idea, described in a paper on neural machine translation. Instead of forcing the AI to remember everything, what if it could just look back at the original sentence, whenever it actually needed to?

That idea became something called attention. Instead of treating every word the same, the AI learns which earlier words actually matter for the word it's trying to predict right now, and focuses on those. It didn't fix everything by itself. But it was the spark that led to the next big breakthrough.

Then a team at Google rebuilt the whole thing around it

Three years later, in 2017, a team at Google took that spark and built something much bigger around it. They called it the Transformer, in a paper called “Attention Is All You Need.”

Here's what made it different. Instead of reading a sentence word by word like RNNs and LSTMs did, a Transformer reads the whole sentence all at once. It looks at every word at the same time and figures out how each word relates to every other word. This is called self-attention.

That one change fixed a lot of problems at once. Transformers understood long sentences better. They trained faster. And because they could work on many words at the same time, they could run on powerful GPU chips and scale up to sizes nobody had tried before.

Quick heads-up, because this trips people up later: “all at once” describes training. While the model is learning, it can look at a whole sentence in one go, which is what makes training so much faster. But once training is done and it's actually writing a reply to you, it's back to producing one piece at a time, each new piece built on the ones before it. You'll see exactly what that looks like in a minute. Both things are true. They just happen at different moments.

Here's a simple way to see why that matters. Read these two sentences:

“Apple released a new laptop.”
“I ate an apple.”

Same word. Two completely different meanings. You figured out the difference instantly, just from the words around it. A Transformer does something very similar. It looks at the neighboring words to figure out which “apple” it's dealing with.

Today, almost every well-known AI model, including GPT, Llama, Gemini, Claude, Mistral, DeepSeek, and Qwen, is built on this same Transformer idea. Different companies, different training data, same basic blueprint underneath.

Line it all up, and it's less one invention, more a chain of fixes

Each step below exists because of exactly what broke in the version before it.

Which brings us back to where we started

The name ChatGPT is really just two words stuck together: Chat, and GPT.

Chat simply means “a conversation.” Older software made you click through menus and buttons, or learn special commands. ChatGPT doesn't. You just type what you want, in your own words. Ask a question, request a poem, get help with your code, and it replies like a person would. That's a big reason so many people could use it right away, without needing to be a tech expert.

Here's a distinction worth making early, because it saves confusion later. ChatGPT and the model underneath it are not the same thing.

The model, GPT, is really just an engine. Hand it some text, and it predicts what comes next. That's genuinely all it does. On its own it doesn't manage a conversation, remember your name, search the web, or open a file you upload.

ChatGPT is the app built around that engine. It's the part that quietly gathers everything useful, instructions for how to behave, your conversation so far, maybe a file you shared, and hands the whole bundle to the model. The model writes a reply, and the app shows it to you as a clean chat bubble.

Keeping these two apart is a genuinely useful habit. When something feels off, the AI forgets what you said, or won't do something you asked, it's very often a choice the app made, not some fixed law of how the model works.

Wait, what's a token?

Before we get to Generative, one word is worth two minutes right now: token. It's going to keep showing up.

A model doesn't actually read text one whole word at a time. First, it chops your text into small pieces called tokens. A token can be a full word, a piece of a word, a punctuation mark, even a space.

Take “Kubernetes is powerful.” You read that as three words. A model is more likely to see several smaller pieces, maybe something like Kub, ernetes, is, and powerful. The exact split depends on the model. That's the general idea, not an exact rule.

Here's the part that actually matters: a neural network can't read letters. It only works with numbers. So every token gets swapped for a number, called a token ID, before the model ever touches it.

One more small step happens next. A plain ID number doesn't carry any sense of meaning by itself, it's just a label. So each one gets turned into an embedding: a longer list of numbers that captures a bit of what the token means. Do this well, and words with similar meanings land close together, mathematically speaking. “King” and “queen” end up as neighbors. “King” and “banana” end up far apart. You don't need the math behind that. Just the idea: an embedding is how meaning gets turned into something a neural network can actually work with.

How much can it actually keep in view?

That resending trick from a minute ago raises an obvious question. If the whole conversation gets resent every single time, what happens once a conversation gets really long?

There's a limit, called the context window: the maximum number of tokens a model can look at in one go. That's why you'll sometimes see it described in a slightly odd unit, like “128,000 tokens,” not words or pages. Go past that limit, and older parts of the conversation have to get dropped, summarized, or trimmed to make room.

And that window isn't only your messages. It includes everything the app is quietly attaching too: its own instructions, your past messages, and sometimes a document or a tool's result. All of it is competing for the same limited space.

It builds the answer, one small piece at a time

Generative means the AI doesn't search a database to find an answer. It actually writes a brand new one, from scratch, every single time.

Here's how. It writes one token at a time, the small chunks you just met. It picks the next one, adds it to what it already wrote, and asks itself: okay, what comes next? It repeats this, over and over, until it decides the answer is done.

Because each new piece depends on what it just wrote a moment ago, this is called an autoregressive model. That's just a fancy way of saying it builds on its own last step, again and again. If you want to see this in action, try this notebook. It shows the AI picking words one at a time, right in front of you, starting from a sentence like “After weeks of consistent training, the runner finally…”

Curious what a real run actually produced? Here's the genuine output from that notebook: the model's real probabilities for the very next piece after “…the runner finally.”

Now, a natural question. Looking at that list, does the model always just take the top pick, “cross” at 64.2%, every time?

Not necessarily. That's controlled by a setting called temperature. At a low temperature, the model plays it safe and almost always takes the top candidate. Turn the temperature up, and it gets more willing to gamble on something further down the list, which is a big part of why the same prompt can give you a slightly different answer each time you ask. You don't need the math behind it, just the idea: temperature is a dial for how adventurous the guessing gets.

Before it ever met you, it did the reading

Before we get into what “pre-trained” means, let's quickly talk about what a neural network actually is.

A neural network is a computer program loosely inspired by the human brain. Your brain has billions of neurons, all connected together, learning from experience. A neural network works a bit like that too, except its “neurons” are just numbers, connected by other numbers, and it learns by slowly adjusting those numbers over time.

See those connections carrying numbers into each neuron? Every single one has a weight attached: a number that decides how much that particular input should matter. Those weights, plus a few other adjustable numbers, are what people mean by a model's parameters.

This is also where those model names that get thrown around, 7B, 70B, 405B, finally make sense. That's just the parameter count: 7 billion, 70 billion, 405 billion individual numbers, each one nudged into place during training. More parameters usually means more room to capture subtle patterns. It doesn't automatically mean a better model, though. How a model is trained matters just as much as how big it is.

Okay, now back to Pre-trained. It means the AI already read a huge amount of text before you ever typed a single word to it. Think of it like a student who spends years in school before their very first day at work. In the same way, a large language model spends a long time “reading” before it ever answers a real question, picking up grammar, facts, and how ideas connect to each other.

A good real example is Meta's Llama model. Meta actually shared what they used to train it, in the Llama paper: Wikipedia, books, code from GitHub, Common Crawl (a huge archive of the internet), and more. Here's the exact mix, straight from the paper:

Here's roughly how that reading actually works, broken into small steps. The model reads a piece of text and predicts the next token. Then it checks that guess against what the text really says next.

The gap between its guess and the real answer gets turned into a single number, called the loss. Think of the loss as a score for how wrong it was: lower is better. From there, a method called backpropagation traces that error backward through the network and works out which of the model's billions of parameters were most responsible for it. An optimizer then nudges each of those parameters very slightly, just enough to do a bit better next time.

Once all that reading is done, the model is considered pre-trained. But here's a detail worth knowing: pre-training alone doesn't get you something like ChatGPT. What comes out is called a base model, and it's a little raw: excellent at continuing text, but not yet great at being a helpful, well-behaved assistant. Getting from there to something you'd actually want to talk to takes a couple more steps.

The part you already met

And T is the same architecture from a few sections back: the Transformer, introduced in 2017. Instead of reading one word at a time like the RNNs and LSTMs before it, a Transformer looks at every word in a sentence at the same time. It's the foundation that almost every AI model today is built on.

One thing worth holding onto from earlier: that “all at once” happens during training. When it's actually writing a reply to you, it's back to one token at a time, the Generative part you already met.

So does it actually know what's true?

Here's a question worth sitting with for a second. If GPT is just predicting the next likely token, based on patterns it picked up during training, does it actually know whether what it's saying is true?

Not exactly. It's generating text that sounds right, based on patterns, not checking each sentence against reality as it goes. Most of the time, those two things line up fine. Sometimes they don't, and it says something confidently wrong anyway. That gap has a name, and it's a big enough topic to earn its own lesson later in this course.

Put it all together, and it stops looking random

Chat is the conversation you have with it. Generative means it writes new answers, one piece at a time. Pre-trained means it already did a huge amount of reading before it ever met you. Transformer is the design, built in 2017, that made all of this fast enough to actually work.

Four words. One name. And on November 30, 2022, all of it came together into a product that 100 million people were using within two months.

Zoom out, and everything in this lesson is really one loop, repeated: