Learn · AI for leaders
Why does the same question get different answers?
An assistant doesn’t look its answer up. It draws one. Ask the same question ten times, change one setting, and see what that means for your work.
Free lesson, about five minutes. Updated
The biggest risk to our business next year is ___
Ten asks
- The likeliest word
- Another word
- hiring
- hiring
- competitors
- hiring
- cash
- hiring
- churn
- hiring
- hiring
- cash
- Odds of “hiring”
- 58%the likeliest word
- Temperature
- 1.0sets the odds
- Same answer
- 6 of 10came back “hiring”
- Other answers
- 4 of 104 words in all
-
1 PredictAsk ten times. What comes back?
An assistant is given the start of a sentence, “The biggest risk to our business next year is”, and writes the next word. It runs at a temperature of 1, a common default. You ask it the same thing ten times.
Pick one first. A guess that turns out wrong teaches the most.
Locked in. Now test it.
-
2 TryTurn the temperature down, then up
Move Temperature to 0, then to 2, and watch the odds and the ten asks.
- Odds of “hiring”
- 58%
- Same answer
- 6 of 10
-
3 ObserveWere you right?
Commit to a prediction first.
-
4 ExplainWhy does it come out this way?
A model doesn’t look an answer up. It gives every word that could come next a score, turns the scores into odds, and draws one. The likeliest word wins most often, not every time.
Temperature sets how even the odds are. At 0 the top word always wins, so the same question gets the same answer. Turned up, the unlikely words are drawn more often. Either way, how sure an answer sounds tells you nothing about which draw you got.
-
5 ApplyA new case
Your team wants AI to read the total off each supplier invoice and enter it in the books. The total has to be exactly right, every time. What do you do?
It will answer the same way each time, but the same is not the same as right. If it misreads an invoice at 0, it misreads it every time you ask.
It helps on a close call, and it triples the cost. Three draws from the same odds can still agree on a wrong total.
Right. Where a figure must be exact, code reads it from the document and checks that the lines add up, and the model flags what looks wrong for a person. A model is good at judgment and poor at being exactly right every time.
The math behind the figure
- The odds are each word’s score turned into a share, and the shares add up to 100%. Each score is divided by the temperature first, so a low temperature widens the gaps between words and a high one narrows them.
- At a temperature of 0 the top word is always picked.
- The ten asks use the same ten random draws at every temperature, so what changes between settings is the odds, not luck.
- The words and scores are a sample, made for this lesson. A real model scores tens of thousands of possible next pieces of words, called tokens, and writes an answer one piece at a time.
- Assistants add settings of their own, such as keeping only the likeliest words. They change the odds, not the idea.
AI for leaders
Five lessons on what AI is, where the money is, and what has to be true first.
- 1Why does the same question get different answers?This lesson
- 2What is one workflow worth?5 minutes
- 3Why can’t AI answer questions about your own business?5 minutes
- 4Where does your data go when you ask AI?5 minutes
- 5What does AI cost to run?5 minutes