Lesson 4 of 5 · Beyond the Basics

Voice, video, and seeing

Modern AI can look at a photo, hold a spoken conversation, and make a short video clip. Underneath, it is the same trick you already know: turn everything into tokens and predict what comes next.

For most of its short life, chat AI was a text box. You typed, it typed back. Today the same apps will look at a picture of your sprinkler valve, talk with you while you drive, and watch through your phone camera while you point at a breaker panel. That can feel like a brand new kind of machine. It mostly is not. This lesson shows what actually changed, what still goes wrong, and why voice mode in particular feels so different from typing.

One model, many senses

Multimodal means a model that works with more than one kind of input or output: text, images, sound, and sometimes video. "Mode" here just means a type of data. A text only model is unimodal. A model that can read your question and look at your photo in the same breath is multimodal.

The key idea is that the model did not grow eyes and ears. Instead, engineers found ways to turn pictures and sound into the same kind of thing the model already handles, which is tokens. You met tokens in Tokens, context, and system prompts: the small chunks a model reads and writes. Once a photo or a sound clip becomes tokens, it sits in the context right next to your words, and the model predicts a reply the same way it always does.

That has two practical effects. First, images and audio take up room. A photo can use as many tokens as several pages of text, which is part of why long voice or camera sessions run into limits sooner. Second, the model's strengths and weaknesses with pictures come from how that conversion works, which is the next section.

How a model "sees" a photo

When you upload a picture, the system cuts it into a grid of small squares, like tiles on a floor. Each tile is turned into a token that describes what is in that patch: color, edges, texture, a bit of shape. Anthropic's documentation, as of October 2026, describes Claude viewing images "in patches instead of pixels," with each patch a 28 by 28 pixel block. Other vendors use their own sizes, but the idea is the same everywhere.

Big photos are usually shrunk before they are tiled, so there is a cap on how many tiles one image gets. A phone photo of a whole room and a close up of one receipt may end up with a similar number of tiles, which means the receipt gets far more detail per word.

During training, the model saw an enormous number of images paired with text that described them. It learned which tile patterns tend to go with words like "dog," "stucco," or "cracked windshield." So when it describes your photo, it is doing what it always does: predicting the most likely good answer, now with picture tokens in the mix.

What it gets wrong

Vision models are strong at the overall gist. They can tell you a photo shows a patio with a pergola, name the style of a ceiling fan, or read a clear sign. They stumble in a few predictable places. Anthropic's own documentation lists several of them as of October 2026, and the same pattern shows up across tools:

Counting. "About a dozen chairs" comes out fine. "Exactly fourteen" often does not, especially with many small, similar objects.
Small or blurry text. Fine print, a serial number on a water heater, or a sign far in the background may get guessed rather than read. If the photo was shrunk, the letters may be gone before the model ever sees them.
Left, right, and exact position. "Which pipe is on the left?" or "how far is the fence from the house?" are shakier than "what is in this picture?"
Spotting fakes. Anthropic says plainly that Claude cannot tell whether an image is AI-generated. Do not ask a chatbot to judge if a photo is real. The next lesson covers what to do instead.

Just like with text, the model will not usually say "I am not sure." It gives a confident answer either way. That is the same problem you met in Trust and limits, now wearing a camera.

The hard part, simply

Why can the AI describe your photo beautifully and still miscount the chairs? Because describing and counting are different jobs. A good description only needs the gist, and the gist is spread across many tiles. Counting needs the model to keep track of each separate chair, one by one, and nothing in the tiles says "this is chair number six."

Think of glancing into a busy restaurant in St. George on a Friday night. In one second you can say "packed, families, lots of booths." Ask how many people are inside and you would have to stop and count, pointing as you go. The model is very good at the glance and has no finger to point with.

Worked example: you photograph a banquet room set up for a Hurricane wedding, with 48 chairs around six tables. The AI might say something like "an elegant reception space with round tables, white linens, and about 40 chairs." The description is right. The number is off, and it sounds just as sure about the number as about the linens. If the count matters, ask it to count one table at a time, or count yourself.

Voice mode: pipeline versus native

There are two broad ways to build an AI you can talk to. The older way is a pipeline, three separate tools chained in a row. A speech to text model writes down what you said. The chat model reads that text and writes a reply. A text to speech model reads the reply out loud. It works, but each handoff adds a pause, and everything except your words is thrown away at the first step. Sarcasm, worry, and a long pause all become the same flat line of text.

The newer way is native voice: one model that takes in sound as tokens and produces sound as tokens. Nothing gets flattened into text in between, so it can pick up on tone and pace, and it can keep listening while it talks. That is why you can cut it off mid sentence and it stops, the way a person would.

Tap: two ways an AI can talk back

1
You speak
The app waits until you stop talking before it does anything.
2
Speech to text
A transcription model turns your voice into plain words. Your tone is dropped here.
3
The model reads and writes
The chat model sees only the typed words and writes a text reply.
4
Text to speech
A separate voice model reads the reply out loud.
5
You hear it
Three handoffs later, the answer arrives.

Wait before you hear a reply: Noticeable pause

Waits for you to finishHears words, not toneHard to interrupt

Illustrative. Step names are simplified and the delay bars show the idea, not measured times. Real apps vary by product and connection.

Where things stand in the big apps, as of October 2026:

ChatGPT offers two voice options. Live runs on OpenAI's GPT-Live-1 or GPT-Live-1 mini models, which OpenAI describes as able to listen and speak at the same time, so you can interrupt naturally. Advanced is the earlier real time voice, and it is the one that supports video and screen sharing.
Gemini Live lets you talk with Gemini in the mobile app and interrupt it just by talking.
Claude has a voice mode, still labeled beta, on all plans across mobile, desktop, and web. You can interrupt it, and it offers a hands free mode and a push to talk mode.

From the outside you often cannot tell how a given voice feature is built, and vendors change it often. The quick test is in the explainer: if it can hear that you sound frustrated and lets you interrupt cleanly, it is probably closer to native.

Live camera and screen sharing

The next step is letting the AI watch while you talk. As of October 2026, ChatGPT's Advanced voice lets eligible subscribers turn on the camera or share their screen in the iPhone and Android apps, with usage limits. The newer Live option does not do video or screen sharing. Gemini Live lets you share your camera and your screen from the Gemini mobile app; the camera turns off if you leave the app or lock your phone. Claude's voice mode help page, as of October 2026, does not list a camera or screen sharing feature.

A simple way to picture it: a live camera works less like a movie and more like a fast series of snapshots, each one handled much like a photo. That is why everything from the "what it gets wrong" list still applies, and why a slow, steady camera helps more than you would think. It also means the AI can only use what the camera is showing right now. Point the phone away and that part is gone.

One privacy note: screen sharing usually sends your whole screen, notifications and all. Close anything private before you start. For more on what happens to what you share, see Privacy for your files.

How AI makes video

Video generators run the same core process as image generators, which you saw in How AI builds images: start from static and remove noise step by step until a picture is left. The difference is that a video model clears the noise from every frame at once, so each frame has to agree with the ones before and after it. A truck that is red in frame one has to stay red, and it has to move a believable distance by frame thirty.

That agreement is the hard and expensive part. A few seconds of video is dozens of images that all have to match each other, so it takes far more computing power than a single picture. That is why clips are short. Google's Veo 3.1 developer documentation, as of October 2026, lists clip lengths of 4, 6, or 8 seconds, with sound generated along with the picture. Longer videos are usually several short clips stitched or extended.

The field moves fast in both directions. OpenAI's Sora video models, for example, were removed from OpenAI's developer API on September 24, 2026. Treat any specific video product as something to check the day you need it, not something to memorize.

Try this yourself

You do not need anything new for this. Use the chat app already on your phone, and just watch what happens.

1. Count test. Take a photo of a room or shelf with somewhere between 10 and 25 similar things in it: chairs, cans, flower pots. Count them yourself first. Then upload the photo and send:

Describe this photo in two sentences. Then tell me exactly how many [chairs] are in it, and how sure you are.

Notice how good the description is compared with the count, and whether its stated confidence matched how right it was.

2. Left or right test. In the same chat, ask:

What object is closest to the left edge of this photo, and what is closest to the right edge?

3. Interrupt test. If your app has a voice mode, ask it to explain how a swamp cooler works, then start talking halfway through its answer. Does it stop right away, or finish its sentence first? Try sounding annoyed and see if it notices. You are watching the pipeline versus native difference from the explainer, live.

Next lesson5. Spotting fakes and AI scams

Last updated October 4, 2026