One Long Chat vs a Fresh Chat: When to Start Over With AI

Should I start a new chat or keep the same one?

Start a fresh chat when the thread has changed jobs. Keep the current one while every message in it is still doing work.

That is the whole rule, and it is easier to apply than it sounds. A chat thread is a stack of text that gets resent to the model on every turn. While that stack is all relevant, it is an asset: the model holds your constraints, your tone, the file you uploaded. The moment part of it stops being relevant, it becomes noise the model still has to read before it answers you.

The cost of guessing wrong is asymmetric. Starting over too early costs you one paste of context. Staying too long costs you answers that quietly degrade without ever announcing it.

Why do long AI chats get worse?

Because important details get buried, not because the model gets tired. Published research on long contexts found that models retrieve information best when it sits near the beginning or the end of the input, and measurably worse when it sits in the middle, including models built specifically for long context (Liu et al., TACL 2024). A thirty message thread puts the constraint you stated at message four squarely in that middle.

Two other things pile up:

• Rejected drafts stay in the room. Every version you told the model not to write is still sitting in the context, still shaping the next attempt.
• Corrections turn into a tangle. "Shorter" plus "more formal" plus "put the numbers back" reads, by turn twenty, as a set of instructions that fight each other.

And there is a hard ceiling. Once a thread outgrows the model's context window, something has to give: the interface either drops the oldest messages or the request fails outright. If it silently drops them, the model is now answering without the setup you assume it still has.

What does a long chat actually cost?

Every turn pays for the whole thread again. Chat is stateless underneath: the entire conversation is resent as input tokens with each new message, so a thread that has grown to 20,000 tokens charges you 20,000 input tokens even for a one line follow up. Prompt caching, offered by the major providers, cuts the price of resending an unchanged prefix, but it does not make a bloated thread free and it does nothing for the accuracy problem above.

You do not have to take this on faith. In aiDex, the cost of each message is visible as it happens, so you can watch a thread get more expensive turn by turn and decide when it stopped being worth it. Use your own provider keys or the ones we manage, and pick the models you want.

When is one long chat the right call?

Keep the thread when the accumulated context is the whole point. Four cases where starting over is the mistake:

1. You are iterating on one artifact. A contract clause, a function, a paragraph. Version eight only makes sense against versions one to seven.
2. You uploaded a document. Re-uploading is friction, and in a panel chat the file is already shared with every model in the room. This is the same logic behind reviewing a long document with more than one model.
3. You are debugging something cumulative. Each failed hypothesis narrows the search. Throwing that away means repeating it.
4. You have built shared vocabulary. If the model now correctly uses your internal names for things, that is real context worth carrying.

How do I carry context into a new chat without the clutter?

Ask the model for a handoff before you leave. Paste this at the end of the old thread:

Summarize this conversation as a brief for a fresh assistant: the goal, the constraints we agreed on, the decisions already made, what we ruled out and why, and the current state of the draft. Under 200 words. No preamble.

Then open a new chat and paste the result as the first message. You keep the decisions and drop the rejected drafts, which is exactly the trade you wanted. If the answers in the old thread had gone strange rather than merely long, it is worth debugging the prompt itself before you carry anything forward.

Does this change when several models are in the room?

Yes, and it gets stricter. In a panel chat every model reads the same thread, so a bloated conversation multiplies across the panel instead of costing you once. The same middle of the context problem now applies to four models at once, and they will disagree for reasons that have nothing to do with your question.

This is where mode choice does more work than chat hygiene. In aiDex, Compare sends one question to several models at once, Judge picks a winner among their answers, Team lets them talk to each other, and Pipeline hands the output of one model to the next in stages. Pipeline is often the real answer to "my chat is getting long": instead of one thread dragging every stage behind it, each stage gets a clean handoff, which is the point of building a pipeline rather than a monologue. If you are unsure which fits, start with when to use each mode, and browse the Dex to see which models you can seat at the table.

The broader habit sits inside multi-model AI workflows: treat a conversation as a working file, not a diary. Files get closed when the job changes.