"Design ChatGPT" is a backend question wearing a model's name. The vLLM paper by Woosuk Kwon and colleagues reports 2-4 times the throughput of state-of-the-art serving systems such as FasterTransformer and Orca at the same latency, mostly by managing the model's key-value cache memory better. That result is a good summary of where this interview is actually scored. The model is a box you are allowed to treat as given. The system around it is the answer.
It is tempting to spend the round on transformers, training data and reinforcement learning from human feedback. In our view, a panel testing a backend or infrastructure engineer usually cares about something else. They want to know how a token gets from a GPU to a browser, what happens when a thousand people hit send at once, and where the conversation history lives. This guide covers that system, section by section, so you can say it out loud in a typical 45-minute round.
What should you clarify before designing ChatGPT?
Clarify which product you are building, because "ChatGPT" is at least three systems. A consumer chat app with accounts and saved conversations is one. A developer API that streams completions is another. An internal playground for comparing prompts is a third. Ask which one, and say the answer back before you draw anything.
Then pin down the requirements that change the architecture. Are replies streamed token by token, or returned whole? Do conversations persist across devices? Is there a free tier that needs rate limiting? Are file uploads or images in scope? Each of these adds a component, and the panel wants to see you choose scope deliberately rather than draw everything.
State one honest assumption out loud. You are designing a reference system from public documentation and published research, and you do not know how OpenAI runs its production stack. Separating what is published from what you are guessing shows judgment, and it protects you if the interviewer knows the real system.
Finish the section with two scale numbers you will design against, such as concurrent users at peak and average reply length. State them so every later decision can point back at them. Precision is optional.
How do you stream replies to the browser?
Stream over server-sent events, and say why. OpenAI's streaming guide focuses on HTTP streaming over server-sent events (SSE), and Anthropic's Messages API documentation says you set a stream flag to incrementally stream the response using server-sent events (SSE). OpenAI also offers a WebSocket mode for persistent connections with incremental inputs, which is exactly the trade-off worth naming. In our view, SSE is a reasonable default for a chat reply, and a persistent two-way channel earns its cost only when the client sends a lot mid-reply.
The reasoning is about direction. A chat reply is one request followed by a long, one-way flow of tokens. SSE is plain HTTP, so it fits the request and response infrastructure you already have. The browser's EventSource API reconnects automatically, but it only makes GET requests, and a chat send carries the prompt in a POST body. A POST stream read with fetch needs its own resume logic, which is worth saying before the interviewer finds it.
There is one detail worth raising unprompted. MDN's guide to server-sent events warns that when SSE is not used over HTTP/2, the browser limits open connections per browser and domain to a very low number (6), which bites users with several tabs open. Over HTTP/2, the number of simultaneous streams is negotiated and defaults to 100. Serve the stream over HTTP/2 and that limit stops being your problem.
Mention the stream's shape too. Anthropic's documentation describes a stream as a message start event, one or more content blocks each made of start, delta and stop events, one or more message delta events, and a final stop event, with ping events interspersed. Designing the client around that structure tells you exactly what was delivered when a stream drops.
Why does GPU batching shape the whole design?
Batching shapes the design because GPU time is the scarcest and most expensive resource in the system, and a GPU serving one request at a time wastes most of it. Every other component exists to keep the accelerators busy without making users wait.
The naive approach batches whole requests together. The Orca paper, presented at OSDI 2022, explains why that fails for text generation. Replies finish at different lengths, so requests that finish early cannot return to the client, and newly arrived requests have to wait until the current batch completely finishes. Orca's answer was iteration-level scheduling, which schedules work one decoding step at a time so requests can join and leave the batch between steps. Many serving systems call the same idea continuous batching.
Memory is the second constraint. Each in-flight request holds a key-value (KV) cache that grows with every token. The vLLM paper found that this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. Its PagedAttention technique borrows virtual-memory paging from operating systems, and vLLM, built on it, achieves near-zero waste in KV cache memory. That is largely how vLLM reached 2-4 times the throughput of FasterTransformer and Orca.
In the interview, give the chain of reasoning and leave the papers' math on the page. Throughput depends on batch size, batch size depends on KV cache memory, and so memory management is a product decision.
How do you make generation feel faster?
Separate the two latencies users feel. Time to first token is how long the screen stays blank. Time per output token is how fast the text appears once it starts. Streaming already hides most of the total, so a panel will usually push on the first number.
Generation is slow for a structural reason. The speculative decoding paper by Yaniv Leviathan, Matan Kalman and Yossi Matias notes that decoding K tokens takes K serial runs of the model. Their technique has a smaller approximation model propose several tokens that the large model then checks in parallel, and the paper reports a 2X-3X acceleration on T5-XXL compared to the standard T5X implementation, with identical outputs. It is worth naming as an option once your baseline design is on the board.
In a design like this one, time to first token is mostly queueing and prompt processing. Long conversations make it worse, because the whole history is processed before the first new token appears. Trimming what goes into the prompt is the most direct lever on that wait.
Where does the conversation history live?
Store conversations in a database keyed by conversation, with messages appended in order, and treat the model's context window as a separate, smaller view of that history. The database is the record. The prompt is a selection from it.
That separation lets you answer the hard follow-up cleanly. When a conversation outgrows the context window, something has to give. You can keep the most recent turns, summarize older ones into a running note, or retrieve the older turns most relevant to the new message. Each trades cost for fidelity, and naming which one fits the product earns more credit than naming all three.
Say where the prompt gets assembled. A context service reads the history, applies the truncation rule, adds any system instructions, and hands a finished prompt to the inference tier. Keeping that logic out of the API servers makes it testable and lets you change the rule without touching the GPU fleet.
What happens when demand outruns GPU capacity?
Overload is normal for this system, so design for it rather than around it. Anthropic's streaming documentation notes that during periods of high usage a client might receive an overloaded error mid-stream, which would correspond to an HTTP 529 outside a stream. Your design needs a path for the same moment.
Put admission control in front of the inference tier. Rate-limit by user and tier at the gateway, queue requests with a timeout, and shed load from the free tier before the paid one. When a request cannot be served, fail it quickly with a clear message rather than letting it hang. A user who sees "try again in a minute" is better served than one staring at a frozen cursor.
Add safety checks as pipeline stages with a latency cost you acknowledge. Input moderation runs before inference. Output checks can run on the stream as it is produced. Mention that each stage spends part of your time-to-first-token budget.
What is overrated
Explaining the transformer is overrated. Unless the role is in model research, a panel that asked you to design ChatGPT rarely wants attention heads drawn on the whiteboard, and every minute spent there is a minute not spent on batching or overload.
WebSockets by reflex are overrated too. They work, but a candidate who reaches for them without weighing SSE has skipped the one transport decision this prompt hands them.
So is an exact GPU count. Back-of-envelope sizing is useful when it changes a decision. A precise number of accelerators, built on assumptions about model size you cannot check, is decoration.
A six-step playbook for the round
- Pick the product. Consumer app, API or playground, then two scale numbers you will design against.
- Draw the request path. Gateway, API servers, context service, inference tier, conversation store.
- Choose the transport. SSE over HTTP/2, with recovery logic for a dropped POST stream.
- Explain the batching chain. Continuous batching, KV cache memory, and why memory sets throughput.
- Handle long conversations. Name one truncation or retrieval rule and its cost.
- Plan for overload. Rate limits, queues with timeouts, fast failures and a moderation stage.
Knowing these steps and delivering them under a clock are different skills. Four-Leaf's AI mock interviews include a system design round for software roles that you answer by voice, and in its question-by-question mode the feedback on each answer checks whether you clarified scope, called out trade-offs explicitly, and built a design that handles the scale you stated. For the latency vocabulary this round leans on, see how to discuss latency in a system design interview, and for the classic prompts beside it, the system design interview questions guide.
Where this is heading
"Design ChatGPT" will keep appearing because it compresses a modern backend into one prompt. Streaming, scheduling scarce hardware, managing state and failing gracefully all show up in a typical 45-minute round. The candidates who do well treat the model as a slow, expensive, stateless function and spend their time on everything around it. That is where the engineering is, and the panel knows it.
