Getting a model to a good validation score is the easy part. The hard part starts when somebody else needs to call it: a mobile app, a nightly batch job, another service on the other side of the building. That is when you find out that a notebook with a hard-coded checkpoint path and a cached tensor is not a product. What follows is the path from that notebook to a small containerized REST API — and the mistakes that turn a demo endpoint into a 3 a.m. page.

What production really asks of a model server

In a notebook, your model runs once over a batch you control. In production it runs thousands of times while other people depend on the answer. Three things change at once: the process has to stay up, it has to serve concurrent requests, and it has to be replaceable without downtime. Every decision below serves one of those three requirements.

Before you write a Dockerfile, settle the contract. What does a request look like — one image, or a JSON object with fields and a size limit? What comes back? Callers will build against that contract, so choose it deliberately and version it from the first day.

Wrap the model in a FastAPI service

FastAPI fits this job well: you get request validation through Pydantic, a production-grade server, and very little boilerplate. Keep the service in a small package — an app file, a schemas file for request and response models, and a short module that knows how to load and run the model. Nothing in the serving code should know how training worked.

Load the model once, at startup

Reloading weights on every request is the most common beginner mistake, and it is brutally expensive: a checkpoint that takes eight seconds to load turns a 60 ms prediction into an 8-second response. Use FastAPI's lifespan handler — the current replacement for the deprecated startup event — to load the model into memory when the process boots, and stash it on app.state. The handler then does preprocessing, one forward pass, and postprocessing. That is all.

Call model.eval() and wrap inference in torch.inference_mode() (or no_grad()). Skipping eval() leaves dropout and batch norm in training mode, so identical inputs return slightly different answers on each call. That bug is nearly invisible when you test with a single request and very visible to the user who sends the same thing twice.

Validate inputs and serialize outputs

Define a Pydantic model for the body — a string with a maximum length, a list with a maximum element count — and return a plain dictionary. Do not return a tensor or a numpy array: FastAPI cannot serialize them, and the failure surfaces as an opaque 500 rather than a useful message. Convert with .tolist() and cast to native Python types. Reject oversized payloads explicitly, because a request carrying 200 MB of data will consume a worker's memory long before your model ever sees it.

Keep CPU work off the event loop

If your handler is declared as async def but performs a blocking forward pass, it blocks every other request on that worker. Two ways out: declare the handler with plain def so FastAPI runs it in a thread pool, or keep it async and run several worker processes. For most computer vision and NLP models, one worker process per container plus more replicas is the least surprising configuration — memory scales linearly and a crash takes down one replica, not the fleet.

Build an image you won't regret pulling

Start from a slim Python base such as python:3.11-slim, not a GPU base image, unless you genuinely run inference on a GPU.

Pin versions and install CPU-only wheels

Pin every dependency in your requirements file and pin the base image by tag with a patch version, ideally by digest, so a rebuild six months from now produces the same thing. The default framework install pulls CUDA libraries that add gigabytes to your image and do nothing on a CPU host; install from the CPU-only package index instead. Your image gets smaller, pulls faster, and cold-starts sooner — which matters more than you think on a fresh node.

Order layers and trim the build context

Copy the requirements file first, install dependencies, and only then copy the application code. Docker caches layers in order, so editing a handler no longer triggers a full dependency reinstall. A .dockerignore that excludes notebooks, training checkpoints you don't serve, test fixtures and .git can shrink a build context by hundreds of megabytes.

Next, decide where the weights live. Baking a checkpoint into the image makes the artifact immutable and rollback trivial, at the cost of a bigger pull on every new node. Downloading at startup keeps the image lean but adds an external dependency and a checksum check you must not skip. With a handful of nodes, baking weights in is usually the simpler answer. Either way, run the container as a non-root user and set an explicit working directory.

The pitfalls that appear only under load

• Memory math. Each worker holds a full copy of the model. Four workers on a 1.5 GB checkpoint means 6 GB before Python overhead. Out-of-memory kills look like random restarts, not like errors in your logs.
• Cold start. A container that needs 45 seconds to load weights will fail a liveness probe expecting an answer in five. Give startup its own grace period and treat readiness separately from liveness.
• Timeouts. Set a request timeout at the reverse proxy and in the client, longer than your 99th-percentile inference latency but shorter than the patience of whoever is calling you.
• One item per request. Simple and slow. If your model benefits from batches, queue requests and process them together — but only when the added latency is acceptable to the caller.
• Weak observability. Log the model version at startup and attach a request ID to every line. Without that, chasing a bad prediction in production is guesswork.
• Careless data handling. Never bake API keys into an image, and think twice before logging raw input, which may contain user data you are not allowed to retain.

Ship it: health checks, versioning, rollout

Expose two endpoints. A liveness route returns 200 as soon as the process is up; a readiness route returns 200 only once the model is loaded and warm. Keep inference work out of both — a health check that runs a forward pass will report your service as dead during a traffic spike.

Version the model artifact, the API contract and the image independently, and return the model version in the response payload or a header. Then deploy with a canary: send a small slice of traffic to the new container, compare latency and a sample of outputs against the old one, and only then shift the rest. Keep the previous image tag handy so rollback is a configuration change rather than a rebuild.

Before you announce anything, run a load test with realistic payloads. Watch memory and p99 latency as concurrency climbs; the point where latency bends sharply upward is your real capacity, and it is usually lower than the number you guessed.

A pre-flight checklist

1. Model loads once at startup, in eval mode, with inference mode enabled.
2. Request and response schemas validated, documented and versioned.
3. Dependencies pinned, CPU-only wheels installed, base image pinned.
4. Container runs as a non-root user with a trimmed build context.
5. Model version logged at startup and returned with every response.
6. Liveness and readiness probes separated, with a realistic startup grace period.
7. Timeouts configured at proxy and client; memory limit set above workers × model size.
8. Load-tested with real payloads, and the rollback path tried once on purpose.

Deployment is not the finish line of a modeling project. It is the point where the model starts receiving real feedback, and the service you build here doubles as a measuring device: once traffic flows through it, you can sample inputs, compare outputs against expectations, and find the cases your validation set never covered. Build it small, version it honestly, and keep rollback one command away.

Photo: Jan van der Wolf / Pexels