SageMaker AI Is Not the Right Way to Use LLMs

Watch on YouTube ↗

SageMaker is good infrastructure. It handles training job orchestration, experiment tracking, model versioning, and serving for traditional ML workflows. The problem is that "traditional ML" and "LLM workflows" are not the same thing, and teams that reach for SageMaker as their default AI platform for LLM deployments are making their lives significantly harder than they need to be.

This is not a criticism of SageMaker. It's a note on fit.

What SageMaker Was Built For

SageMaker's design reflects the ML workflows of 2017–2021: feature engineering → model training → batch inference or real-time serving via a persistent endpoint. The model is a bounded object with a defined input schema and output schema. You upload it, deploy it, and call it.

The deployment model is a managed container that runs your model server. You define the instance type, the scaling policy, and the request routing. SageMaker handles the infrastructure. This is genuinely useful for hosting a tabular classifier, a recommendation model, or a custom deep learning model with predictable input/output behavior.

LLM workloads break this model in several ways.

The Mismatch

Token streaming. Modern LLM APIs return tokens as they're generated — the client sees each token as it arrives, enabling responsive UX for chat-style applications. SageMaker's real-time inference endpoints support streaming, but configuring it correctly is significantly more involved than using a purpose-built LLM serving stack like vLLM, TGI, or a managed API. The default path is blocking responses.

Continuous batching. Efficient LLM serving relies on continuous batching: dynamically grouping in-flight requests to maximize GPU utilization. TGI and vLLM implement this natively. SageMaker's managed serving infrastructure doesn't expose the knobs that efficient LLM batching requires — you end up with either over-provisioned idle capacity or under-provisioned queuing.

KV cache management. The key-value cache is what makes sequential token generation efficient. LLM serving stacks manage it carefully — evicting, reusing, and quantizing it based on available memory. This is transparent in vLLM or TGI. On SageMaker, you're managing this inside your container, which means you're effectively building a mini serving stack on top of managed infrastructure.

Model weight distribution. For large models (70B+), you need tensor parallelism across multiple GPUs. SageMaker supports multi-GPU instances and has added some tensor parallel support, but it lags behind purpose-built stacks in ergonomics for this use case. Configuring efficient multi-GPU LLM serving on SageMaker is more work than doing the same on a raw instance or via a hosted endpoint.

Context length and memory. LLM memory requirements are dynamic — they depend on the current sequence length, which varies per request. SageMaker's instance configuration assumes relatively stable memory requirements. Dynamic memory pressure from variable context lengths adds complexity to capacity planning.

The Practical Cost

The mismatch isn't a dealbreaker for every use case. If you're doing offline batch processing — running a large document corpus through an LLM and storing the results — SageMaker's batch transform works fine. The streaming and batching issues don't matter for offline workloads.

The cost shows up in real-time applications: interactive chat, document Q&A, real-time classification with LLMs. Here, the friction compounds:

• Configuring streaming requires custom inference container code
• Getting efficient GPU utilization requires working around the serving stack instead of with it
• Debugging latency and throughput issues requires understanding both the SageMaker layer and the serving behavior, and they interact

Teams that go down this path typically spend several weeks arriving at a configuration that a purpose-built stack would have given them in hours.

What to Use Instead

The decision tree:

If you need a managed hosted API (no infrastructure ownership): Use the Bedrock API for Anthropic/Amazon/Meta models. Use the Anthropic API directly for Claude. Use the OpenAI API for GPT models. These are purpose-built for LLM API delivery, handle all the serving complexity, and price on tokens-consumed rather than instance-hours. For most production LLM applications, managed APIs are the right answer — the infrastructure cost is irrelevant compared to the model cost.

If you need to self-host for data sovereignty or cost reasons: Deploy vLLM or TGI on a raw EC2 instance with a GPU, or on EKS. These stacks are purpose-built for LLM serving — continuous batching, streaming, KV cache management, and tensor parallelism are first-class features. The operational overhead of managing a raw instance is lower than the operational overhead of fighting SageMaker's abstractions for a workload they weren't designed for.

If you need batch offline processing: SageMaker Batch Transform is appropriate. The streaming and batching issues don't apply.

If you have existing SageMaker workflows and need to add LLM capabilities: Use SageMaker to call an external LLM API, or deploy a vLLM container on SageMaker using a custom container. Don't try to use the native SageMaker serving stack for online LLM inference.

The Data Sovereignty Caveat

The most legitimate reason to use SageMaker for LLMs: your data cannot leave AWS, and you need to run a specific open-source model. In this case, self-hosting on SageMaker (with a custom vLLM container) is a reasonable path. The data plane stays within your VPC; SageMaker handles the job and endpoint lifecycle.

Even in this case, the recommendation is to run vLLM inside the SageMaker-managed container infrastructure, not to use SageMaker's built-in model server for the LLM. You're using SageMaker for the AWS integration, not the serving stack.

The Underlying Point

SageMaker is good infrastructure for the problems it was designed to solve. LLMs are a new category of model with serving requirements that differ from traditional ML in ways that matter for production deployments. Matching the tool to the problem — rather than defaulting to the most familiar tool — is where the leverage is.

The LLM serving ecosystem (vLLM, TGI, managed APIs) has been built specifically for these requirements. Using it doesn't mean abandoning AWS or your existing ML infrastructure. It means using the right layer for each component of your system.