Kubeflow vs BentoML vs Seldon

Once you’ve trained a model and want to serve it, the Kubernetes-native serving landscape has three credible options: Kubeflow (specifically KServe, the serving piece), BentoML, and Seldon Core. All three deploy models behind HTTP/gRPC endpoints on Kubernetes. The choice is less about features than about how your team thinks about ML deployment and what’s already in your stack.

We’ve shipped all three across hospital ML (clinical NLP, image classification) and banking (fraud detection, customer scoring). Here’s how we actually decide.

The thirty-second framing#

Kubeflow / KServe is the CNCF-friendly, Kubernetes-native model serving system. Defines a InferenceService CRD; runs models with autoscaling, traffic splitting, and standardized predict/explain/health endpoints. Designed to be part of the broader Kubeflow MLOps platform.
BentoML is “package your model and serving code as a Bento; deploy the Bento anywhere.” Strong Python developer experience, framework-agnostic, deploys to Kubernetes, AWS Lambda, ECS, or anywhere you can run a container.
Seldon Core (v2) is enterprise-grade model serving on Kubernetes. Mature, supports complex inference graphs (A/B routing, shadow models, ensemble), excellent observability, commercial Seldon Deploy product for advanced workflows.

All three solve “serve my model with autoscaling and good observability.” They differ in: developer experience, ecosystem fit, and how they handle complex deployment patterns.

What’s actually different#

Dimension	KServe	BentoML	Seldon Core v2
License	Apache 2.0	Apache 2.0	OSS BSL (Seldon Core), commercial Deploy
Primary abstraction	InferenceService CRD	Bento (model + service code)	Model + Pipeline CRDs
Framework support	sklearn, PyTorch, TF, XGBoost, ONNX, custom	Any Python framework	Most common frameworks + custom
Custom serving code	Yes via custom transformers	Yes (first-class)	Yes
Multi-model serving	Yes (ModelMesh)	Yes (Runners)	Yes
A/B / canary routing	Built-in via Knative	DIY (or use Istio)	First-class
Inference graphs / ensemble	Limited	DIY	First-class (pipelines)
Autoscaling	Knative scale-to-zero	HPA / KEDA	HPA / KEDA
Explainability	Built-in (Alibi integration)	DIY	Built-in (Alibi)
Adversarial robustness	DIY	DIY	Built-in (Alibi Detect)
Local dev story	Heavy (needs cluster)	Excellent (`bentoml serve` locally)	Heavy (needs cluster)
Deploy outside K8s	No	Yes (Lambda, ECS, anywhere)	No (Kubernetes-only)
Community / momentum	Strong (CNCF)	Strong, growing	Mature, smaller

Where KServe wins#

Native Kubernetes shape. If you’re all-in on Kubernetes and want serving to feel like a regular K8s resource, KServe’s InferenceService CRD is exactly that. kubectl apply your model deployment.

Knative scale-to-zero. For models with sporadic traffic, scaling pods to zero between requests is real cost savings. KServe’s Knative backbone handles this natively.

ModelMesh for many small models. Serving hundreds of small models on a shared pool (instead of one pod per model) — KServe’s ModelMesh handles this well. Memory-efficient for “one model per tenant” patterns.

Integration with the Kubeflow MLOps platform. If you’re using Kubeflow Pipelines for training, KServe is the natural serving counterpart. Same project, shared concepts.

Standardized inference protocols. KServe defines v1 and v2 inference protocols (OpenAPI-style and gRPC). Models expose consistent endpoints regardless of framework.

Where KServe hurts:

Heavy operational surface — Knative, Istio (recommended), KServe controllers, ModelMesh controller. Real cluster ops cost.
Local dev requires cluster (or KIND/k3d). No equivalent of bentoml serve.
The custom transformer / predictor / explainer split is powerful but requires structural thinking upfront.

Where BentoML wins#

Developer experience. Write a service.py that loads your model and defines endpoints. Test locally with bentoml serve. Package with bentoml build. Deploy with bentoml deploy. The whole loop feels like normal Python development.

Framework neutral. PyTorch, TensorFlow, sklearn, XGBoost, Hugging Face transformers, custom — all first-class. Save your model with bentoml.<framework>.save_model(), load in your service.

Deploys anywhere. A Bento builds to a container. That container deploys to Kubernetes, but also to AWS Lambda (BentoML supports it), ECS, Cloud Run, or anywhere. Not locked to Kubernetes.

Custom pre/post-processing is natural. Tokenization, image resizing, feature engineering — all written as normal Python in the service.py. No separate transformer pod.

Yatai (BentoML’s K8s operator) for cluster deployments. Get the K8s-native deployment, autoscaling, traffic splitting story when you want it.

Where BentoML hurts:

For “deploy 200 small models on shared pool” workloads, ModelMesh (KServe) is more memory-efficient.
A/B and canary routing aren’t first-class — you DIY via Istio/service mesh.
The ecosystem is smaller than KServe’s.

Where Seldon Core wins#

Inference graphs. Seldon’s pipelines support multi-step inference: model → post-processor → outlier detector → router → ensemble. For complex serving scenarios (fraud scoring with multiple ML models + business rule layer), Seldon’s graph model fits naturally.

Mature explainability and monitoring. Built-in integration with Alibi (explainability) and Alibi Detect (outlier and drift detection). For regulated industries (healthcare, finance) where “why did the model predict this?” is a real requirement, this is meaningful.

A/B and canary as first-class. Define traffic-splitting in the resource itself — no Istio gymnastics.

Production maturity. Seldon has been deployed at financial institutions and regulated industries for years. The patterns are well-understood and the docs reflect that.

Where Seldon hurts:

Operational surface is similar to KServe — real cluster infra to operate.
Local dev story isn’t great.
Smaller community than KServe.
License change: Seldon Core v2 is now under BSL (Business Source License), not pure Apache 2.0. Doesn’t matter for most users; matters for some.

When we pick what#

Pick KServe if:

Your team is Kubernetes-native and wants serving to be K8s-shaped
You’re already running Kubeflow Pipelines
You need scale-to-zero (sporadic traffic patterns)
You’re serving many small models (ModelMesh pattern)
You want CNCF-graduated tooling for compliance/governance reasons

Pick BentoML if:

Your team is Python-engineer-led (not platform-engineer-led)
Local development experience matters
You want to deploy outside Kubernetes (Lambda, Cloud Run, etc.) at least sometimes
Your service has meaningful custom pre/post-processing logic
You’re a small team that doesn’t want to operate Knative + Istio

Pick Seldon if:

Your serving topology involves complex graphs (multi-step inference, ensembles, routing)
You need built-in explainability and drift detection (regulated industries)
You want enterprise support (commercial Seldon Deploy)
A/B and canary routing are core to your release strategy

A pragmatic alternative: just FastAPI#

For 80% of model serving needs we encounter in practice, the right answer is none of the above. It’s:

FastAPI with the model loaded into memory at startup
Containerized with the right Python deps
Deployed on Kubernetes as a regular Deployment with HPA
Behind your existing API gateway or load balancer
Observability via Prometheus + Grafana + the same logging stack as your other services

This stack is more code than KServe/BentoML, but it’s normal code. Operations teams already know it. New engineers onboard in a day. The “serving framework” doesn’t become its own learning curve.

We reach for KServe/BentoML/Seldon when one of these is true:

Auto-scaling complexity is real (scale-to-zero, multi-model serving)
The team wants standardized inference protocols across many models
The serving graph is genuinely complex (A/B routing + ensemble + post-processors)

If your needs are “one model, predictable traffic, normal autoscaling” — write FastAPI.

What we deploy by default#

For client work:

FastAPI for the bulk of model serving. Real production deployments on hospital and banking ML serve traffic via FastAPI + Uvicorn on Kubernetes.
BentoML when the team is Python-engineer-led and wants the developer ergonomics, or when the model packaging story matters (sharing models across teams).
KServe when the org is committed to Kubeflow and wants serving to fit the broader Kubeflow shape.
Seldon Core when the inference topology is genuinely complex (multi-model ensembles, explainability requirements) — common in regulated industries.

For most projects: start with FastAPI. Move to a serving framework when you hit a wall the framework specifically solves.

The thing none of them solve#

All three deploy a model. None of them solve:

Feature consistency between training and serving (your feature store does)
Model monitoring beyond response logging (drift detection, performance over time)
Cost attribution per prediction
Deciding when to retrain

These are the discipline parts of ML operations. The serving framework is the substrate; the discipline is your job. See our MLflow vs W&B piece for the experiment-tracking discipline and our LLM observability piece for the runtime observability discipline (applies to non-LLM models too).

The pattern of patterns#

Model serving frameworks are over-recommended. They’re great when you need them and overhead when you don’t. The question “which serving framework should I use” should be preceded by “do I need one at all.”

The teams that ship ML reliably aren’t the ones using the most sophisticated serving stack. They’re the ones who picked the simplest tool that fit and spent the saved time on data quality, monitoring, and the boring parts of MLOps.

The right serving stack is the simplest one that fits your workload. If you’re building production ML and want a sanity check on the serving choice, our ML & MLOps team has shipped all four (including plain FastAPI). Tell us about the workload.