Overview

Model serving overview

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark (✓) indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

FeatureSub-featureKFServingSeldon Core
FrameworkTensorFlow✓ sample✓ docs
XGBoost✓ sample✓ docs
scikit-learn✓ sample✓ docs
NVIDIA Triton Inference Server✓ sample✓ docs
ONNX✓ sample
PyTorch✓ sample✓
GraphTransformers✓ sample✓ docs
CombinersRoadmap✓ sample
Routers including MABRoadmap✓ docs
AnalyticsExplanations✓ sample✓ docs
ScalingKnative✓ sample
GPU AutoScaling✓ sample
HPA✓✓ docs
CustomContainer✓ sample✓ docs
Language Wrappers✓ Python, Java, R
Multi-Container✓ docs
RolloutCanary✓ sample✓ docs
Shadow✓
Istio✓✓

Notes:

  • KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
  • A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
  • KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow, Pytorch, ONNX and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale. Triton Inference Server was previously known as TensorRT Inference Server.

You can use NVIDIA Triton Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA Triton Inference Server.

BentoML

BentoML is an open-source platform for high-performance ML model serving. It makes building production API endpoint for your ML model easy and supports all major machine learning training frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn and etc.

BentoML comes with a high-performance API model server with adaptive micro-batching support, which achieves the advantage of batch processing in online serving. It also provides model management and model deployment functionality, giving ML teams an end-to-end model serving workflow, with DevOps best practices baked in.

Last modified 04.08.2020: OWNERS update for Serving Docs (#2079) (4e66d033)