A tale of two flows: Metaflow and Kubeflow

Metaflow is a Python framework for building and operating ML and AI projects, originally developed and open-sourced by Netflix in 2019. In many ways, Kubeflow and Metaflow are cousins: closely related in spirit, but designed with distinct goals and priorities.

Metaflow emerged from Netflix’s need to empower data scientists and ML/AI developers with developer-friendly, Python-native tooling, so that they could easily iterate quickly on ideas, compare modeling approaches, and ship the best solutions to production without heavy engineering or DevOps involvement. On the infrastructure side, Metaflow started with AWS-native services like AWS Batch and Step Functions, later expanding to provide first-class support for the Kubernetes ecosystem and other hyperscaler clouds.

In contrast, Kubeflow began as a set of Kubernetes operators for distributed TensorFlow and Jupyter Notebook management. Over time, it has evolved into a comprehensive Cloud Native AI ecosystem, offering a broad set of tools out of the box. These include Trainer, Katib, Spark Operator for orchestrating distributed AI workloads, Workspaces for interactive development environments, Hub for AI catalog and artifacts management, KServe for model serving, and Pipelines to deploy end-to-end ML workflows and stitching Kubeflow components together.

Over the years, Metaflow has delighted end users with its intuitive APIs, while Kubeflow has delivered tons of value to infrastructure teams through its robust platform components. This complementary nature of the tools motivated us to build a bridge between the two: you can now author projects in Metaflow and deploy them as Kubeflow Pipelines, side by side with your existing Kubeflow workloads.

Why Metaflow → Kubeflow

In the most recent CNCF Technology Radar survey from October 2025, Metaflow got the highest positive scores in the “likelihood to recommend” and “usefulness” categories, reflecting its success in providing a set of stable, productivity-boosting APIs for ML/AI developers.

Metaflow spans the entire development lifecycle—from early experimentation to production deployment and ongoing operations. To give you an idea, the core features below illustrate the breadth of its API surface, grouped by project stage:

Development

Straightforward APIs for creating and composing workflows.
Automated state transfer and management through artifacts, allowing you to build flows incrementally and resume them freely (see a recent article by Netflix about the topic)
Interactive, real-time visual outputs from tasks through cards - a perfect substrate for custom observability solutions, created quickly with AI copilots.
Choose the right balance between code and configuration through built-in configuration management.
Create domain-specific abstractions and project-level policies through custom decorators.

Scaling

Scale flows horizontally and vertically: Both task and data parallelism are supported.
Handle failures gracefully.
Package dependencies automatically with support for Conda, PyPI, and uv.
Leverage distributed computing paradigms such as Ray, MPI, and Torch Distributed.
Checkpoint long-running tasks and manage checkpoints consistently.

Deployment

Maintain a clear separation between experimentation, production, and individual developers through namespaces.
Adopt CI/CD and GitOps best practices through branching.
Compose large, reactive systems through isolated sub-flows with event triggering.

These features provide a unified, user-facing API for the capabilities required by real-world ML and AI systems. Behind the scenes, Metaflow is built on integrations with production-quality infrastructure, effectively acting as a user-interface layer over platforms like Kubernetes - and now, Kubeflow. The diagram below illustrates the division of responsibilities: kubeflow-metaflow-arch

The key benefit of the Metaflow–Kubeflow integration is that it allows organizations to keep their existing Kubernetes and Kubeflow infrastructure intact, while upgrading the developer experience with higher-level abstractions and additional functionality, provided by Metaflow.

Currently, the integration supports deploying Metaflow flows as Kubeflow Pipelines. Once you have Metaflow tasks running on Kubernetes, you can access other components such as Katib and Trainer from Metaflow tasks through their Python clients as usual.

Metaflow → Kubeflow in practice

As the integration requires no changes in your existing Kubeflow infrastructure, it is straightforward to get started. You can deploy Metaflow in an existing cloud account (GCP, Azure, or AWS) or you can install the dev stack on your laptop with a single command.

Once you have Metaflow and Kubeflow running independently, you can install the extension providing the integration (you can follow instructions in the documentation):

pip install metaflow-kubeflow

The only configuration needed is to point Metaflow at your Kubeflow Pipelines service, either by adding the following line in the Metaflow config or by setting it as an environment variable:

METAFLOW_KUBEFLOW_PIPELINES_URL = "http://my-kubeflow"

After this, you can author a Metaflow flow as usual and test it locally:

python flow.py run

which runs the flow quickly as local processes. If everything looks good, you can deploy the flow as a Kubeflow pipeline:

python flow.py kubeflow-pipelines create

This will package all the source code and dependencies of the flow automatically, compile the Metaflow flow into a Kubeflow Pipelines YAML and deploy it to Kubeflow, which you can see alongside your existing pipelines in the Kubeflow UI. The following screencast shows the process in action:

The integration doesn’t have 100% feature coverage yet: Some Metaflow features such as conditional and recursive steps are not yet supported. In future versions, we may also provide additional convenience APIs for other Kubeflow components, such as KServe - or you can easily implement them by yourself as custom decorators with the Kubeflow SDK!

If you want to learn more about the integration, you can watch an announcement webinar on Youtube.

Feedback welcome!

Like Kubeflow, Metaflow is an open-source project actively developed by multiple organizations — including Netflix, which maintains a dedicated team working on Metaflow, and Outerbounds, which provides a managed Metaflow platform deployed in customers’ own cloud environments.

The Metaflow community convenes at the Metaflow Slack. We welcome you to join, ask questions, and give feedback about the Kubeflow integration, and share your wishlist items for the roadmap. We are looking forward to a fruitful collaboration between the two communities!