Deploying Kubeflow 1.3 in an enterprise with existing Kubernetes infrastructure, a Service Mesh (Istio), and Argo, presents a host of challenges. This blog will address how those challenges were overcome while retaining the best practices that both the organization and Kubeflow prescribe.

Lay of the land at Intuit

Intuit has invested heavily in building out a robust Kubernetes infrastructure that powers all of Intuit’s products: TurboTax, QuickBooks, and Mint. There are thousands of services that run on over a hundred Kubernetes clusters. Managing these clusters is the Intuit Kubernetes Service (IKS) control plane. The IKS control plane provides services such as namespace management, role management, and isolation, etc. Connecting the services is an advanced, Istio-based service mesh, which complements Intuit’s API Gateway. In combination, they provide robust authentication, authorization, rate limiting, and other routing capabilities.

The Intuit ML Platform is built on this ecosystem and provides model training, inference, and feature management capabilities, leveraging the best of Intuit’s Kubernetes infrastructure and AWS SageMaker. This is the backdrop against which we started exploring Kubeflow to provide advanced orchestration, experimentation, and other services.

Kubeflow and Istio

Our first challenge with running Kubeflow was the compatibility of Kubeflow’s Istio with Intuit’s existing Service Mesh built on top of Istio. Two key problems emerged: version compatibility and operational maintenance.

Kubeflow v1.3 defaults to Istio (v1.9), and luckily it is compatible with the older versions of Istio (v1.6), which is what Intuit runs on. Running two Istio versions is impractical, as that would defeat the benefit of a large, interconnected existing service mesh. Hence, we wanted Kubeflow to work seamlessly with Intuit’s service mesh running Istio v1.6.

If you are new to Istio, you might want a primer on these key Traffic Management Components and Security Components:

  1. VirtualService
  2. DestinationRule
  3. Gateway
  4. EnvoyFilter
  5. AuthorizationPolicy

Step 1: Remove default Istio configurations and Argo from Kubeflow

The first step to running Kubeflow was to remove the Istio and Argo bundled with Kubeflow so that it could be integrated with the Intuit service mesh.

To Remove Kubeflow’s default Istio

We have used Kustomize to build the manifest we need for our Kubeflow installation and we are using ArgoCD to deploy the Kubeflow Kubernetes manifests.

.
├── base                        # Base folder for the kubeflow out of the box manifests
│   ├── kustomization.yaml      
│   ├── pipelines               # Folder for Kubeflow Pipelines module
│   │   ├── kustomization.yaml
│   ├── other modules           # Similar to the Pipelines module you can bring other modules as well
│       ├── kustomization.yaml
├── envs                        # Folder for all the Kubeflow environments
│   ├── prod               
│   │   ├── kustomization.yaml
│   ├── dev               
│       ├── kustomization.yaml

base -> kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- github.com/kubeflow/manifests/common/kubeflow-roles/base?ref=v1.3.0
- github.com/kubeflow/manifests/common/kubeflow-namespace/base?ref=v1.3.0
- github.com/kubeflow/manifests/common/oidc-authservice/base?ref=v1.3.0
- github.com/kubeflow/manifests/apps/admission-webhook/upstream/overlays/cert-manager?ref=v1.3.0
- github.com/kubeflow/manifests/apps/profiles/upstream/overlays/kubeflow?ref=v1.3.0
- github.com/kubeflow/manifests/apps/centraldashboard/upstream/overlays/istio?ref=v1.3.0
- pipelines

base -> pipelines -> kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - github.com/kubeflow/pipelines/manifests/kustomize/base/installs/multi-user?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/base/metadata/base?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/base/metadata/options/istio?ref=1.5.0
  # To remove the default Argo from Pipelines module
  # - github.com/kubeflow/pipelines/manifests/kustomize/third-party/argo/installs/cluster?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/third-party/mysql/base?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/third-party/mysql/options/istio?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/third-party/minio/base?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/third-party/minio/options/istio?ref=1.5.0
  - github.com/kubeflow/pipelines/manifests/kustomize/third-party/metacontroller/base?ref=1.5.0

# Identifier for application manager to apply ownerReference.
# The ownerReference ensures the resources get garbage collected
# when application is deleted.
commonLabels:
  application-crd-id: kubeflow-pipelines

# !!! If you want to customize the namespace,
# please also update base/cache-deployer/cluster-scoped/cache-deployer-clusterrolebinding.yaml
namespace: kubeflow

Note: we had to create a separate folder for pipelines because we didn’t want to use Argo, which comes with the Pipelines module. If you can use default Argo, then you can simply use https://github.com/kubeflow/manifests/apps/pipeline/upstream/env/platform-agnostic-multi-user-pns?ref=v1.3.0 instead of the pipelines folder.

If you don’t want to use ArgoCD, you can build the manifest using the kustomize build command, which is essentially what ArgoCD does. The configuration above has been tested for Kustomize 3.8.x and 4.0.x, and it works with both.

Step 2: Kustomize the Kubeflow manifests

Given the managed Kubernetes ecosystem at Intuit, protocols for service to service communication and namespace isolation is opinionated, and we had to make the following changes:

  1. Enable Kubeflow namespace for Istio injection by adding the label istio-injection: enabled in the namespace specification. This label is then used by Istio to add the sidecar into the namespace.
  2. Enable sidecar injection to all the deployments and statefulsets in Kubeflow by adding the annotation sidecar.istio.io/inject: "true", along with some Intuit-specific custom labels and annotations to the Deployments and StatefulSets.
  3. Intuit’s security policies forbid the direct use of external container registries. Intuit’s internal container registry runs regular vulnerability scans and certifies Docker images for use in various environments. The internal container registry also has an allow list that enables external registries to be proxied and held to the same, high-security standards. We enabled it for all Kubeflow containers.
  4. Changes in VirtualService to route all the traffic from one central gateway instead of using Kubeflow gateway.

We have used Kustomize to modify the Kubeflow application manifest.

  1. For adding labels, we have used LabelTransformer 
         apiVersion: builtin
     kind: LabelTransformer
     metadata:
       name: deployment-labels
     labels:
       <Intuit custom labels>
       istio-injected: "true"
     fieldSpecs:
     - path: spec/template/metadata/labels
       kind: Deployment
       create: true
     - path: spec/template/metadata/labels
       kind: StatefulSet
       create: true

  2. For adding annotations, we have used AnnotationsTransformer

     apiVersion: builtin
 kind: AnnotationsTransformer
 metadata:
   name: deployment-annotations
 annotations:
   <Intuit custom annotations>
   sidecar.istio.io/inject: "true"
 fieldSpecs:
 - path: spec/template/metadata/annotations
   kind: Deployment
   create: true
 - path: spec/template/metadata/annotations
   kind: StatefulSet
   create: true

  3. For replacing docker image URLs, we used ImageTagTransformer

     apiVersion: builtin
 kind: ImageTagTransformer
 metadata:
   name: image-transformer-1
 imageTag:
   name: gcr.io/ml-pipeline/cache-deployer
   newName: docker.intuit.com/gcr-rmt/ml-pipeline/cache-deployer

    It will be helpful for any organization which has a proxy for accessing the internet, cloning all the container images local to your org is the way to go as the internet will not be required to access those images.

  4. For transforming VirtualServices

     - op: remove
   path: /spec/hosts/0
 - op: replace
   path: /spec/gateways/0
   value: <custom gateway>
 - op: add
   path: /spec/hosts/0
   value: <kubflow host name>
 - op: add
   path: /spec/exportTo
   value: ["."]

  5. Putting it all together

    envs -> prod/dev -> kustomization.yaml

     apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization

 resources:
 - ../base

 transformers:
 - transformers/image-transformers.yaml
 - transformers/label-transformers.yaml
 - transformers/annotations-transformers.yaml

 patchesJson6902:
 # patch VirtualService with explicit host
 # add multiple targets like below for all the VirtualServices which you need
 - path: patches/virtual-service-hosts.yaml
   target:
     group: networking.istio.io
     version: v1alpha3
     kind: VirtualService
     name: centraldashboard
  6. You might face issues with the metadata_envoy service, in our case we were getting the following error 
     [debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Server destroyed
 unable to bind domain socket with id=0 (see --base-id option)
 2021-01-29T23:32:26.680310Z error Epoch 0 exited with error: exit status 1

    After looking up, we found that, when you run this docker image with Istio Sidecar injection, this problem occurs. The reason is, both these containers are essentially envoyproxy containers and the default base-id for both containers is set to 0.

    So to make it work, we had to change CMD in this Dockerfile

     CMD ["/etc/envoy.yaml", "--base-id", "1"]

Step 3: Custom changes needed for SSO

There are two major components around authentication using SSO:

  1. Authservice: It is a StatefulSet that runs the oidc-auth service. It runs in the istio-system namespace and directly talks to an OIDC service for authentication
  2. Authn-filter: It’s an EnvoyFilter that filters the traffic to authservice and checks the Kubeflow auth header and redirects to authservice if the request is not authorized, check the presence of header called kubeflow-userid

Note: Intuit SSO supports OIDC, so we did not need to use dex for the integration. If your org’s SSO does not support OIDC, then you can use dex in the middle; details can be found here.

For our installation, we needed the authservice to be mesh-enabled, and it made more sense to move authservice to the kubeflow namespace as well, which was already enabled for Istio sidecar injection.

After enabling Istio mesh on authservice, some more changes were required in the default manifest for it to work. The authservice pod was not able to communicate with the Intuit SSO HTTPS URL, because outbound traffic from the main container pod is intercepted by Istio sidecar to enforce mtls (default behavior). So, we had to exclude the HTTPS port (443) to disable mtls. This can be done using the annotation traffic.sidecar.istio.io/excludeOutboundPorts: "443".

Step 4: Setting up ingress

We exposed the istio-ingressgateway service as LoadBalancer using the following mechanism:

  1. Setting up public hosted zone in Route 53, add hostname you would like to use, like example.com
  2. Setting up an ACM certificate for the hostname you want to use for the Kubeflow installation, the hostname can be kubeflow.example.com
  3. Updating the service manifest by adding a few annotations: 
    # Note that the backend talks over HTTP.
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
# TODO: Fill in with the ARN of your certificate.
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <cert arn from step 2>
service.beta.kubernetes.io/aws-load-balancer-security-groups: <to restrict access within org>
# Only run SSL on the port named "https" below.
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
external-dns.alpha.kubernetes.io/hostname: kubeflow.example.com

After applying the new manifest, AWS will automatically add the appropriate A and TXT entries in your hosted zone (example.com) and Kubeflow will be accessible at kubeflow.example.com.

To secure the Gateway with https, you can also change the gateway port and add the key and certificate in the Gateway.

More about these annotations can be found at Terminate HTTPS traffic on Amazon EKS and SSL support on AWS blog.

Step 5: Using an external Argo installation

Kubfelow uses Argo workflows internally to run the pipeline in a workflow fashion. Argo generates artifacts after the workflow steps and all we need to do is configure the artifact store if we are planning to use the external Argo:

  1. Install Argo workflows in your cluster, it gets installed in a namespace called argo.
  2. Remove all the Argo-related manifests from Kubeflow.
  3. To override the artifact store, you need to change the ConfigMap workflow-controller-configmap which comes with the Kubeflow manifest. It uses minio as the store but you can configure it to use S3 as well. More details can be found from the ArgoWorkflow Controller Configmap GitHub page.
  4. The latest version of Argo has the option to override artifact store for namespace as well.

Debugging tricks

  1. Check if EnvoyFilter is getting applied: you should have the istioctl cmd tool:

    istioctl proxy-config listeners <pod name> --port 15001 -o json

    See if the envoy filter is getting listed in the output. More about Istio proxy debugging can be found here.

  2. Check istio-ingressgateway:

     # Port forward to the first istio-ingressgateway pod
 kubectl -n istio-system port-forward $(kubectl -n istio-system get pods -listio=ingressgateway -o=jsonpath="{.items[0].metadata.name}") 15000

 # Get the http routes from the port-forwarded ingressgateway pod (requires jq)
 curl --silent http://localhost:15000/config_dump | jq '\''.configs.routes.dynamic_route_configs[].route_config.virtual_hosts[]| {name: .name, domains: .domains, route: .routes[].match.prefix}'\''

 # Get the logs of the first istio-ingressgateway pod
 # Shows what happens with incoming requests and possible errors
 kubectl -n istio-system logs $(kubectl -n istio-system get pods -listio=ingressgateway -o=jsonpath="{.items[0].metadata.name}") --tail=300

 # Get the logs of the first istio-pilot pod
 # Shows issues with configurations or connecting to the Envoy proxies
 kubectl -n istio-system logs $(kubectl -n istio-system get pods -listio=pilot -o=jsonpath="{.items[0].metadata.name}") discovery --tail=300

  3. Check the authservice connectivity: istio-ingressgateway pod should be able to access authservice. You can check that using the following command:

    kubectl -n istio-system exec $(kubectl -n istio-system get pods -listio=pilot -o=jsonpath="{.items[0].metadata.name}") -- curl -v http://authservice.istio-system.svc.cluster.local:8080

    Also, make sure authservice can reach dex:

    In our case, authservice is in the kubeflow namespace so we made changes accordingly using the command below:

    kubectl -n kubeflow exec authservice-0 -- wget -q -S -O '-' <oidc auth url>/.well-known/openid-configuration

    It should look something similar to:

     {
   "issuer": "http://dex.kubeflow.svc.cluster.local:5556/dex",
   "authorization_endpoint": "http://dex.kubeflow.svc.cluster.local:5556/dex/auth",
   "token_endpoint": "http://dex.kubeflow.svc.cluster.local:5556/dex/token",
   "jwks_uri": "http://dex.kubeflow.svc.cluster.local:5556/dex/keys",
   "userinfo_endpoint": "http://dex.kubeflow.svc.cluster.local:5556/dex/userinfo",
   "response_types_supported": [
     "code"
   ],
   "subject_types_supported": [
     "public"
   ],
   "id_token_signing_alg_values_supported": [
     "RS256"
   ],
   "scopes_supported": [
     "openid",
     "email",
     "groups",
     "profile",
     "offline_access"
   ],
   "token_endpoint_auth_methods_supported": [
     "client_secret_basic"
   ],
   "claims_supported": [
     "aud",
     "email",
     "email_verified",
     "exp",
     "iat",
     "iss",
     "locale",
     "name",
     "sub"
   ]
 }

  4. Check connectivity between services: try using curl or wget from one service to another. Usually one or the other is always available, otherwise, you can always install using the apt-get command. Example use case: from the ml-pipeline deployment pod you can check if pipeline APIs are accessible.

    kubectl -n kubeflow exec $(kubectl -n kubeflow get pods -lapp=ml-pipeline-ui -o=jsonpath="{.items[0].metadata.name}") -- wget -q -S -O '-' ml-pipeline.kubeflow.svc.cluster.local:8888/apis/v1beta1/pipelines

Asks for the Kubeflow Community

The challenges that we encountered at Intuit are not unique and will be faced by any enterprise that wants to adopt Kubeflow.

It would be nice to have Kubeflow play well with the available Kubernetes infrastructure in an enterprise, rather than mandating its own set of infrastructure. Here are some suggestions/bugs for improving the ecosystem, some of which Intuit will work with the community to build out:

  1. We saw Kubeflow manifest repo went through major folder restructuring for v1.3 but we think there is still room for improvements.
  2. Multi-Cluster / Multi-Region support. #5467
  3. Upgrade seems to be an issue in general, should figure out a way to manage this better. #5440
  4. Multi-tenancy with group support. #4188
  5. Installing Kubeflow in any custom namespace. #5647
  6. Existing metadata service is not performant, we did try some settings with more resources and horizontal scaling. The community is already working on KFP v2.0, which might address a lot of concerns around metadata service.

References