Building a Local Kubeflow MLOps Pipeline with Kubernetes

In AI, CI/CD Pipelines, DevOps Tools, Kubernetes, Linux Basics
July 16, 2025
120 Views

This guide walks through setting up a local Kubernetes cluster, installing Kubeflow, and running an end-to-end ML pipeline (training and serving) with Kubeflow Pipelines and KServe. We’ll use a scikit-learn model example and show commands and sample outputs at each step. All commands are run on a Linux or macOS terminal unless noted otherwise.

1. Setting Up Kubernetes Locally

First, create a local Kubernetes cluster. You can use Minikube or kind for this.

Minikube: Download and install Minikube, then start a cluster. For example:

  $ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  $ minikube start --driver=docker

On success you’ll see output like:

  * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

(Verify installation: minikube version shows something like minikube version: v1.33.1….)

kind: Install kind (for example via Homebrew on macOS or curl on Linux) and create a cluster:

  $ brew install kind                # or use curl on Linux as per Kubeflow docs:contentReference[oaicite:3]{index=3}
  $ kind create cluster --name mlops  # creates a new kind cluster

You should see Kubernetes components coming up, e.g.:

  Creating cluster "mlops" …

After this, kubectl get nodes should show your single node in Ready status.

2. Installing Kubeflow Locally

With Kubernetes ready, install Kubeflow (including Pipelines and KServe) on it. We use the Kubeflow manifests and kustomize to apply the full platform.

Clone Kubeflow manifests (recommended stable release, e.g. v1.10). Then apply with kustomize. For example:

   $ git clone -b v1.10.0 https://github.com/kubeflow/manifests.git
   $ cd manifests
   # Install all Kubeflow components (this can take several minutes)
   $ while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do \
       echo "Retrying apply..."; sleep 20; done

This uses the provided example kustomization.yaml to install the full Kubeflow suite. (Adjust memory/CPU per your machine if needed.)

Enable KServe (model serving component). If not already included, install KServe (formerly KFServing) with:

   $ kustomize build apps/kserve/kserve | kubectl apply --server-side --force-conflicts -f -:contentReference[oaicite:5]{index=5}

This deploys the KServe components into the cluster.

Verify Pods: Wait until all Kubeflow pods are Running/Completed. You can run kubectl get pods -n kubeflow --watch to watch status. When ready, port-forward the Kubeflow central dashboard:

   $ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80:contentReference[oaicite:6]{index=6}

Then open http://localhost:8080 in your browser to see the Kubeflow UI. (The Istio ingress gateway routes the UI and Pipelines.)

Sample Output: After port-forward, Kubeflow Central Dashboard should become accessible. No output is printed on success.

3. Preparing a Sample ML Model

Next, create and serialize a simple ML model. For example, train a scikit-learn classifier on the Iris dataset and save it with joblib:

# train_model.py
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import joblib

iris = datasets.load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, random_state=42
)
model = RandomForestClassifier()
model.fit(X_train, y_train)
joblib.dump(model, 'model.joblib')
print("Model training complete. Saved to model.joblib.")

Run the script:

$ python train_model.py
Model training complete. Saved to model.joblib.

This produces a file model.joblib that contains the trained model. We will use this serialized model later for serving.

4. Defining a Kubeflow Pipeline

Now define an ML workflow pipeline using the Kubeflow Pipelines Python SDK (kfp). A pipeline is a Python function decorated with @dsl.pipeline, and steps are ContainerOp (or @component) calls.

For example, you might create a pipeline with a single “TrainModel” step:

# pipeline.py
import kfp
from kfp import dsl

def train_op():
    return dsl.ContainerOp(
        name='TrainModel',
        image='python:3.8-slim',
        command=['bash', '-c'],
        arguments=['python -u train_model.py'],
        file_outputs={'model': '/app/model.joblib'}
    )

@dsl.pipeline(
    name='SampleTrainingPipeline',
    description='Train and save a model'
)
def training_pipeline():
    train_step = train_op()

Compile the pipeline to an IR YAML file. For example:

$ kfp dsl compile --py pipeline.py --output pipeline.yaml

(If using the Python API:)

from kfp import compiler
compiler.Compiler().compile(training_pipeline, 'pipeline.yaml')

This produces pipeline.yaml, which can be uploaded to the Kubeflow Pipelines UI.

5. Uploading and Running the Pipeline

With the pipeline definition (pipeline.yaml) ready, use the Kubeflow Pipelines UI to run it:

In the Kubeflow dashboard, go to Pipelines → Upload pipeline. Choose the pipeline.yaml file (or a zipped archive) and click Create.
After upload, switch to the Runs tab and click Create run. Select your uploaded pipeline, give the run a name, set any parameters (none in this simple example), and click Start.

The pipeline will begin executing. The UI will show a graph of components; each component’s status (Pending/Running/Succeeded/Failed) will update in real time.

6. Monitoring Pipeline Execution

To track progress and inspect outputs:

Click Experiments in the Pipelines UI, then select the experiment and the specific run you started.
Open the Run Output tab to see the overall run status and outputs of each step.
On the Graph tab, clicking a step node slides in details for that component. For example, you can view its stdout logs, which are useful for debugging. The Visualizations tab (appearing when a step is clicked) can show plots or metrics if your pipeline wrote them.

For example, after the “TrainModel” step finishes, you should see something like:

TrainModel   Succeeded

Clicking it reveals logs (from the Python stdout) and the output file model.joblib if any.

7. Packaging the Model for Inference (KServe)

Once the model is trained, we package it for serving via KServe. KServe uses a Kubernetes InferenceService CRD. Create a YAML manifest such as:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    sklearn:
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"

Here, storageUri points to the model artifacts (you can use a local PVC or MinIO bucket instead of GCS). This tells KServe to deploy a Scikit-learn predictor serving the model.

Apply the manifest:

$ kubectl apply -f inferenceservice.yaml

Check the InferenceService status:

$ kubectl get inferenceservices sklearn-iris -n kserve-test
NAME           URL                                             READY   PREV   LATEST   LATESTREADY   AGE
sklearn-iris   http://sklearn-iris.kserve-test.example.com      True    100    …        100           7d23h

The READY column should be True once the service is up.

8. Exposing the Model Endpoint

Finally, expose the model endpoint via Kubernetes:

Istio Ingress: KServe typically sets up an Istio gateway for inference traffic. Find the ingress IP and port:

  $ kubectl get svc istio-ingressgateway -n istio-system
  NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                           AGE
  istio-ingressgateway   LoadBalancer   172.21.109.129  130.211.10.121   80:31380/TCP,443:31390/TCP        17h

(Here EXTERNAL-IP and PORT(S) show where the gateway is listening.)

On platforms without a LoadBalancer (like bare Minikube/kind), you can port-forward the gateway or use NodePort. For example:

  kubectl port-forward svc/istio-ingressgateway -n istio-system 8088:80

Then http://localhost:8088 routes to the Istio ingress.

Access the model: Use the service name and namespace from the InferenceService manifest. If using DNS, the URL might be http://sklearn-iris.<namespace>.example.com/v1/models/sklearn-iris:predict. Or with the ingress host/port:

  export INGRESS_HOST=$(kubectl -n istio-system get svc istio-ingressgateway \
      -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  export INGRESS_PORT=$(kubectl -n istio-system get svc istio-ingressgateway \
      -o jsonpath='{.spec.ports[0].nodePort}')

Send a prediction request to http://$INGRESS_HOST:$INGRESS_PORT/v1/models/sklearn-iris:predict with your input JSON. The model will return the inferred class.

At this point, you have a fully local Kubeflow pipeline that trains a model, and you can serve that model via KServe on Kubernetes. You can monitor logs and outputs through the Kubeflow UI as described, and the model endpoint is exposed via Kubernetes services/ingress.