🧬 Colorectal Cancer Prediction – End-to-End MLOps Pipeline

An end-to-end machine learning system for colorectal cancer prediction built using modern MLOps practices. The project demonstrates how machine learning models can be developed, tracked, automated, and deployed using tools such as MLflow, DagsHub, Kubeflow, and Kubernetes.

The system includes data preprocessing, model training, experiment tracking, automated pipelines, containerization, and scalable deployment.

---

🚀 Tech Stack

---

📌 Project Overview

This project focuses on building a machine learning model to predict colorectal cancer risk using patient data.

The objective is to create a reproducible and scalable ML pipeline that integrates modern MLOps tools to manage the entire lifecycle of the model.

The system demonstrates how machine learning workflows can transition from research experiments to production deployment.

---

🏗 System Architecture

The project follows a cloud-native MLOps architecture where machine learning pipelines are automated and executed using Kubeflow pipelines on Kubernetes.

Workflow

1. Developer pushes code to GitHub
2. ML pipeline is defined using Kubeflow Pipelines
3. MLflow tracks experiments and model metrics
4. DagsHub manages experiment collaboration and model tracking
5. Docker containerizes the ML application
6. Kubernetes (Minikube) orchestrates deployment
7. Flask application serves predictions to users

Developer
   │
   ▼
GitHub Repository
   │
   ▼
Kubeflow Pipeline
   │
   ├── Data Processing
   ├── Model Training
   ├── Model Evaluation
   │
   ▼
MLflow Experiment Tracking
   │
   ▼
DagsHub Model Repository
   │
   ▼
Docker Container
   │
   ▼
Kubernetes (Minikube)
   │
   ▼
Flask Prediction API

---

🤖 Machine Learning Pipeline

The automated ML workflow consists of the following stages:

Raw Medical Dataset
   │
   ▼
Data Ingestion
   │
   ▼
Data Preprocessing
   │
   ▼
Feature Engineering
   │
   ▼
Model Training
   │
   ▼
Model Evaluation
   │
   ▼
Experiment Tracking (MLflow)
   │
   ▼
Model Packaging (Docker)
   │
   ▼
Pipeline Automation (Kubeflow)
   │
   ▼
Deployment (Kubernetes)

---

⚙ Kubeflow Pipeline

The Kubeflow pipeline orchestrates the machine learning workflow.

Pipeline stages include:

• Data ingestion
• Data preprocessing
• Model training
• Model evaluation
• Model artifact generation
• Deployment preparation

Kubeflow enables reproducible ML workflows and scalable pipeline execution.

---

🌐 Web Application

A Flask-based web interface allows users to interact with the prediction model.

Features include:

• Input patient health parameters
• Generate cancer risk prediction
• Real-time inference using trained ML model
• REST API for prediction requests

---

📁 Project Structure

Colorectal-Cancer-Prediction
│
├── artifacts/              # Processed datasets and model outputs
├── kubeflow_pipeline/      # Kubeflow pipeline definitions
├── notebook/               # Data analysis and experimentation
├── src/                    # Model training and ML logic
├── static/                 # Static assets for Flask UI
├── templates/              # HTML templates
│
├── application.py          # Flask web application
├── mlops_pipeline.yaml     # Kubeflow pipeline configuration
├── Dockerfile              # Docker container setup
├── requirements.txt        # Python dependencies
├── setup.py                # Project setup
└── README.md

---

🧠 Skills Demonstrated

This project demonstrates practical experience with production-ready MLOps workflows.

• Healthcare Machine Learning Modeling
• Kubeflow Pipeline Orchestration
• MLflow Experiment Tracking
• DagsHub ML Collaboration
• Docker Containerization
• Kubernetes Deployment
• Flask Model Serving
• Reproducible ML Pipelines

---

📜 License

This project is licensed under the GPL-3.0 License.