1 MongoDB Migration Guide: Upgrading from MongoDB 4.4/5.x to 8.2
- 1.1 Overview
- 1.2 Prerequisites
  - 1.2.1 Critical Requirements
- 1.3 Migration Steps
  - 1.3.1 Step 0: Stop Application (Keep MongoDB Running)
  - 1.3.2 Step 1: Create External Backup
    - 1.3.2.1 Option A: Using Volume Snapshots (Cloud environments)
    - 1.3.2.2 Option B: Using mongodump (if no backup infrastructure is available)
      - 1.3.2.2.1 Validate Your Backup
  - 1.3.3 Step 2: Scale Down
  - 1.3.4 Step 3: Configure Migration in Your Values File
    - 1.3.4.1 Configure MongoDB Architecture
    - 1.3.4.2 Verify and Configure Volume Mount Path
  - 1.3.5 Step 4: Update Your Helm Deployment
  - 1.3.6 Step 5: Monitor the Migration
  - 1.3.7 Step 6: Verify Migration Success
  - 1.3.8 Step 7: Verify MongoDB 8.2 is Running
  - 1.3.9 Step 8: Disable Migration Job (Cleanup)
  - 1.3.10 Step 9: Verify Application Functionality
- 1.4 Migration Process Details
  - 1.4.1 What Happens During Migration
- 1.5 Configuration Option
  - 1.5.1 Security Context
- 1.6 Troubleshooting
  - 1.6.1 Migration Job Fails to Start
    - 1.6.1.1 Symptom 1: Job is not created or fails immediately
    - 1.6.1.2 Symptom 2: Migration pod stuck in Pending state
  - 1.6.2 Preflight Checks Fail
    - 1.6.2.1 Symptom: Init container preflight-check fails
  - 1.6.3 Migration Fails During Execution
  - 1.6.4 Automatic Backup Fails
- 1.7 Rollback Procedure
  - 1.7.1 Using Automatic Backup (if enabled)
  - 1.7.2 Using External Backup
- 1.8 Best Practices
- 1.9 Support
- 1.10 Additional Notes

MongoDB Migration Guide: Upgrading from MongoDB 4.4/5.x to 8.2

This guide provides step-by-step instructions for migrating your MongoDB database from version 4.4 or 5.x to version 8.2 using the integrated mongodbMigration subchart.

Overview

The MongoDB migration is performed automatically by a Kubernetes Job that is integrated as a subchart in your product deployment (e.g., Smartfacts, OSLC Connector).

The migration process includes:

Automated preflight checks to verify system readiness
Automatic local snapshot for rollback capability (optional but recommended)
In-place data migration using MongoDB's upgrade tools
Automatic rollback if migration fails

The migration is performed with a migration image that contains all necessary mongodb versions. The image can be pulled from the following locations:

repo.mid.de/mongodb-migration-4-to-82:1.1.0
registry.mid.de/deployment/production/mongodb-migration-4-to-82:1.1.0

Prerequisites

Critical Requirements

External Backup (MANDATORY)
- Create a complete backup of your MongoDB data. If you don't have a backup ready, you can backup manually using mongodump by following these instructions.
- Store the backup in a safe location outside the Kubernetes cluster
- Verify the backup is complete and not corrupted
- The migration will NOT proceed without confirming this step
Sufficient Disk Space
- At least 30% free space on the MongoDB data volume
- Additional space if using volume-based automatic backup (recommended: 50% free)
Maintenance Window
- Plan for application downtime during migration
- Migration duration depends on database size (typically 5-30 minutes)
Kubernetes Access
- kubectl access to your cluster
- Permissions to scale StatefulSets, create and view Jobs/Pods

Migration Steps

Step 0: Stop Application (Keep MongoDB Running)

Stop all application components but keep MongoDB running for backup.

This ensures no new data is written during the backup process, while MongoDB remains available for creating the backup.

NAMESPACE="<your-namespace>"
# Scale down all Deployments and StatefulSets except MongoDB
kubectl get deployments,statefulsets -n $NAMESPACE -o name | \
  grep -v mongodb | \
  xargs -I {} kubectl scale {} --replicas=0 -n $NAMESPACE

# Verify only MongoDB is still running
kubectl get pods -n $NAMESPACE
# Expected: Only mongodb pod(s) should be in Running state

Note: If you have other databases or services in the namespace that should keep running, adjust the grep -v filter accordingly (e.g., grep -v "mongodb\|postgres").

Step 1: Create External Backup

Before proceeding, you MUST create a backup of your MongoDB data.

IMPORTANT: The backup must be created WHILE MongoDB is still running. Do NOT scale down MongoDB before creating the backup (unless using volume snapshots).

Option A: Using Volume Snapshots (Cloud environments)

If you have a backup infrastructure at hand, use this to performe the backup.

Option B: Using mongodump (if no backup infrastructure is available)

You can use this shell script to perform the backup and copy it to your host. Create a file called create-mongodb-backup.sh with the following content:

#!/bin/bash
# MongoDB Backup Script for Migration
# Creates a backup using mongodump and copies it to a local directory

set -e

SCRIPT_NAME=$(basename "$0")

usage() {
    echo "Usage: $SCRIPT_NAME <namespace> <backup-directory>"
    echo ""
    echo "Arguments:"
    echo "  namespace         Kubernetes namespace where MongoDB is running"
    echo "  backup-directory  Local directory where backup will be stored"
    echo ""
    echo "Example:"
    echo "  $SCRIPT_NAME my-namespace /backup/mongodb"
    exit 1
}

# Check arguments
if [ $# -ne 2 ]; then
    usage
fi

NAMESPACE="$1"
BACKUP_DIR="$2"
MONGODB_NAME="mongodb"  # Adjust if your MongoDB has a different name

echo "=========================================="
echo "MongoDB Backup Script"
echo "=========================================="
echo "Namespace:      $NAMESPACE"
echo "Backup Dir:     $BACKUP_DIR"
echo "MongoDB Name:   $MONGODB_NAME"
echo ""

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Automatically detect MongoDB pod name
echo "[Step 1/3] Detecting MongoDB pod..."
MONGODB_POD=$(kubectl get pods -n $NAMESPACE | grep $MONGODB_NAME | grep Running | awk '{print $1}' | head -n 1)

if [ -z "$MONGODB_POD" ]; then
  echo "❌ ERROR: No running MongoDB pod found in namespace $NAMESPACE"
  echo "Make sure MongoDB is running and the pod name contains '$MONGODB_NAME'"
  exit 1
fi

echo "✓ Using MongoDB pod: $MONGODB_POD"
echo ""

# Create backup using mongodump
echo "[Step 2/3] Creating backup using mongodump..."
echo "This may take several minutes depending on database size..."
BACKUP_TIMESTAMP=$(date +%Y.%m.%d_%H.%M.%S)
BACKUP_FILE="mongobackup_${BACKUP_TIMESTAMP}.gz"

kubectl exec $MONGODB_POD -n $NAMESPACE -- \
  mongodump --gzip --archive=/tmp/$BACKUP_FILE --verbose

echo "✓ Backup created in pod: /tmp/$BACKUP_FILE"
echo ""

# Copy backup to local machine
echo "[Step 3/3] Copying backup to local directory..."
kubectl cp $NAMESPACE/$MONGODB_POD:/tmp/$BACKUP_FILE "$BACKUP_DIR/$BACKUP_FILE"

if [ ! -f "$BACKUP_DIR/$BACKUP_FILE" ]; then
  echo "❌ ERROR: Backup file not found at $BACKUP_DIR/$BACKUP_FILE"
  exit 1
fi

BACKUP_SIZE=$(du -h "$BACKUP_DIR/$BACKUP_FILE" | awk '{print $1}')
echo "✓ Backup copied to: $BACKUP_DIR/$BACKUP_FILE (Size: $BACKUP_SIZE)"
echo ""

echo "=========================================="
echo "✓ Backup completed successfully!"
echo "=========================================="
echo "Backup file: $BACKUP_DIR/$BACKUP_FILE"
echo "Size:        $BACKUP_SIZE"
echo ""
echo "Next steps:"
echo "1. Verify backup integrity:"
echo "   gzip -t $BACKUP_DIR/$BACKUP_FILE"
echo ""
echo "2. Scale down MongoDB:"
echo "   kubectl scale deployment,statefulset $MONGODB_NAME --replicas=0 -n $NAMESPACE"
echo "   kubectl wait --for=delete pod/$MONGODB_POD -n $NAMESPACE --timeout=300s"
echo ""

Usage:

NAMESPACE="<your-namespace>"
# Make script executable
chmod +x create-mongodb-backup.sh
# Run the script with namespace and backup directory
./create-mongodb-backup.sh $NAMESPACE <backup-directory>

Important: Do not proceed with migration until you have verified your backup is complete and accessible.

Validate Your Backup

After creating the backup, you must verify it is complete and not corrupted. The most reliable validation is comparing the collections in your backup with the collections in your database.

Use the following script to compare the number of user collections contained in the backup with the number of user collections in the mongodb.

Create a file called validate-backup.sh with the following content:

#!/bin/bash

set -e

SCRIPT_NAME=$(basename "$0")

usage() {
    echo "Usage: $SCRIPT_NAME <chart-path> <values-file> <backup-file> <namespace>"
    echo ""
    echo "Arguments:"
    echo "  chart-path       Product Helm chart (e.g., smartfacts/smartfacts or library/genoslc)"
    echo "  values-file      Path to the values.yaml file"
    echo "  backup-file      Path to the MongoDB backup archive (.gz file)"
    echo "  namespace        Kubernetes namespace"
    echo ""
    echo "Example:"
    echo "  $SCRIPT_NAME /path/to/genoslc values.yaml /backup/mongobackup_2024.12.21.gz default"
    exit 1
}

if [ $# -ne 4 ]; then
    usage
fi

CHART_PATH="$1"
VALUES_FILE="$2"
BACKUP_FILE="$3"
NAMESPACE="$4"
MONGODB_NAME="mongodb"  # Adjust if your MongoDB has a different name

echo "=========================================="
echo "MongoDB Backup Validation Script"
echo "=========================================="
echo "Chart:     $CHART_PATH"
echo "Values:    $VALUES_FILE"
echo "Backup:    $BACKUP_FILE"
echo "Namespace: $NAMESPACE"
echo "=========================================="
echo ""

# Automatically detect MongoDB pod name
echo "[Step 1/5] Detecting MongoDB pod..."
MONGODB_POD=$(kubectl get pods -n "$NAMESPACE" | grep $MONGODB_NAME | grep Running | awk '{print $1}' | head -n 1)

if [ -z "$MONGODB_POD" ]; then
  echo "❌ ERROR: No running MongoDB pod found in namespace $NAMESPACE"
  echo "Make sure MongoDB is running and the pod name contains '$MONGODB_NAME'"
  exit 1
fi

echo "✓ Using MongoDB pod: $MONGODB_POD"
echo ""

# Step 2: List collections from database
echo "[Step 2/5] Extracting collections from MongoDB database..."
kubectl exec "$MONGODB_POD" -n "$NAMESPACE" -- mongosh --quiet --eval "
  db.adminCommand('listDatabases').databases.forEach(function(database) {
    if (database.name !== 'admin' && database.name !== 'local' && database.name !== 'config') {
      db.getSiblingDB(database.name).getCollectionNames().forEach(function(collection) {
        print(database.name + '.' + collection);
      });
    }
  });
" | sort > database_collections.txt

DB_COLLECTION_COUNT=$(wc -l < database_collections.txt)
echo "✓ Found $DB_COLLECTION_COUNT collections in database"
echo ""

# Step 3: Verify backup file
echo "[Step 3/6] Verifying backup archive integrity..."
if [ ! -f "$BACKUP_FILE" ]; then
    echo "❌ ERROR: Backup file not found: $BACKUP_FILE"
    exit 1
fi

BACKUP_SIZE=$(du -h "$BACKUP_FILE" | awk '{print $1}')
echo "✓ Backup file exists (Size: $BACKUP_SIZE)"

gzip -t "$BACKUP_FILE"
if [ $? -eq 0 ]; then
    echo "✓ Backup archive integrity OK"
else
    echo "❌ ERROR: Backup archive is corrupted!"
    exit 1
fi
echo ""

# Step 4: Determine MongoDB image
echo "[Step 4/6] Determining MongoDB image from chart..."
MONGODB_IMAGE=$(helm template temp-release "$CHART_PATH" -f "$VALUES_FILE" 2>/dev/null | grep "image:.*mongo" | head -n 1 | awk '{print $2}' | tr -d '"')

if [ -z "$MONGODB_IMAGE" ]; then
    echo "❌ ERROR: Could not determine MongoDB image from chart"
    exit 1
fi

echo "✓ Using MongoDB image: $MONGODB_IMAGE"
echo ""

# Step 5: Create temporary pod and extract collections from backup
echo "[Step 5/6] Creating temporary pod to analyze backup..."
kubectl run mongodb-backup-check --image="$MONGODB_IMAGE" --restart=Never \
  --command -n "$NAMESPACE" -- sleep 3600 2>/dev/null || true

echo "Waiting for pod to be ready..."
kubectl wait --for=condition=Ready pod/mongodb-backup-check -n "$NAMESPACE" --timeout=60s

echo "Copying backup to temporary pod..."
kubectl cp "$BACKUP_FILE" "$NAMESPACE/mongodb-backup-check:/tmp/backup.gz"

echo "Extracting collection list from backup..."
echo "Starting temporary MongoDB instance in pod..."

# Start a temporary MongoDB instance in the background
kubectl exec mongodb-backup-check -n "$NAMESPACE" -- \
  sh -c "mkdir -p /tmp/mongodata && mongod --dbpath /tmp/mongodata --port 27099 --bind_ip 127.0.0.1 --fork --logpath /tmp/mongod.log" || {
    echo "❌ ERROR: Failed to start temporary MongoDB instance"
    kubectl exec mongodb-backup-check -n "$NAMESPACE" -- cat /tmp/mongod.log 2>/dev/null || true
    kubectl delete pod mongodb-backup-check -n "$NAMESPACE" --wait=false 2>/dev/null || true
    exit 1
}

echo "Waiting for MongoDB to be ready..."
sleep 5

# Run mongorestore with the temporary MongoDB instance
kubectl exec mongodb-backup-check -n "$NAMESPACE" -- \
  sh -c "mongorestore --host=127.0.0.1:27099 --gzip --archive=/tmp/backup.gz --dryRun --verbose 2>&1" > restore_output.txt

# Extract collection names from mongorestore output
grep -oE "[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+" restore_output.txt | \
  grep -v "^admin\." | \
  grep -v "^local\." | \
  grep -v "^config\." | \
  grep -v "\.system\." | \
  grep -v "\.log$" | \
  grep -v "\.sock$" | \
  grep -v "^[0-9]\+\.[0-9]\+$" | \
  grep "[a-zA-Z]" | \
  sort -u > backup_collections.txt

BACKUP_COLLECTION_COUNT=$(wc -l < backup_collections.txt)
echo "✓ Found $BACKUP_COLLECTION_COUNT collections in backup"

# Shutdown temporary MongoDB
kubectl exec mongodb-backup-check -n "$NAMESPACE" -- \
  sh -c "mongosh --port 27099 --eval 'db.getSiblingDB(\"admin\").shutdownServer()' --quiet" 2>/dev/null || true

echo "Cleaning up temporary pod..."
kubectl delete pod mongodb-backup-check -n "$NAMESPACE" --wait=false 2>/dev/null || true
echo ""

# Step 6: Compare collections
echo "[Step 6/6] Comparing collections..."
echo "=========================================="

MISSING_IN_BACKUP=$(comm -23 database_collections.txt backup_collections.txt)
EXTRA_IN_BACKUP=$(comm -13 database_collections.txt backup_collections.txt)
COMMON=$(comm -12 database_collections.txt backup_collections.txt)
COMMON_COUNT=$(echo "$COMMON" | grep -c . || echo 0)

if [ -z "$MISSING_IN_BACKUP" ]; then
    echo "✓ All database collections are present in backup!"
else
    echo "❌ WARNING: Collections in database but NOT in backup:"
    echo "$MISSING_IN_BACKUP"
    echo ""
fi

if [ -n "$EXTRA_IN_BACKUP" ]; then
    echo "ℹ Collections in backup but not in current database (usually OK):"
    echo "$EXTRA_IN_BACKUP"
    echo ""
fi

echo "Summary:"
echo "  Collections in database: $DB_COLLECTION_COUNT"
echo "  Collections in backup:   $BACKUP_COLLECTION_COUNT"
echo "  Collections in both:     $COMMON_COUNT"
echo "=========================================="
echo ""

# Final result
if [ -z "$MISSING_IN_BACKUP" ]; then
    echo "✅ VALIDATION SUCCESSFUL"
    echo "Your backup contains all collections."
    echo ""
    echo "Files created:"
    echo "  - database_collections.txt (collections in database)"
    echo "  - backup_collections.txt (collections in backup)"
    exit 0
else
    echo "❌ VALIDATION FAILED"
    echo "Some collections are missing from the backup."
    echo "Please recreate the backup before proceeding with migration."
    exit 1
fi

Usage:

NAMESPACE="<your-namespace>"
# Make the script executable
chmod +x validate-backup.sh

# Run the script
./validate-backup.sh \
  chart-name \
  values.yaml \
  /path/to/mongobackup_*.gz \
  $NAMESPACE

# Example:
./validate-backup.sh \
  library/genoslc \
  values_genoslc.yaml \
  /backup/mongobackup_2024.12.21_10.30.45.gz \
  $NAMESPACE

# Expected output on success:
# ✅ VALIDATION SUCCESSFUL
# Your backup is complete and contains all collections.

# If validation fails:
# ❌ VALIDATION FAILED
# Some collections are missing from the backup.

After successful validation:

Backup is stored in a safe location outside the Kubernetes cluster
Backup location is documented and accessible to your team
You can proceed with Step 2 (Scale Down MongoDB)

Step 2: Scale Down

Stop all pods before starting the migration.

The migration job requires MongoDB and all other pods to be stopped to safely upgrade the data files.

# Set variables
NAMESPACE="<your-namespace>"

# Scale down MongoDB
kubectl scale deployment,statefulset --all --replicas=0 -n $NAMESPACE

# Verify all pods are stopped
kubectl get pods -n $NAMESPACE
# Expected: No pods should be running

Important: Do not proceed to Step 3 until MongoDB is completely stopped.

Step 3: Configure Migration in Your Values File

Edit the values file of your deployment to enable the migration and update MongoDB to version 8.2:

# Update MongoDB to version 8.2
mongodb:
  enabled: true
  image:
    tag: "8.2.5-1"

# Enable the migration job
mongodbMigration:
  enabled: true
  
  # CRITICAL: Set to true ONLY after completing external backup - see instructions for backup using mongodump (https://smartfacts.atlassian.net/wiki/x/CoDPFw) if you don`t have a backup infrastructure at hand
  mongodbBackupDone: false # set to true after backup is completed

  mongodb:
    # Overwrite archtecture with "replicaset", if you have done this for the mongodb in your values yaml.
    architecture: "standalone"
    # Overwrite useStatefulSet with "true", if you have done this for the mongodb in your values yaml.
    useStatefulSet: false
    # Important: Set the replicaCount to the actual number of replicas in your deployment.
    replicaCount: 1
    persistence:
      # IMPORTANT for architecture "standalone": State the name of the mongodb pvc here. (This value is ignored for architecture "replicaset".)
      existingClaim: "<your-pvc-name>"
      # IF your architecture is "replicaset" and you did override the names of the PVCs 
      # (so that they cannot be deduced byp the migration job), state the list of ALL mongodb 
      # claims (master and replicas) here:
      replicaSetClaims: []
  
  # Automatic rollback dump configuration (recommended)
  # This creates a local mongodump for automatic rollback if migration fails
  # This is NOT a disaster recovery backup - you still need the external backup above
  autoRollbackDump:
    enabled: true
    # "emptyDir" = faster, ephemeral storage (recommended)
    # "volume" = stores dump on MongoDB data volume (requires more disk space)
    location: "emptyDir"

Important: Both the new MongoDB image update and the migration job must be configured in the same Helm upgrade. The migration job will run first and upgrade the data, then MongoDB 8.2 will start with the migrated data.

Configure MongoDB Architecture

You must configure the MongoDB architecture settings to match your current MongoDB deployment. The migration job needs to know:

Whether MongoDB runs as a Deployment or StatefulSet
Whether it's a standalone instance or a replicaset
The correct PVC name to mount

If your architecture is “replicaset”, the chart iterates through PVCs <release-name>-mongodb-0 to <release-name>-mongodb-(replicaCount-1) and sequentially migrates all volumes in the range. Therefore, it is important that the mongodbMigration.mongodb.replicaCount key is set to the correct number of replicas.

PVC name derivation works as long your PVC names have the default structure datadir-<fullname>-<ordinal>.

If your PVCs have a different naming scheme, the names cannot be derived. In this case, specify ALL Mongodb PVC names in mongodbMigration.mongodb.persistence.replicaSetClaims. The chart iterates through this list and migrates all PVCs specified there.

Step 1: Determine your MongoDB configuration

Check your current MongoDB deployment type:

NAMESPACE="<your-namespace>"

# Check if MongoDB is a Deployment or StatefulSet
kubectl get deployment,statefulset -n $NAMESPACE | grep mongodb

# Check the PVC name
kubectl get pvc -n $NAMESPACE | grep mongodb

Verify and Configure Volume Mount Path

IMPORTANT: The migration job must use the same volume mount path as your MongoDB deployment.

Step 1: Check MongoDB's volume mount path

NAMESPACE="<your-namespace>"
MONGODB_POD=$(kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=mongodb -o jsonpath='{.items[0].metadata.name}')

# Check the mount path
kubectl get pod $MONGODB_POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].volumeMounts[?(@.name=="datadir")].mountPath}'
# Common outputs:
# - /bitnami/mongodb (Bitnami MongoDB - default)
# - /data/db (Official MongoDB image)

Step 2: Configure the mount path if different from default

If your MongoDB uses a non-default mount path (anything other than /bitnami/mongodb), you must configure it:

mongodbMigration:
  enabled: true
  mongodbBackupDone: true
  
  mongodb:
    architecture: "standalone"
    useStatefulSet: false
    persistence:
      existingClaim: "<your-pvc-name>"
      mountPath: "/your/custom/path"  # Set this to match MongoDB's mount path

Default mount paths by MongoDB image:

Bitnami MongoDB: /bitnami/mongodb (default, no configuration needed)
Official MongoDB: /data/db (requires configuration)
Custom images: Check with kubectl command above

Step 4: Update Your Helm Deployment

Deploy the updated configuration:

# Update Helm repository to get latest chart versions
helm repo update

# Apply the upgrade to start the migration
helm upgrade <release-name> <repo-name>/<chart-name> \
  -f values.yaml \
  -n <your-namespace>

Step 5: Monitor the Migration

The migration job will start automatically. Monitor its progress:

# Follow migration logs in real-time (recommended)
kubectl logs -f job/<mongodb-name>-migration -n $NAMESPACE

# Example for genoslc:
# kubectl logs -f job/genoslc-mongodb-migration -n default

# If no logs appear, check if the job was created:
kubectl get jobs -n <namespace> | grep migration

# View job details if there are issues:
kubectl describe job <mongodb-name>-migration -n $NAMESPACE

Step 6: Verify Migration Success

Check the job status:

# Check job completion status
kubectl get job <mongodb-name>-migration -n <namespace>

# Expected output for successful migration:
# NAME                      COMPLETIONS   DURATION   AGE
# mongodb-migration         1/1           5m23s      6m

Success indicators:

Job shows COMPLETIONS: 1/1
No error messages in logs
Final log message: "Migration completed successfully"

Step 7: Verify MongoDB 8.2 is Running

After successful migration, MongoDB 8.2 will automatically start. Verify it's running correctly:

NAMESPACE="<your-namespace>"
# Check MongoDB pod is running
kubectl get pods -n $NAMESPACE | grep mongodb

# Check MongoDB version and that it is accepting connections
kubectl exec <mongodb-pod-name> -n $NAMESPACE -- mongosh --eval "db.version()"
# Expected: 8.2.x

Step 8: Disable Migration Job (Cleanup)

After verifying MongoDB 8.2 is running successfully, disable the migration job to prevent it from running again:

# Disable migration job
mongodbMigration:
  enabled: false

# Keep MongoDB 8.2 configuration
mongodb:
  image:
    tag: "8.2-MID1.0.0"

Step 9: Verify Application Functionality

Check that MongoDB is running and accepting connections
Verify your application can connect to MongoDB
Test critical application functionality
Monitor application logs for any database-related errors

Migration Process Details

What Happens During Migration

Preflight Checks
- Verifies MongoDB data directory exists
- Confirms MongoDB is stopped (no active lock file)
- Checks sufficient disk space (>30% free)
- Validates backup confirmation flag
Automatic Local Backup (if enabled)
- Creates a mongodump for rollback purposes
- Stored in ephemeral or persistent storage
- Note: This is NOT a disaster recovery backup
Data Migration
- Runs feature compatibility version upgrade with every major version of mongodb
- Rebuilds indexes if needed
- Validates data integrity
Automatic Rollback (if migration fails)
- Restores data from automatic backup
- Logs detailed error information
- Job fails with clear error message

Configuration Option

Security Context

Adjust if running on OpenShift or with specific security requirements:

mongodbMigration:
  podSecurityContext:
    enabled: true
    fsGroup: 0
    runAsUser: 1001
  
  containerSecurityContext:
    enabled: true
    runAsUser: 1001
    runAsGroup: 0

Troubleshooting

Migration Job Fails to Start

Symptom 1: Job is not created or fails immediately

Possible causes:

mongodbBackupDone is not set to true
MongoDB persistence is disabled
Incorrect MongoDB configuration

Solution:

# Check Helm template rendering
helm template <release-name> /path/to/your/chart -f values.yaml | grep -A 50 "kind: Job"

# Verify values are correct
helm get values <release-name> -n <namespace>

Symptom 2: Migration pod stuck in `Pending` state

Possible causes:

PVC does not exist (most common)
- Wrong architecture setting (standalone vs replicaset)
- Wrong useStatefulSet setting (true vs false)
- Wrong existingClaim name
PVC exists but is not bound
Insufficient cluster resources
Node selector/affinity issues

Solution:

NAMESPACE="<your-namespace>"

# Check pod events for detailed error
kubectl describe pod -n $NAMESPACE -l job-name=<mongodb-name>-migration

# Common error: "persistentvolumeclaim not found"
# This means the PVC name is wrong - verify your configuration

# Check if PVC exists
kubectl get pvc -n $NAMESPACE | grep mongodb

# Check MongoDB deployment type
kubectl get deployment,statefulset -n $NAMESPACE | grep mongodb

# Verify your values.yaml matches the actual MongoDB deployment
# See "Verify and Configure Volume Mount Path" section above

Preflight Checks Fail

Symptom: Init container `preflight-check` fails

Common errors:

MongoDB still running
ERROR: MongoDB appears to be running (mongod.lock file exists)
Solution: Scale down MongoDB StatefulSet
kubectl scale statefulset <mongodb-name> --replicas=0 -n <namespace>
Insufficient disk space
ERROR: Disk usage is 85% - need at least 30% free space
Solution: Free up disk space or expand the volume
No WiredTiger files found / Wrong data directory
ERROR: No WiredTiger files found in /bitnami/mongodb/data/db This directory does not contain MongoDB data!
Possible causes:
- Wrong mongodb.persistence.mountPath configured
- Wrong PVC mounted (incorrect architecture or useStatefulSet setting)
- Wrong existingClaim name
Solution: Verify your configuration matches the actual MongoDB deployment
# Check MongoDB's actual mount path NAMESPACE="<your-namespace>" MONGODB_POD=$(kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=mongodb -o jsonpath='{.items[0].metadata.name}') kubectl get pod $MONGODB_POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].volumeMounts[?(@.name=="datadir")].mountPath}' # Check MongoDB's PVC kubectl get pvc -n $NAMESPACE | grep mongodb # Check if MongoDB is Deployment or StatefulSet kubectl get deployment,statefulset -n $NAMESPACE | grep mongodb
Then update your values.yaml:
mongodbMigration: mongodb: architecture: "standalone" # or "replicaset" useStatefulSet: false # or true persistence: existingClaim: "<actual-pvc-name>" # if Deployment mountPath: "/bitnami/mongodb" # match MongoDB's mount path

Migration Fails During Execution

Symptom: Main container fails with errors

Solution:

Check detailed logs:
kubectl logs job/<mongodb-name>-migration -c migration -n <namespace>
If automatic rollback is enabled, data will be restored automatically
Review error messages and contact support if needed

Automatic Backup Fails

Symptom: Init container create-rollback-backup fails

Common causes:

Insufficient disk space
Corrupted MongoDB data
Permission issues

Solution:

# Check init container logs
kubectl logs job/<mongodb-name>-migration -c create-rollback-backup -n <namespace>

# Consider using emptyDir instead of volume
# Edit values.yaml:
mongodbMigration:
  autoBackup:
    location: "emptyDir"

Rollback Procedure

If you need to rollback to the previous MongoDB version after migration:

Using Automatic Backup (if enabled)

The migration job automatically rolls back if it fails. Manual rollback is not typically needed.

IMPORTANT: The automatic rollback restores data using MongoDB 5.0. After an automatic rollback, you must use MongoDB 5.0 or higher - you cannot return to MongoDB 4.4.

If you started with MongoDB 4.4 and the migration failed, the rollback will restore your data but with FCV 5.0 set. MongoDB 4.4 cannot start with FCV 5.0 data.

After automatic rollback:

Update your MongoDB image to 5.0 or higher:
mongodb: image: tag: "5.0-MID1.0.0"
Disable migration to prevent it from running again:
mongodbMigration: enabled: false
Deploy and verify your application works with MongoDB 5.0

If you must return to MongoDB 4.4: You need to restore from your external backup using MongoDB 4.4 binaries (see "Using External Backup" section below).

Using External Backup

CRITICAL: If migration failed partially (e.g., disk full during migration), you MUST clear the data directory before restoring.

MongoDB 5.0 cannot read data with FCV 6.0 or higher. If the migration progressed beyond FCV 5.0, the WiredTiger metadata may be incompatible with MongoDB 5.0.

Scale down MongoDB:
kubectl scale statefulset <mongodb-name> --replicas=0 -n <namespace>
Clear the data directory completely:
# Get a shell in a pod with the MongoDB PVC mounted kubectl run -it --rm mongodb-cleanup \ --image=busybox \ --overrides='{"spec":{"containers":[{"name":"cleanup","image":"busybox","command":["sh"],"volumeMounts":[{"name":"datadir","mountPath":"/data"}]}],"volumes":[{"name":"datadir","persistentVolumeClaim":{"claimName":"<your-pvc-name>"}}]}}' \ -n <namespace> # Inside the pod, clear the data directory rm -rf /data/data/db/* # Or if using different mount structure: rm -rf /data/* # Exit the pod exit
Restore data from your external backup:
IMPORTANT: Use the same MongoDB version as your backup!
- If your backup is from MongoDB 4.4, use MongoDB 4.4 for restore
- If your backup is from MongoDB 5.0, use MongoDB 5.0 for restore
# Start temporary MongoDB pod with the SAME version as your backup # For MongoDB 4.4 backup: kubectl run -it --rm mongodb-restore \ --image=bitnami/mongodb:4.4 \ --overrides='{"spec":{"containers":[{"name":"restore","image":"bitnami/mongodb:4.4","command":["sh"],"volumeMounts":[{"name":"datadir","mountPath":"/bitnami/mongodb"}]}],"volumes":[{"name":"datadir","persistentVolumeClaim":{"claimName":"<your-pvc-name>"}}]}}' \ -n <namespace> # Inside the pod, start MongoDB and restore mongod --dbpath /bitnami/mongodb/data/db --port 27017 --bind_ip 0.0.0.0 --fork --logpath /tmp/mongod.log mongorestore --port 27017 --dir /path/to/your/backup --drop mongosh --port 27017 --eval "db.getSiblingDB('admin').shutdownServer()" exit
Update values to use MongoDB 5.0:
mongodb: image: registry: repo.mid.de repository: mongodb tag: "5.0-MID1.0.0" # Disable migration to prevent it from running again mongodbMigration: enabled: false
Deploy and verify

Why clearing the data directory is necessary:

If migration reached FCV 6.0, 7.0, or 8.0, WiredTiger metadata is incompatible with MongoDB 5.0
MongoDB 5.0 will fail to start or corrupt data if it tries to read higher FCV data
Clearing ensures a clean state for the restore operation

Why matching MongoDB versions is critical:

MongoDB 4.4 cannot start with FCV 5.0 or higher data
MongoDB 5.0 cannot start with FCV 6.0 or higher data
Always restore with the same MongoDB version that created the backup
After successful restore, you can attempt migration again

Best Practices

Always create external backups before migration
Test migration in non-production environment first
Schedule migration during maintenance window with minimal user activity
Monitor disk space before and during migration
Keep automatic backup enabled for safety
Document your specific configuration for future reference
Verify application functionality thoroughly after migration

Support

If you encounter issues during migration:

Collect diagnostic information:
kubectl describe job <mongodb-name>-migration -n <namespace> > migration-job.txt kubectl logs job/<mongodb-name>-migration -c migration -n <namespace> > migration-logs.txt kubectl get events -n <namespace> --sort-by='.lastTimestamp' > events.txt
Contact MID DevOps Team at cops@mid.de with:
- Migration logs
- Job description
- Your values.yaml configuration (sanitized)
- Error messages

Additional Notes

The migration job has a TTL of 1 hour after completion (configurable via job.ttlSecondsAfterFinished)
Job will not retry on failure (backoffLimit: 0)
Migration is designed to be idempotent - safe to re-run if it fails
The mongodbMigration chart is automatically included as a dependency in product charts

Excerpt: Upgrading from MongoDB 4.x/5.x to 8.2

MongoDB Migration Guide: Upgrading from MongoDB 4.4/5.x to 8.2

Overview

Prerequisites

Critical Requirements

Migration Steps

Step 0: Stop Application (Keep MongoDB Running)

Step 1: Create External Backup

Option A: Using Volume Snapshots (Cloud environments)

Option B: Using mongodump (if no backup infrastructure is available)

Validate Your Backup

Step 2: Scale Down

Step 3: Configure Migration in Your Values File

Configure MongoDB Architecture

Verify and Configure Volume Mount Path

Step 4: Update Your Helm Deployment

Step 5: Monitor the Migration

Step 6: Verify Migration Success

Step 7: Verify MongoDB 8.2 is Running

Step 8: Disable Migration Job (Cleanup)

Step 9: Verify Application Functionality

Migration Process Details

What Happens During Migration

Configuration Option

Security Context

Troubleshooting

Migration Job Fails to Start

Symptom 1: Job is not created or fails immediately

Symptom 2: Migration pod stuck in Pending state

Preflight Checks Fail

Symptom: Init container preflight-check fails

Migration Fails During Execution

Automatic Backup Fails

Rollback Procedure

Using Automatic Backup (if enabled)

Using External Backup

Best Practices

Support

Additional Notes

Symptom 2: Migration pod stuck in `Pending` state

Symptom: Init container `preflight-check` fails