Hero Section: What We Are Building

Building a single "Hello World" container rarely prepares an engineer for modern enterprise environments. Production architectures are rarely single-process scripts; they are polyglot distributed systems with disparate runtimes, database write-locks, network isolation policies, memory limits, and strict non-root security boundaries.

In this guide, we take a real-world, 5-tier polyglot microservices system and build it from the bare source code up to an orchestrated, production-hardened deployment. We will not sweep real debugging failures under the rug: we will reproduce actual runtime connection drops, broken shared libraries, stale connection pools, and container cache invalidation traps, step-by-step, explaining the systems engineering behind each fix.

What You’ll Learn

• How the Docker daemon manipulates Linux namespaces and cgroups to construct isolated execution boundaries.
• How to structure Dockerfiles around deterministic layer-cache invalidation rules to slash build times from minutes to seconds.
• How to architect isolated user-defined bridge networks and leverage Docker’s internal embedded DNS engine.
• How storage engines interact with Linux bind mounts and Docker named volumes to guarantee data persistence.
• How to eliminate race conditions during stack startup using declarative health checks and synchronization barriers.
• How to shrink container image footprints by over 80% with multi-stage builds and drop runtime privileges to non-root users.
• How to orchestrate across nodes using Docker Swarm, overlay networks, and cryptographically secure, memory-only Docker Secrets.

End-to-End System Architecture

Prerequisites & Environment Setup

Before issuing build instructions, ensure your host environment contains the required container toolchains.

Host Requirements

• Operating System: Linux (Ubuntu 22.04+ or Debian 12 recommended), macOS 13+, or Windows 11 with WSL2 enabled.
• Docker Engine: Version 24.0.0 or higher.
• Docker Compose: Version v2.20.0 or higher (Compose V2 plugin).
• Core Knowledge: Basic familiarity with terminal execution and standard web protocols (HTTP, TCP).

Verification Check

Execute this diagnostic command in your terminal to verify engine readiness:

docker info --format 'Engine Version: {{.ServerVersion}} | OS: {{.OperatingSystem}} | Architecture: {{.Architecture}}'

Expected output:

Engine Version: 24.0.7 | OS: Ubuntu 22.04.3 LTS | Architecture: x86_64

Tip: If the command returns permission denied while trying to connect to the Docker daemon socket, your user is missing from the local socket group. Run sudo usermod -aG docker $USER, log out, and log back in.

Repo/Project Setup

First, clone the project repository and move into the project directory:

git clone <YOUR_GITHUB_REPOSITORY_URL>
cd <YOUR_REPOSITORY_NAME>

Note: Replace <YOUR_GITHUB_REPOSITORY_URL> and <YOUR_REPOSITORY_NAME> with your actual GitHub repository URL and directory name.

You can verify that the project files are present with:

ls

Project Repository Blueprint Preview

We will construct and maintain this directory tree across our implementation phases:

voting-app-infrastructure/
├── .env                    # Externalized environment secrets & port variables
├── compose.yaml            # Declarative multi-container orchestration spec
├── docker-stack.yml        # Production Swarm cluster deployment spec
├── vote/
│   ├── app.py              # Front-end Flask web server (binds to 0.0.0.0:80)
│   ├── Dockerfile          # Python runtime container definition
│   ├── requirements.txt    # Frozen Python dependencies (Flask, Redis, gunicorn)
│   ├── static/             # Static CSS and client logic
│   └── templates/          # Jinja2 frontend view templates
├── result/
│   ├── Dockerfile          # Node.js runtime container definition
│   ├── package.json        # Node.js dependencies and run scripts
│   ├── package-lock.json   # Deterministic dependency tree lockfile
│   ├── server.js           # Express + Socket.IO server (binds to 0.0.0.0:4000)
│   └── views/              # Front-end dashboard templates
└── worker/
    ├── Dockerfile          # .NET Core compilation pipeline definition
    ├── Program.cs          # Asynchronous queue consumer and schema generator
    └── Worker.csproj       # Target runtime manifest (.NET 7.0)

Remove Existing Docker Resources

f you have already experimented with this project, it is a good idea to remove the existing containers, images, networks, and volumes so that we can start from a clean Docker environment.

Stop and remove all containers

docker stop $(docker ps -aq) 2>/dev/null || true
docker rm -f $(docker ps -aq) 2>/dev/null || true

Remove all unused Docker images

docker rmi -f $(docker images -aq) 2>/dev/null || true

Remove unused Docker networks

docker network prune -f

Remove unused Docker volumes

docker volume prune -f

Clean up everything unused

Alternatively, you can perform the cleanup with a single command:

docker system prune -a --volumes -f

Warning: These commands remove Docker resources from your machine. In particular, docker system prune -a --volumes can delete unused images, containers, networks, and volumes from other projects as well. Only run it if you are comfortable starting with a clean Docker environment.

The "Why": The Case for Modern Containerization

Monolithic local development environments crumble under the weight of divergent runtime requirements. Running Node.js 20, Python 3.11, and .NET Core 7 side-by-side directly on a developer workstation introduces system package collisions, disparate runtime version dependencies, conflicting global environment variables, and OS-specific socket abstractions.

The Paradigm Shift: Bare-Metal vs. Containerized

Stage-by-Stage Tutorial

Stage 1: Raw Applications & Layer Cache Engineering

• Core Milestone: Containerize standalone services and optimize image build layer caching.
• Time Estimate: 20 Minutes

Concepts First

A Docker Image is an immutable, read-only stack of tarballs known as filesystem layers. Each layer represents the difference (diff) introduced by a single line in your Dockerfile.

A Container is an active runtime instance of an image. When you spin one up, the Docker storage driver (such as OverlayFS) stacks all read-only image layers together and adds a thin, transient Read-Write (R/W) layer on top.

+-------------------------------------------+
| Container Writable Layer (Read/Write)     | <-- Deleted on container termination
+-------------------------------------------+
| Layer 4: CMD ["python", "app.py"]         | \
+-------------------------------------------+  \
| Layer 3: COPY . .                         |   | Read-Only Image Layers
+-------------------------------------------+   | (Shared safely across containers)
| Layer 2: RUN pip install -r reqs.txt      |  /
+-------------------------------------------+ /
| Layer 1: FROM python:3.11                 |
+-------------------------------------------+

The Mechanics of Layer Caching and Cache Invalidation

Key Takeaway: Always place slowly-changing dependencies (package installations, system libraries) near the top of your Dockerfile, and rapidly-changing code (application logic, templates) near the bottom.

The Build (Hands-On)

# vote/Dockerfile (ANTI-PATTERN: Naive Implementation)
FROM python:3.11
WORKDIR /app

# ANTI-PATTERN: Copying the whole directory invalidates the cache on every code change!
COPY . .

# Inefficient: Runs on EVERY source code modification
RUN pip install -r requirements.txt

EXPOSE 80
CMD ["python", "app.py"]

docker build -t vote-app:naive .

First, let's build the image and note the build time as our baseline:

docker build -t vote-app:naive .

Note the time taken. We’ll compare it with the optimized Dockerfile later.

Now, let's fix this using an optimized, cache-friendly layer order.

Create or update vote/Dockerfile with these explicit cache boundaries:

# vote/Dockerfile
# Base image providing the official Python 3.11 runtime environment
FROM python:3.11

# Set the active working directory inside the container namespace
WORKDIR /app

# Step 1: Isolate the dependency manifest to leverage build cache
# This layer invalidates ONLY when dependencies change
COPY requirements.txt .

# Step 2: Install dependencies into the image layer
# This step is cached as long as requirements.txt stays identical
RUN pip install --no-cache-dir -r requirements.txt

# Step 3: Copy volatile source code (changes frequently)
# Placing this AFTER dependency installation keeps build times fast
COPY . .

# Document that the application listens on port 80 inside the container
EXPOSE 80

# Define the container's entry process
CMD ["python", "app.py"]

Next, write the Dockerfile for the Node.js results service using the same cache-friendly structure:

# result/Dockerfile
# Base image providing the official Node.js 20 LTS runtime environment
FROM node:20

# Create and switch to the target application directory
WORKDIR /app

# Step 1: Copy both package manifests (lockfile guarantees exact dependencies)
COPY package*.json ./

# Step 2: Install dependencies; cached unless package manifests change
RUN npm install

# Step 3: Copy application source code and template views
COPY . .

# Document that the Express server binds internally to port 4000
EXPOSE 4000

# Launch the Node.js application process
CMD ["node", "server.js"]

Step 3: Build the Images and Inspect the Cache Mechanics

Run the build commands from your project root:

# Build the Python vote image with tag v1
docker build -t vote-app:v1 ./vote

Expected output:

[+] Building 18.4s (10/10) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 450B
 => [internal] load .dockerignore
 => [1/5] FROM docker.io/library/python:3.11
 => [2/5] WORKDIR /app
 => [3/5] COPY requirements.txt .
 => [4/5] RUN pip install --no-cache-dir -r requirements.txt
 => [5/5] COPY . .
 => exporting to image
 => => naming to docker.io/library/vote-app:v1

Now, simulate a quick code change to see the cache in action:

# Touch a source file to update its modification time
touch vote/app.py

# Rebuild the image
docker build -t vote-app:v2 ./vote

Expected output:

[+] Building 0.4s (10/10) FINISHED
 => CACHED [2/5] WORKDIR /app
 => CACHED [3/5] COPY requirements.txt .
 => CACHED [4/5] RUN pip install --no-cache-dir -r requirements.txt
 => [5/5] COPY . .                                                         0.1s
 => exporting to image

Now, build the Node.js results service:

docker build -t result-app:v1 ./result

Step 4: Run the Standalone Container and Verify Host Port Mapping

docker run -d -p 5000:80 --name vote-test vote-app:v1

Check the running processes:

docker ps --filter "name=vote-test"

Expected output:

CONTAINER ID   IMAGE         COMMAND           STATUS         PORTS                  NAMES
f92a10b48c12   vote-app:v1   "python app.py"   Up 4 seconds   0.0.0.0:5000->80/tcp   vote-test

Debugging & Common Pitfalls

Common Mistake: Confusing EXPOSE with Port Publishing

• The Symptom: You added EXPOSE 80 in your Dockerfile, but visiting http://localhost:80 or http://localhost:5000 in your browser fails with ERR_CONNECTION_REFUSED.
• The Root Cause: EXPOSE is purely documentation. It does not alter your host machine's firewall or expose ports to your network. Containers get their own private network interfaces, which are isolated from the host by default.
• The Fix: You must explicitly publish ports using the p (or -publish) flag on the CLI:Bash

  docker run -p <HOST_PORT>:<CONTAINER_PORT> <IMAGE_NAME>
  

  This command instructs the daemon to set up an internal docker-proxy process and configure host iptables rules to route external traffic into the container's network namespace.

  Verification

  Test the running container by querying the published endpoint using curl

  curl -I http://localhost:5000
  

Expected output:

HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3128
Server: Werkzeug/2.3.7 Python/3.11.0

Clean up the test container:

docker rm -f vote-test

What We Learned

• Docker builds filesystems layer-by-layer; any changed layer breaks the cache for all steps below it.
• Separating your dependency manifests (requirements.txt, package.json) from your source code drastically speeds up daily development builds.
• Port binding requires an explicit runtime mapping (p HOST:CONTAINER); EXPOSE inside a Dockerfile is purely informational.

If you only remember one thing:
Order your Dockerfile instructions from least frequently changed to most frequently changed. Never place your source code COPY before your package manager's install command.

Good point to commit:

git add vote/Dockerfile result/Dockerfile
git commit -m "stage-1: implement layer-cache-optimized dockerfiles for vote and result"

Stage 2: Custom Networking & Storage Persistence

• Core Milestone: Configure inter-container DNS resolution, run multi-runtime services, and persist database storage.
• Time Estimate: 25 Minutes

Concepts First

Default Bridge vs. User-Defined Bridge Networks

[ Default Bridge: docker0 ]
Container A (172.17.0.2)  --- PING "redis" --->  FAILED: Unknown Host
(Requires manual, brittle IP tracking)

[ User-Defined Bridge: voting-net ]
Container A (172.18.0.3)  --- DNS Query: "redis" ---> [ Embedded DNS (127.0.0.11) ]
                                                                |
                          <-- Resolves IP: 172.18.0.2 ----------+
Container A (172.18.0.3)  ==========================> Container B (172.18.0.2

Data Persistence: Container Layers vs. Named Volumes

The Build (Hands-On)

Step 1: Create an Isolated User-Defined Bridge Network

Create an isolated bridge network for our microservices:

docker network create --driver bridge voting-net

Inspect the network to see its allocated subnet:

docker network inspect voting-net --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'

Expected output:

172.18.0.0/16

Step 2: Start the Message Broker (Redis)

Run the official Redis image, attaching it to our new network:

docker run -d \
  --name redis \
  --network voting-net \
  redis:alpine

Step 3: Run the Web App and Confirm DNS Discovery

docker run -d \
  -p 5000:80 \
  --name vote-app \
  --network voting-net \
  vote-app:v1

Now, cast a test vote to make sure the app can talk to Redis over the network:

curl -s -d "vote=a" -X POST http://localhost:5000 > /dev/null

docker exec -it redis redis-cli lrange votes 0 -1

Expected output:

1) "{\"voter_id\": \"b93b04c8f188\", \"vote\": \"a\"}"

Step 4: Write the Worker Dockerfile (.NET Core)

Our background worker is a C# .NET Core service that pops votes from Redis and saves them into PostgreSQL.

Because .NET is a compiled language, we need the .NET Software Development Kit (SDK) to compile the code into Intermediate Language (IL) assemblies, and the Common Language Runtime (CLR) to run it.

Create worker/Dockerfile:

# worker/Dockerfile
# Use the official .NET 7 SDK image for compilation
FROM mcr.microsoft.com/dotnet/sdk:7.0

# Set working directory inside container
WORKDIR /app

# Step 1: Copy the project manifest first to cache dependency restoration
COPY Worker.csproj .

# Step 2: Restore project dependencies via NuGet
RUN dotnet restore

# Step 3: Copy remaining source code files
COPY . .

# Step 4: Compile optimized release binaries into the /out directory
RUN dotnet publish -c Release -o /out

# Step 5: Execute the compiled IL assembly using the .NET runtime
CMD ["dotnet", "/out/Worker.dll"]

Build the worker image:

docker build -t worker-app:v1 ./worker

Step 5: Create a Named Volume and Launch PostgreSQL

Create a managed Docker volume to store our database files:

docker volume create db-data

docker run -d \
  --name db \
  --network voting-net \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -v db-data:/var/lib/postgresql/data \
  postgres:15-alpine

Step 6: Start the Worker and Result Dashboard

Run the background worker to start processing votes:

docker run -d \
  --name worker-app \
  --network voting-net \
  worker-app:v1

docker run -d \
  -p 5001:4000 \
  --name result-app \
  --network voting-net \
  result-app:v1

Debugging & Common Pitfalls

Common Mistake: .NET SDK and Runtime Version Mismatch The Exact Error:

You must install or update .NET to run this application.
App: /out/Worker.dll
Architecture: x64
Framework: 'Microsoft.NETCore.App', version '7.0.0' (x64)
.NET location: /usr/share/dotnet/
The following frameworks were found:
  8.0.30 at [/usr/share/dotnet/shared/Microsoft.NETCore.App]

The Fix: Always align your Docker base image tags directly with the framework version specified in your project files:

• The Exact Error:

  OCI runtime exec failed: exec failed: unable to start container process: exec: "ping": executable file not found in $PATH
  

• The Root Cause: Lightweight, production-focused base images strip out non-essential utilities like iputils-ping to keep image sizes small and minimize security risks.

• The Fix: Instead of installing troubleshooting tools into your production containers, use the built-in language runtimes to test network connectivity:Bash

  # Test DNS and network routing directly using Python's socket library
  docker exec -it vote-app python -c "import socket; print(socket.gethostbyname('redis'))"
  

Verification

Testing Database Persistence

Let's verify our database persistence by running a destructive test:

1. View the votes stored in PostgreSQL:BashPlaintext

   docker exec -it db psql -U postgres -d postgres -c "SELECT * FROM votes;"
   

   Expected output:

   id |      voter_id      | vote
   ----+--------------------+------
     1 | b93b04c8f188       | a
   (1 row)
   

2. The Destruction Test: Delete the database container entirely:Bash

   docker rm -f db
   

3. Spin up a new PostgreSQL container using the exact same named volume:Bash

   docker run -d \
     --name db \
     --network voting-net \
     -e POSTGRES_USER=postgres \
     -e POSTGRES_PASSWORD=postgres \
     -v db-data:/var/lib/postgresql/data \
     postgres:15-alpine
   

4. Query the database in the new container:Bash

   docker exec -it db psql -U postgres -d postgres -c "SELECT * FROM votes;"
   

   The data is still there. Because PostgreSQL's files were stored inside the db-data volume rather than the container's ephemeral writable layer, the new container picks up right where the old one left off.

What We Learned

• User-defined bridge networks provide an automatic embedded DNS server (127.0.0.11) that lets containers resolve each other by name.
• Ports only need to be published to the host (p) if they need to receive traffic from outside the host; containers on the same network can communicate over their internal ports without publishing them.
• Docker named volumes bypass the container's copy-on-write storage driver, writing data directly to host storage to protect it across container lifecycles.

🎯 If you only remember one thing:
Containers are disposable execution environments. Store all persistent data inside managed Named Volumes, and connect your containers using User-Defined Networks rather than the default bridge.

Good point to commit:

Bash

git add worker/Dockerfile
git commit -m "stage-2: configure user networks, persistence volumes, and dotnet worker"

Stage 3: Declarative Orchestration via Docker Compose

• Core Milestone: Replace manual CLI commands with declarative YAML, run healthchecks, and manage configuration via environment variables.
• Time Estimate: 20 Minutes

Concepts First

Imperative vs. Declarative Infrastructure Management

So far, we have built and managed our containers using imperative commands:

docker network create ...
docker volume create ...
docker run -d --name redis ...
docker run -d --name db ...

This approach has significant drawbacks: it is manual, error-prone, hard to document, and difficult to reproduce across different developer machines.

[ compose.yaml Definition ]
       |
       | Reads Desired State
       v
[ Docker Compose Engine ]
       |
       +---> Checks active system state
       +---> Identifies missing bridges, volumes, and containers
       +---> Applies changes in dependency order
       v
[ Running Multi-Container Stack ]

Startup Sequencing: Process Alive vs. Application Ready

T = 0.0s: 'docker compose up' starts containers.
T = 0.5s: PostgreSQL container spawns (PID 1 is active).
          CRITICAL GAP: Standard 'depends_on' sees PID 1 and starts the worker!
T = 1.0s: Worker tries to connect to db:5432 -> CONNECTION REFUSED (Crash).
T = 1.5s: PostgreSQL finishes replaying logs and opens its TCP listening socket.
T = 3.0s: Healthcheck triggers: "pg_isready" -> Exits with code 0 (SUCCESS).

The Build (Hands-On)

Step 1: Clean Up Imperative Containers

Before moving to Docker Compose, remove the containers we created manually:

docker rm -f vote-app result-app worker-app redis db

# .env - Runtime Configuration & Secrets

# Database Authentication
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres_secure_pass_2026

# Host Public Port Assignments
HOST_PORT_VOTE=5000
HOST_PORT_RESULT=5001

# compose.yaml - Declarative Stack Orchestrator Specification
services:
  # Python / Flask Web UI
  vote:
    build: ./vote
    ports:
      - "${HOST_PORT_VOTE}:80"
    restart: unless-stopped
    depends_on:
      redis:
        condition: service_healthy

  # Node.js / Express Dashboard
  result:
    build: ./result
    ports:
      - "${HOST_PORT_RESULT}:4000"
    restart: unless-stopped
    environment:
      DATABASE_URL: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/postgres"
    depends_on:
      db:
        condition: service_healthy

  # .NET Core Background Data Consumer
  worker:
    build: ./worker
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy

  # Redis In-Memory Message Broker
  redis:
    image: redis:alpine
    restart: unless-stopped
    healthcheck:
      # Pings the Redis engine using the CLI tool
      test: ["CMD", "redis-cli", "ping"]
      interval: 3s
      timeout: 3s
      retries: 5

  # PostgreSQL Relational Storage Engine
  db:
    image: postgres:15-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      # Validates database socket availability
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
      interval: 3s
      timeout: 3s
      retries: 5

# Named Persistent Storage Volumes
volumes:
  db-data:

Step 4: Validate and Launch the Stack

Validate your Compose file and verify that environment variables are interpolated correctly:

docker compose config

Now, launch the entire application stack in the background:

docker compose up -d

Expected output:

[+] Running 6/6
 ✔ Network voting-app-infrastructure_default  Created  0.1s
 ✔ Volume "voting-app-infrastructure_db-data" Created  0.0s
 ✔ Container voting-app-infrastructure-redis-1  Healthy  3.2s
 ✔ Container voting-app-infrastructure-db-1     Healthy  3.2s
 ✔ Container voting-app-infrastructure-vote-1   Started  3.3s
 ✔ Container voting-app-infrastructure-worker-1 Started  3.3s
 ✔ Container voting-app-infrastructure-result-1 Started  3.4s

Debugging & Common Pitfalls

⚠️ Common Mistake: Race Conditions on Downstream Readers

• The Exact Error:

  Error performing query: error: relation "votes" does not exist
  

• The Root Cause: PostgreSQL started up and passed its health check, but the worker service had not yet finished processing incoming votes or creating the votes table before result-app executed SELECT vote, count(id) FROM votes.

• The Fix: Handle initialization sequences gracefully in your microservice code. In your applications, write retry logic or catch table-missing exceptions during startup. If you encounter this error during development, restarting the consumer service clears the issue once the database schema is created:

  docker compose restart result
  

Verification

Check the status and health of all running services:

docker compose ps

Expected output:

NAME                                     STATUS                    PORTS
voting-app-infrastructure-db-1           Up 2 minutes (healthy)    5432/tcp
voting-app-infrastructure-redis-1        Up 2 minutes (healthy)    6379/tcp
voting-app-infrastructure-result-1       Up 2 minutes              0.0.0.0:5001->4000/tcp
voting-app-infrastructure-vote-1         Up 2 minutes              0.0.0.0:5000->80/tcp
voting-app-infrastructure-worker-1       Up 2 minutes

What We Learned

• Docker Compose provides a declarative way to manage your containers, storage volumes, and networks together.
• Standard depends_on only waits for a container process to spawn; pairing it with condition: service_healthy ensures dependent services wait until upstream applications are actually ready to accept traffic.
• Using .env files lets you manage configuration and sensitive values without hardcoding credentials into your repository files.

🎯 If you only remember one thing:
A container that is Up is not necessarily ready for traffic. Always use explicit healthchecks for databases and message brokers to prevent race conditions during startup.

Good point to commit:

git add compose.yaml .env
git commit -m "stage-3: introduce declarative compose with healthchecks and env configuration"

Stage 4: Hardening, Multi-Stage Builds & Least Privilege

• Core Milestone: Shrink image footprints using multi-stage builds, run containers as non-root users, and set cgroup resource limits.
• Time Estimate: 25 Minutes

Concepts First

Multi-Stage Builds: Leaving Build Tools Behind

In traditional single-stage Dockerfiles, your final container image often ends up packaging compilers, build tools, package caches, and SDKs. These extra tools balloon your image size and needlessly widen your security attack surface.

[ Stage 1: Build Environment ]
FROM mcr.microsoft.com/dotnet/sdk:7.0 AS builder
- Full .NET SDK (~850 MB)
- Restores dependencies, compiles code, runs linters
- Outputs: /app/publish/Worker.dll

                |
                | COPY --from=builder /app/publish /app
                v

[ Stage 2: Hardened Runtime ]
FROM mcr.microsoft.com/dotnet/runtime:7.0-alpine
- Minimal .NET Runtime (~138 MB)
- No compilers, no SDKs, reduced attack surface

The Principle of Least Privilege: Running as Non-Root

Resource Management: Cgroups and OOM Prevention

Without resource boundaries, a memory leak or sudden spike in traffic can allow a single container to consume all available memory on your host machine. This can trigger the Linux kernel's Out-Of-Memory (OOM) Killer, which may randomly shut down critical system processes.

By setting explicit limits in Compose using Linux Control Groups (cgroups), you cap the maximum CPU and RAM a container is allowed to use.

The Build (Hands-On)

Update worker/Dockerfile to use a two-stage build that swaps the heavy SDK for a minimal Alpine runtime:

# worker/Dockerfile
# ==========================================
# Stage 1: Build & Compilation Environment
# ==========================================
FROM mcr.microsoft.com/dotnet/sdk:7.0 AS builder
WORKDIR /src

# Step 1: Copy manifest and restore dependencies
COPY Worker.csproj .
RUN dotnet restore

# Step 2: Copy remaining source code and publish release binary
COPY . .
RUN dotnet publish -c Release -o /app/publish

# ==========================================
# Stage 2: Minimal Production Runtime
# ==========================================
FROM mcr.microsoft.com/dotnet/runtime:7.0-alpine
WORKDIR /app

# Copy ONLY compiled artifacts from the builder stage
COPY --from=builder /app/publish .

# Run as a lightweight compiled binary
CMD ["dotnet", "Worker.dll"]

# vote/Dockerfile
# ==========================================
# Stage 1: Dependencies Compilation Stage
# ==========================================
FROM python:3.11-slim AS builder
WORKDIR /app

# Install dependencies into a local directory using --user
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# ==========================================
# Stage 2: Hardened Runtime Stage
# ==========================================
FROM python:3.11-slim
WORKDIR /app

# Create an unprivileged system group and user
RUN groupadd -r appgroup && useradd -r -g appgroup appuser

# Copy installed libraries into the non-root user's home directory
COPY --from=builder /root/.local /home/appuser/.local
COPY --chown=appuser:appgroup . .

# Add the local user binaries to PATH and set unbuffered output for logging
ENV PATH=/home/appuser/.local/bin:$PATH \
    PYTHONUNBUFFERED=1

# Drop execution privileges from root to appuser
USER appuser

EXPOSE 80
CMD ["python", "app.py"]

# result/Dockerfile
# ==========================================
# Stage 1: Dependency Installation Stage
# ==========================================
FROM node:20-alpine AS builder
WORKDIR /app

COPY package*.json ./

# Install only production dependencies and clean cache
RUN npm ci --only=production && npm cache clean --force

# ==========================================
# Stage 2: Hardened Runtime Stage
# ==========================================
FROM node:20-alpine
WORKDIR /app

# Use the non-root 'node' user provided by the base Alpine image
COPY --chown=node:node --from=builder /app/node_modules ./node_modules
COPY --chown=node:node . .

# Drop privileges
USER node

EXPOSE 4000
CMD ["node", "server.js"]

deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 256M
        reservations:
          memory: 64M

Apply the updated configuration and rebuild your images:

docker compose up -d --build

Debugging & Common Pitfalls

⚠️ Common Mistake: Permission Denied When Running as Non-Root

• The Exact Error:

  Error: EACCES: permission denied, open '/app/server.js'
  

• The Root Cause: Files copied into an image via COPY default to root:root ownership. If you switch to an unprivileged user using USER node without updating file ownership, the application will not have permission to read or execute those files.
• The Fix: Always use the -chown flag when copying files into your final image stage to grant ownership to your unprivileged user:

  COPY --chown=node:node . .
  

Verification

Image Footprint Reductions

Let's see how much storage our multi-stage builds saved:

Verifying Non-Root Execution

docker compose exec vote whoami
docker compose exec result whoami

Expected output:

appuser
node

Inspecting Resource Limits (Cgroups)

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

Expected output:

NAME                                 CPU %     MEM USAGE / LIMIT     MEM %
voting-app-infrastructure-vote-1     0.04%     48.2MiB / 256MiB      18.83%
voting-app-infrastructure-result-1   0.08%     39.1MiB / 256MiB      15.27%
voting-app-infrastructure-worker-1   0.12%     28.4MiB / 256MiB      11.09%

What We Learned

• Multi-stage builds keep your production images lean by excluding heavy compilers, package managers, and SDKs.
• Running containers as non-root users (USER appuser) protects your host system by limiting privileges inside the container.
• Setting cgroup resource limits prevents runaway processes or memory leaks from destabilizing your host machine.

🎯 If you only remember one thing:
Compilers and SDKs belong in build stages, not in production images. Strip out build tools using multi-stage builds, and always drop privileges to an unprivileged user.

Good point to commit:

git add vote/Dockerfile result/Dockerfile worker/Dockerfile compose.yaml
git commit -m "stage-4: harden images with multi-stage builds, non-root users, and cgroups"

Stage 5: Swarm Clustering, Ingress Routing & In-Memory Secrets

• Core Milestone: Deploy across a multi-node cluster, configure an overlay network mesh, and secure credentials using Docker Secrets.
• Time Estimate: 25 Minutes

Concepts First

Moving Beyond Single-Host Deployments

Docker Compose is great for local development, but it is limited to a single host machine. If that host goes down, your entire stack goes with it.

Docker Swarm provides built-in clustering and orchestration for multiple Docker engines. In Swarm, individual machines become Nodes within a shared cluster. Nodes are organized into two roles:

• Manager Nodes: Handle cluster state, manage scheduling decisions, and run the Raft consensus algorithm.
• Worker Nodes: Execute the container workloads (known as Tasks) assigned to them by managers.

+-------------------------+
                  |   Swarm Manager Node    |
                  |  (Schedules & Tracks)   |
                  +-------------------------+
                               |
          +--------------------+--------------------+
          | Overlay Network (VXLAN Encapsulated)    |
          v                                         v
+-----------------------+                 +-----------------------+
|     Worker Node 1     |                 |     Worker Node 2     |
|  [vote-app replica 1] |                 |  [vote-app replica 2] |
+-----------------------+                 +-----------------------+

Overlay Networks and the Ingress Routing Mesh

Swarm uses Overlay Networks to let containers on different physical machines communicate seamlessly. It accomplishes this by encapsulating Layer-2 traffic inside host Layer-3 network packets using VXLAN tunnels.

Swarm also provides an Ingress Routing Mesh. When you expose a port (like port 5000), that port is opened on every node across the entire cluster. When traffic hits any node, the routing mesh intercepts the request and automatically routes it over the overlay network to a node currently hosting a healthy container replica.

Managing Sensitive Data with Docker Secrets

The Build (Hands-On)

Step 1: Initialize the Swarm Cluster

Initialize Swarm mode on your current node:

docker swarm init

Expected output:

Swarm initialized: current node (i7z4f8kd3a) is now a manager.
To add a worker to this swarm, run the following command:
    docker swarm join --token SWMTKN-1-4... 192.168.1.50:2377

Check the nodes in your cluster:

docker node ls

Step 2: Create an Encrypted Swarm Secret

Create a secret to store our database password securely:

echo "super_secret_swarm_password_2026" | docker secret create db_password -

Confirm that the secret is stored in the cluster:

docker secret ls

Expected output:

ID                          NAME          DRIVER    CREATED          UPDATED
vy81y48h7f65fegx3o1v6z9qm   db_password             12 seconds ago   12 seconds ago

# docker-stack.yml - Multi-Node Swarm Production Orchestration
version: "3.8"

services:
  # Front-end vote service, scaled across nodes
  vote:
    image: vote-app:v1
    ports:
      - "5000:80"
    networks:
      - voting-overlay
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
      resources:
        limits:
          cpus: '0.50'
          memory: 256M

  # Results dashboard
  result:
    image: result-app:v1
    ports:
      - "5001:4000"
    networks:
      - voting-overlay
    environment:
      DATABASE_URL: "postgres://postgres:super_secret_swarm_password_2026@db/postgres"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  # Background worker
  worker:
    image: worker-app:v1
    networks:
      - voting-overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  # Redis broker
  redis:
    image: redis:alpine
    networks:
      - voting-overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  # Database with Docker Secret mounted
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: postgres
      # Instructs the Postgres init script to read the password from the mounted secret file
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - voting-overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

# Software-Defined Multi-Host Overlay Network
networks:
  voting-overlay:
    driver: overlay

# Managed Volumes
volumes:
  db-data:

# In-Memory Encrypted Cluster Secrets
secrets:
  db_password:
    external: true

Step 4: Deploy the Swarm Stack

Deploy the stack to your Swarm cluster:

docker stack deploy -c docker-stack.yml voting_app

Expected output:

Creating network voting_app_voting-overlay
Creating service voting_app_redis
Creating service voting_app_db
Creating service voting_app_vote
Creating service voting_app_result
Creating service voting_app_worker

Step 5: Scale Replicas Dynamically

Swarm makes it easy to scale stateless services up or down on demand:

docker service scale voting_app_vote=5

Expected output:

voting_app_vote scaled to 5
overall progress: 5 out of 5 tasks
1/5: running   [=================================>]
2/5: running   [=================================>]
3/5: running   [=================================>]
4/5: running   [=================================>]
5/5: running   [=================================>]
verify: Service converged

Debugging & Common Pitfalls

⚠️ Common Mistake: Mounting Secrets in Standard Compose Mode

• The Symptom: You added a secrets: configuration block to a standard docker compose up setup and were greeted by an error or noticed secrets weren't behaving securely.
• The Root Cause: Full encrypted Docker Secrets backed by Raft storage are a feature of Docker Swarm. In standard Compose, Docker emulates secrets by bind-mounting files directly from the host filesystem, which does not provide the same security guarantees.
• The Fix: Use Swarm mode (docker stack deploy) for true, memory-only encrypted secrets. If you are developing locally with standard Compose, use an .env file instead.

Verification

Inspecting Running Stack Services

View your active services and their replica counts:

docker stack services voting_app

Expected output:

ID             NAME                MODE         REPLICAS   IMAGE             PORTS
y24r8t0a0p     voting_app_db       replicated   1/1        postgres:15-alpine
l0p1v8e6f3     voting_app_redis    replicated   1/1        redis:alpine
a1b2c3d4e5     voting_app_result   replicated   1/1        result-app:v1     *:5001->4000/tcp
k9j8h7g6f5     voting_app_vote     replicated   5/5        vote-app:v1       *:5000->80/tcp
m4n3b2v1c0     voting_app_worker   replicated   1/1        worker-app:v1

Verifying the In-Memory Secret Mount

DB_CONTAINER=$(docker ps -q --filter "name=voting_app_db")
docker exec -it $DB_CONTAINER cat /run/secrets/db_password

Expected output:

super_secret_swarm_password_2026

What We Learned

• Docker Swarm groups multiple engines into an orchestrated cluster with Manager and Worker roles.
• Overlay networks and the Ingress Routing Mesh route incoming traffic across nodes to healthy container replicas automatically.
• Docker Secrets encrypt sensitive values and mount them into memory inside containers (/run/secrets/), avoiding the security risks of environment variables.

🎯 If you only remember one thing:
Don't pass sensitive production passwords as plaintext environment variables. Use Docker Secrets to mount them into containers via secure, memory-only filesystems.

Good point to commit:

git add docker-stack.yml
git commit -m "stage-5: deploy resilient multi-host swarm stack with docker secrets"

The Complete Production Project Artifacts

Here is the complete file structure and configuration files for the final project.

Full Project File Tree

voting-app-infrastructure/
├── .env
├── compose.yaml
├── docker-stack.yml
├── vote/
│   ├── app.py
│   ├── Dockerfile
│   └── requirements.txt
├── result/
│   ├── Dockerfile
│   ├── package.json
│   └── server.js
└── worker/
    ├── Dockerfile
    ├── Program.cs
    └── Worker.csproj

The Production Compose File (compose.yaml)

# compose.yaml
services:
  # Python / Flask Web UI
  vote:
    build: ./vote
    ports:
      - "${HOST_PORT_VOTE}:80"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 256M
        reservations:
          memory: 64M
    depends_on:
      redis:
        condition: service_healthy

  # Node.js / Express Dashboard
  result:
    build: ./result
    ports:
      - "${HOST_PORT_RESULT}:4000"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 256M
        reservations:
          memory: 64M
    environment:
      DATABASE_URL: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/postgres"
    depends_on:
      db:
        condition: service_healthy

  # .NET Core Background Data Consumer
  worker:
    build: ./worker
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 256M
        reservations:
          memory: 64M
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy

  # Redis In-Memory Message Broker
  redis:
    image: redis:alpine
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '0.25'
          memory: 128M
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 3s
      timeout: 3s
      retries: 5

  # PostgreSQL Relational Storage Engine
  db:
    image: postgres:15-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - db-data:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          cpus: '0.75'
          memory: 512M
        reservations:
          memory: 128M
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
      interval: 3s
      timeout: 3s
      retries: 5

# Named Persistent Storage Volumes
volumes:
  db-data:

Master Command Cheat Sheet

Field Debugging FAQ

Q1: I get ConnectionError: Error -2 connecting to redis:6379. Name or service not known

• The Issue: Your application cannot resolve the hostname redis.
• The Fix: Ensure your containers are attached to the same user-defined network. The default bridge network (docker0) does not support automatic name-based DNS resolution. Run docker network inspect <network_name> to verify that both containers are attached to the same network.

Q2: Why does pg_isready pass, but my downstream service still fails to connect to PostgreSQL?

• The Issue: A passing healthcheck confirms that PostgreSQL is accepting TCP connections, but your application-level database schema may not be created yet.
• The Fix: Ensure your database migration scripts or schema creation logic (in our case, handled by the worker service) have finished executing before downstream reader services query the database. Add reconnect or retry logic to your consumer code to handle startup initialization gracefully.

Q3: My build fails with .NET SDK does not provide the framework net7.0

• The Issue: Your Docker base image tag does not match the target framework specified in your project file.
• The Fix: Open your .csproj file, inspect the <TargetFramework> tag, and update your Dockerfile base image to match that exact version (e.g., FROM [mcr.microsoft.com/dotnet/sdk:7.0](https://mcr.microsoft.com/dotnet/sdk:7.0)).

Q4: My Node.js container throws Error: EACCES: permission denied on startup

• The Issue: You switched to a non-root user (USER node), but the application files copied into the image are owned by root.
• The Fix: Update your COPY commands to explicitly set file ownership to the unprivileged user using the -chown flag:Dockerfile

  COPY --chown=node:node . .
  

Q5: Docker Compose shows service "db" is unhealthy and blocks dependent services from starting

• The Issue: The command defined in your healthcheck: block failed more times than allowed by the retries: setting.
• The Fix: Inspect the detailed healthcheck error logs using docker inspect:Bash

  docker inspect --format='{{json .State.Health}}' <container_name> | jq
  

  Ensure any environment variables used by the healthcheck command (such as POSTGRES_USER) are defined properly in your .env file.

Resources & Further Reading

Official Documentation (For Reading)

• Docker Documentation: Official Docker Docs
• Dockerfile Best Practices: Dockerfile Reference & Best Practices
• Docker Compose V2: Docker Compose Specification
• Docker Swarm Mode: Docker Swarm Mode Overview
• Linux Namespaces: man7 Linux Namespaces Overview
• Linux Cgroups: man7 Linux Control Groups (cgroups) Reference
• Reference Application Codebase: Docker Samples: Example Voting App Repository

KodeKloud (For Hands-on Practice)

• Docker Training Course & Interactive Labs: KodeKloud Docker Training Course for Beginners

  (Includes browser-based terminal labs for practicing Docker commands, container networking, image creation, volumes, and Compose stacks without needing local installation.)

Authors Note

Containerization is ultimately about understanding the underlying operating system. Once you realize that a container is simply a standard Linux process wrapped in namespaces, governed by cgroups, and layered over an OverlayFS mount, the entire ecosystem becomes much easier to debug and reason about.

Build this project on your local machine, experiment with breaking things, test your own failure scenarios, and share your setup with your team. If you hit an unexpected issue along the way, walk through the debugging steps in this guide to inspect your container states and resolve it. Happy building!