⏱ 9 min read | ~1887 words
AI-Powered Serverless Image Processing Pipeline — Part 6: Containerizing with Docker & Implementing CI/CD
In the first five installments we set up the AI model (a lightweight PyTorch‑based super‑resolution net), wired it to AWS Lambda via aws‑lambda‑powertools, and connected the whole thing to an S3 trigger that stores the processed images in a separate bucket. Based on my technical understanding as a Lead Programmer Analyst, I’ll now show you how to make that solution reproducible, portable, and continuously deployable using Docker and modern CI/CD practices.
Why Docker Matters for a Serverless AI Workflow
- Reproducibility. A Docker image captures the exact OS, runtime, libraries, and model weights you tested locally.
- Speed. Layer caching (as highlighted by Octopus Deploy) means that only the parts of the image that actually change are rebuilt, shaving minutes off each pipeline run.
- Portability. Whether you run on AWS Lambda, Google Cloud Run, or an on‑premise Fargate cluster, the same image works everywhere.
- Security. By pinning base images and using multi‑stage builds you reduce the attack surface and keep your supply chain auditable.
In the context of a serverless AI pipeline, Docker also acts as the glue between the model training environment and the runtime environment. The AWS Prescriptive Guidance for CI/CD and automation in serverless AI stresses that a repeatable build‑test‑deploy loop is the only way to keep up with rapid model iteration without breaking downstream services.
Step 1: Organize the Project Layout
Before we write any Dockerfile, let’s adopt a clean directory structure that works well with both Docker and CI pipelines:
ai-image-pipeline/
├── src/
│ ├── handler.py # Lambda entry point
│ ├── model/
│ │ ├── super_resnet.pt # Pre‑trained weights (git‑ignored)
│ │ └── model.py # PyTorch model definition
│ └── utils/
│ └── image_ops.py
├── tests/
│ └── test_handler.py
├── Dockerfile
├── requirements.txt
├── .dockerignore
├── .github/
│ └── workflows/
│ └── ci-cd.yml # GitHub Actions pipeline
└── README.md
The .dockerignore file keeps large data files (like the model checkpoint) out of the build context unless we explicitly copy them, which speeds up the build and respects repository size limits.
Step 2: Write a Multi‑Stage Dockerfile
We’ll use a two‑stage build: the first stage compiles the Python dependencies in a slim python:3.11‑slim image, the second stage copies only the runtime artefacts. This mirrors the best‑practice patterns described in the Coursera project “Build a CI/CD Pipeline with Docker”.
# ---------- Stage 1: Build ----------
FROM python:3.11-slim AS builder
# Install OS‑level build tools (required for torch & numpy wheels)
RUN apt-get update && \
apt-get install -y --no-install-recommends gcc g++ make && \
rm -rf /var/lib/apt/lists/*
# Set a deterministic working directory
WORKDIR /app
# Install Python dependencies into a virtual environment
ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# Copy only the requirements first to leverage Docker cache
COPY requirements.txt .
RUN pip install --upgrade pip && \
pip install -r requirements.txt
# ---------- Stage 2: Runtime ----------
FROM python:3.11-slim AS runtime
# Create a non‑root user for security
RUN useradd --create-home appuser
WORKDIR /home/appuser
# Copy the virtual environment from the builder stage
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
# Copy source code (excluding .git, __pycache__, etc.)
COPY src/ ./src/
COPY tests/ ./tests/
COPY model/ ./model/
# Ensure the model checkpoint is present at runtime (you can also pull from S3)
# For demo purposes we assume the checkpoint is checked in .gitignore
# and will be added manually in CI before deployment.
# Set the Lambda handler entry point
ENV PYTHONPATH=/home/appuser/src
CMD ["handler.lambda_handler"]
Key points:
- Layer ordering. OS packages and the base image are installed first, then
requirements.txt. Because they change rarely, Docker can reuse the cached layer across builds. - Non‑root user. Security best‑practice (also recommended by AWS Lambda container images).
- Virtual environment. Keeps the runtime lightweight and isolates system packages.
Step 3: Verify the Image Locally
Run a quick sanity check before committing the Dockerfile to source control.
# Build the image
docker build -t ai-image-pipeline:dev .
# Run a container, mounting a test image from the host
docker run --rm \
-e AWS_REGION=us-east-1 \
-v $(pwd)/sample.jpg:/tmp/input.jpg \
ai-image-pipeline:dev \
python -c "
import src.handler as h;
print('Invoking lambda handler locally...');
event = {
'Records': [{
's3': {
'bucket': {'name': 'dummy-bucket'},
'object': {'key': 'sample.jpg'}
}
}]
}
context = type('obj', (object,), {'aws_request_id': 'local-test'})
h.lambda_handler(event, context)
"
If everything is wired correctly you should see the model loading (once) and the processed image saved to /tmp inside the container. This mirrors the local “Docker‑run” test that the Octopus Deploy guide recommends for early feedback.
Step 4: Choose a CI Platform – GitHub Actions
GitHub Actions provides a free, fully‑managed runner pool that integrates natively with the repository. For an enterprise setting you could swap to GitLab CI or Azure Pipelines without changing the core Docker steps.
Below is a complete .github/workflows/ci-cd.yml that implements the following stages:
- Checkout – pull the repository.
- Login to Amazon ECR – secure push destination.
- Build & Test – compile the Docker image, run unit tests inside the container, and push to ECR on success.
- Deploy – trigger an AWS CloudFormation stack update (or Serverless Framework) that references the new image tag.
name: CI/CD – Dockerized AI Image Pipeline
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
env:
AWS_REGION: us-east-1
ECR_REPOSITORY: 123456789012.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/ai-image-pipeline
IMAGE_TAG: ${{ github.sha }}
jobs:
build-test-deploy:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # For OIDC auth to ECR
steps:
# 1️⃣ Checkout source
- name: Checkout repository
uses: actions/checkout@v4
# 2️⃣ Set up Python (needed for unit tests)
- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: '3.11'
# 3️⃣ Cache Docker layers (speed up builds)
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-docker-${{ github.sha }}
restore-keys: |
${{ runner.os }}-docker-
# 4️⃣ Configure Docker BuildKit for caching
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# 5️⃣ Log in to Amazon ECR using OIDC (no static credentials)
- name: Authenticate to Amazon ECR
id: ecr-login
uses: aws-actions/amazon-ecr-login@v2
with:
mask-password: 'true'
# 6️⃣ Build the Docker image
- name: Build Docker image
run: |
docker build \
--cache-from type=local,src=/tmp/.buildx-cache \
--cache-to type=local,dest=/tmp/.buildx-cache,mode=max \
-t ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} \
.
# 7️⃣ Run unit tests inside the image
- name: Run tests
run: |
docker run --rm \
-e PYTHONPATH=/home/appuser/src \
${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} \
pytest /home/appuser/tests
# 8️⃣ Push image to ECR
- name: Push Docker image to ECR
run: |
docker push ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}
# 9️⃣ Deploy with AWS CloudFormation (or Serverless)
- name: Deploy CloudFormation stack
uses: aws-actions/aws-cloudformation-github-deploy@v1
with:
name: ai-image-pipeline-stack
template: infra/cloudformation.yml
parameter-overrides: |
ImageUri=${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}
no-fail-on-empty-changeset: true
Notice how the workflow mirrors the “repeatable workflow” concept from the Coursera project: the same Dockerfile is used for local dev, CI, and production, guaranteeing “code‑to‑deployment” fidelity.
Step 5: CloudFormation Template for Lambda Container Image
Below is a minimal infra/cloudformation.yml that creates a Lambda function using the image we just pushed. It also provisions the S3 buckets and the necessary IAM role. Feel free to replace this with a Serverless Framework serverless.yml if you prefer.
AWSTemplateFormatVersion: '2010-09-09'
Description: AI‑Powered Image Processing Lambda (container image)
Parameters:
ImageUri:
Type: String
Description: ECR image URI for the Lambda function
Resources:
ImageProcessingFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: ai-image-processor
PackageType: Image
Code:
ImageUri: !Ref ImageUri
MemorySize: 1024 # Adjust based on model size
Timeout: 30
Role: !GetAtt LambdaExecutionRole.Arn
Environment:
Variables:
LOG_LEVEL: INFO
MODEL_PATH: /home/appuser/model/super_resnet.pt
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: ai-image-lambda-exec
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
- arn:aws:iam::aws:policy/AmazonS3FullAccess # For writing processed images
SourceBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ai-image-source-bucket
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function: !GetAtt ImageProcessingFunction.Arn
Filter:
S3Key:
Rules:
- Name: suffix
Value: .jpg
DestinationBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ai-image-processed-bucket
Outputs:
LambdaArn:
Description: ARN of the image‑processing Lambda
Value: !GetAtt ImageProcessingFunction.Arn
When the CI/CD pipeline pushes a new image tag, the CloudFormation stack updates the ImageUri parameter, causing a seamless zero‑downtime rollout. This is precisely the “intelligent automation” that AI‑Powered DevOps talks about – the pipeline decides when to promote a new model version based on test results.
Step 6: Adding Model Versioning to the CI Process
In production you rarely want to overwrite the same model file. Instead, tag the model with a semantic version (e.g., v1.2.0) and store it in an S3 model/ prefix. The CI job can then:
- Download the latest model artifact from an S3 bucket.
- Validate the checksum against a stored hash.
- Inject the model into the Docker image using the
--build-arg MODEL_VERSION=...flag.
Here’s a snippet that adds a build‑arg to the Docker build step and copies the model at build time:
# In the builder stage, after installing dependencies
ARG MODEL_VERSION=latest
ENV MODEL_VERSION=${MODEL_VERSION}
RUN mkdir -p /model && \
curl -sSL "https://my-model-bucket.s3.amazonaws.com/${MODEL_VERSION}/super_resnet.pt" \
-o /model/super_resnet.pt && \
echo "Model ${MODEL_VERSION} downloaded"
COPY src/model/ /model/ # Fallback for local dev
Corresponding CI change:
# In the GitHub Actions build step
- name: Build Docker image with model version
run: |
MODEL_VER=${{ github.ref_name }} # Use branch name as version for demo
docker build \
--build-arg MODEL_VERSION=$MODEL_VER \
-t ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} .
With this pattern you can later add an automated model‑registry check that only promotes a new version if performance metrics (e.g., PSNR improvement > 1 dB) meet a threshold—exactly the kind of feedback loop described in the AI‑Powered DevOps article.
Step 7: Running Integration Tests on AWS (Optional)
Unit tests are great, but for a serverless AI pipeline you also want to verify the end‑to‑end flow:
- Upload a sample JPEG to the source bucket.
- Invoke the Lambda (or let S3 trigger it).
- Check that the processed image appears in the destination bucket and meets quality expectations.
GitHub Actions can spin up a temporary AWS environment using aws-actions/configure-aws-credentials and localstack (or the real AWS account with a dedicated testing role). Below is a minimal integration test script written in Python that you can call from the workflow.
import boto3, time, os
import uuid
s3 = boto3.client('s3', region_name=os.getenv('AWS_REGION'))
source_bucket = 'ai-image-source-bucket'
dest_bucket = 'ai-image-processed-bucket'
def upload_test_image():
key = f'test/{uuid.uuid4()}.jpg'
s3.upload_file('tests/fixtures/sample.jpg', source_bucket, key)
return key
def wait_for_result(key, timeout=30):
prefix = key.replace('test/', 'processed/')
for _ in range(timeout):
resp = s3.list_objects_v2(Bucket=dest_bucket, Prefix=prefix)
if 'Contents' in resp:
return resp['Contents'][0]['Key']
time.sleep(1)
raise TimeoutError('Processed image never appeared.')
def main():
test_key = upload_test_image()
print(f'Uploaded {test_key}')
result_key = wait_for_result(test_key)
print(f'Processed image stored at {result_key}')
# Optional: download and run a quick PSNR check
# ...
if __name__ == '__main__':
main()
Add this step to the workflow after the Docker push:
- name: Run end‑to‑end integration test
env:
AWS_REGION: ${{ env.AWS_REGION }}
run: |
pip install boto3
python tests/integration_test.py
Running the integration test against a real AWS account gives you confidence that the Docker image, the Lambda configuration, and the S3 event source are all wired correctly before you promote to production.
Step 8: Optimizing Docker Builds for Faster CI
Two tricks that have saved my teams minutes per build:
- Separate rarely‑changed layers. In the Dockerfile, place the
apt‑get installandpip install -r requirements.txtsteps before copying the source code. This way, changing a single line inhandler.pydoesn’t invalidate the whole dependency layer. - Leverage BuildKit’s cache‑to/cache‑from. The GitHub Actions YAML already uses
--cache-fromand--cache-towith a local directory cache. For even more speed you❓ Frequently Asked Questions
Why should I containerize a serverless AI image processing function with Docker?
Docker packages the OS, runtime, dependencies, and model weights into a single image, ensuring the Lambda environment matches your local tests, eliminates “works on my machine” issues, and speeds up deployments through layer caching.
Can I run a Docker‑based image processing pipeline on AWS Lambda?
Yes. AWS Lambda supports container images up to 10 GB. Build a Docker image with your PyTorch model and use the Lambda Runtime Interface Client (RIC) to invoke the function just like a zip package.
What CI/CD tools work best for automating Docker builds for this project?
GitHub Actions, GitLab CI, and AWS CodeBuild all integrate with ECR. They can build the Docker image, run tests, push to Amazon ECR, and trigger a Lambda update in a single pipeline.
How do I keep the Docker image size small for faster Lambda cold starts?
Use a minimal base image (e.g., python:3.11‑slim or amazonlinux), install only required libraries, copy only necessary model files, and leverage multi‑stage builds to discard build‑time dependencies.
🔗 You Might Also Like
📺 Recommended Video
Watch this video for a practical overview of the topic covered in this article.
✍️ About the Author
Vijay Vinoth — Lead Programmer Analyst with expertise in PHP, Perl, Python, and Shell scripting. Passionate about AI, automation, and building scalable systems. Writing to share practical insights from real-world engineering experience.
Note: This technical analysis reflects my independent understanding as a Lead Programmer Analyst as of September 2026.
As AI ecosystems like Claude 4.6 Opus evolve, actual implementation may vary. Refer to official documentation for final specs.
