AI‑Powered Automated Documentation Generator for Codebases — Part 1: Project Overview & Environment Setup

⏱ 7 min read  |  ~1445 words

AI‑Powered Automated Documentation Generator for Codebases — Part 1: Project Overview & Environment Setup

Welcome back! In the first two installments of this series I introduced the business case for AI‑driven documentation and sketched a high‑level workflow that stitches together static analysis, LLM inference, and continuous integration. Based on my technical understanding as a Lead Programmer Analyst (PHP, Perl, Python, Shell), I’ll now walk you through the concrete foundations of the project: a clear architectural blueprint and a reproducible development environment that lets you start generating system‑level docs from day one.

Why AI‑Generated Documentation Matters in 2026

• Traditional tools (Javadoc, Doxygen, Sphinx) excel at extracting inline comments but fall short when you need contextual narratives that span multiple services, data pipelines, and deployment artifacts.
• According to the Top 7 Documentation Generator Tools to Know About in 2026, even the most sophisticated generators still struggle with “system‑level” documentation because they lack holistic awareness of the repository and its related assets. The missing piece is a multi‑agent AI engine that can read code, understand architecture diagrams, and respect security boundaries.
• Enter the Claude 4.6 Opus Agentic Workflows and GPT‑5.4 Pro Parallel Agents. Both platforms expose high‑throughput, low‑latency inference endpoints that can be orchestrated in parallel, dramatically reducing the time from code commit to a polished documentation draft.

Project Vision

The AI‑Powered Automated Documentation Generator (henceforth AI‑Doc‑Gen) is designed to:

1. Ingest an entire codebase (including Dockerfiles, CI/CD pipelines, and architecture diagrams).
2. Analyze the assets with a fleet of specialized agents (e.g., Parser, Context‑Enricher, Writer).
3. Synthesize a cohesive set of markdown, HTML, and OpenAPI specs that can be published to a static site or a Confluence space.
4. Iterate automatically on pull‑request events, keeping docs in sync with code changes.

All of this is open‑source and inspired by the divar‑ir/ai-doc-gen repository, which already demonstrates multi‑agent orchestration, GitLab integration, and concurrent processing. We’ll reuse many of its concepts but upgrade the LLM stack to the latest Claude 4.6 Opus and GPT‑5.4 Pro APIs for better reasoning and faster throughput.

High‑Level Architecture

This diagram (simplified) captures the data flow:

┌─────────────────────┐
│   Repository (git)   │
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐    async calls
│  File‑Scanner Agent  │───────────────────────► Claude 4.6 Opus
└───────┬─────────────┘                         │
        │                                       ▼
        ▼                               ┌───────────────────┐
┌─────────────────────┐                │ Context‑Enricher │
│ Static‑Analysis Agent│◄───────────────│   Agent (GPT‑5.4)│
└───────┬─────────────┘                └───────┬───────────┘
        │                                    │
        ▼                                    ▼
┌─────────────────────┐                ┌───────────────────┐
│    Writer Agent     │◄───────────────│   Orchestrator    │
└─────────────────────┘                └───────────────────┘

Prerequisites

Before we dive into code, ensure your workstation meets the following baseline:

• OS: Ubuntu 22.04 LTS (or WSL2 on Windows). The container images we’ll use are built for x86_64.
• Docker Engine: ≥ 24.0.0 (required for Compose V2).
• Python: 3.12 (managed via pyenv or conda).
• API Access: Valid Claude 4.6 Opus key and OpenAI API key with GPT‑5.4 Pro enabled.
• Git: 2.43+ (for submodule handling).

Step‑by‑Step Environment Setup

1. Install System Packages

# Update apt and install core utilities
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
    build-essential \
    git \
    curl \
    wget \
    python3-pip \
    python3-venv \
    docker.io \
    docker-compose-plugin

# Verify installations
docker --version
docker compose version
python3 --version
git --version

2. Configure Docker for Non‑Root Users

# Add your user to the docker group (replace $USER if needed)
sudo usermod -aG docker $USER
newgrp docker   # refresh group membership without logout
docker run hello-world   # sanity check

3. Clone the Project Repository

We’ll start from the official repo and then apply a few patches to target Claude 4.6 Opus and GPT‑5.4 Pro.

git clone https://github.com/divar-ir/ai-doc-gen.git
cd ai-doc-gen

# Optional: checkout the latest stable tag
git checkout tags/v2.1.0

4. Create a Python Virtual Environment

python3 -m venv .venv
source .venv/bin/activate

# Upgrade pip and install core dependencies
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt

5. Install Claude 4.6 Opus SDK & OpenAI SDK

Claude’s Python client (v0.9) now supports async batch calls, which we’ll use to fan‑out requests across agents.

pip install anthropic[async]   # Claude 4.6 Opus
pip install openai==1.30.0     # GPT‑5.4 Pro (parallel streaming)

6. Create a .env File with Secrets

Never commit this file; add it to .gitignore (already present in the upstream repo).

# .env
CLAUDE_API_KEY=sk-ant‑xxxxxxxxxxxxxxxxxxxxxxxxxxxx
OPENAI_API_KEY=sk‑openai‑xxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITLAB_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
REDIS_URL=redis://redis:6379/0
PROJECT_ROOT=/workspace/ai-doc-gen
DOC_OUTPUT_DIR=/workspace/ai-doc-gen/.ai/docs

7. Docker Compose Configuration

The project runs three containers:

1. redis – message broker for Celery.
2. worker – runs the agent processes.
3. webhook – FastAPI endpoint that receives Git events.

# docker-compose.yml
version: "3.9"

services:
  redis:
    image: redis:7-alpine
    restart: unless‑stopped
    ports:
      - "6379:6379"

  worker:
    build: .
    environment:
      - &env_vars
        CLAUDE_API_KEY=${CLAUDE_API_KEY}
        OPENAI_API_KEY=${OPENAI_API_KEY}
        REDIS_URL=redis://redis:6379/0
        PROJECT_ROOT=${PROJECT_ROOT}
        DOC_OUTPUT_DIR=${DOC_OUTPUT_DIR}
    depends_on:
      - redis
    command: ["celery", "-A", "ai_doc_gen.worker", "worker", "--loglevel=info"]
    volumes:
      - .:${PROJECT_ROOT}
    restart: unless‑stopped

  webhook:
    build: .
    environment:
      - *env_vars
    depends_on:
      - redis
    ports:
      - "8000:8000"
    command: ["uvicorn", "ai_doc_gen.webhook:app", "--host", "0.0.0.0", "--port", "8000"]
    restart: unless‑stopped
    volumes:
      - .:${PROJECT_ROOT}

8. Build the Docker Image

The Dockerfile is deliberately lightweight – we use python:3.12-slim as the base, install the SDKs, and copy the source tree.

# Dockerfile
FROM python:3.12-slim

# System dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    git \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Create a non‑root user
ARG UID=1000
ARG GID=1000
RUN groupadd -g $GID appgroup && \
    useradd -m -u $UID -g $GID -s /bin/bash appuser

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
    && pip install anthropic[async] openai==1.30.0

# Copy the project
COPY . .

# Switch to non‑root user
USER appuser

EXPOSE 8000

9. Spin Up the Stack

# Load .env into the current shell
export $(grep -v '^#' .env | xargs)

# Bring up the services
docker compose up -d

# Verify Celery worker is alive
docker compose logs worker | tail -n 20

10. Test the End‑to‑End Pipeline

We’ll trigger a manual scan of the repository to see the agents in action. The FastAPI webhook also supports a /run endpoint for ad‑hoc runs.

curl -X POST http://localhost:8000/run \
     -H "Content-Type: application/json" \
     -d '{"repo_path": "/workspace/ai-doc-gen"}'

After a few seconds (Claude 4.6 Opus can process ~150 tokens/ms in parallel, GPT‑5.4 Pro adds ~2× speed), you’ll find generated markdown files under .ai/docs/. Open any file to verify the output:

# Project Overview

**Repository:** ai-doc-gen  
**Generated on:** 2026‑10‑05

## Architecture Summary
- **File‑Scanner Agent**: 0.12 s
- **Static‑Analysis Agent**: 0.45 s
- **Context‑Enricher (Claude 4.6 Opus)**: 1.78 s
- **Writer (GPT‑5.4 Pro)**: 0.89 s

...

Security & Privacy Considerations

When you ship code to an LLM, you expose intellectual property and potentially secrets. The Kodesage blog stresses that “creating truly comprehensive, system‑level documentation still demands deeper context from entire codebases and related assets, which introduces significant security and privacy challenges.” To mitigate:

• Run the agents inside a private VPC or on‑premise hardware; never send code to a public endpoint without encryption.
• Leverage Claude’s Enterprise Data Guard (available in Opus) which enforces a “no‑learning” policy for proprietary content.
• Mask secrets before sending snippets: a pre‑processor replaces any pattern matching API_KEY= or password with ***REDACTED***.

Next Steps (Sneak Peek of Part 2)

With the environment ready, Part 2 will dive into the Agent Design Patterns that make the system resilient: error handling with exponential back‑off, parallel batching of LLM calls, and a “knowledge‑graph” cache that stores previously generated summaries to avoid redundant inference.

📚 References & Further Reading

• The Top 7 Documentation Generator Tools to Know About in 2026 – Kodesage analysis of modern doc‑gen ecosystems.
• divar‑ir/ai-doc-gen – Open‑source multi‑agent documentation generator (source of our architecture).
• Best Code Documentation Tools 2026 – Mintlify’s overview of AI‑assisted doc tools, including “AI Write Assist”.
• How I Built an AI Code Assistant That Generates Documentation in Seconds – Practical tutorial that inspired our CI integration.
• How AI is Transforming Test Case Generation in 2026 – Shows how similar LLM pipelines can be repurposed for testing, reinforcing the value of reusable agent frameworks.

Your Turn

Imagine you have a monorepo that houses both a legacy Perl service and a brand‑new Python micro‑service. How would you adapt the agent pipeline to respect the different security policies (e.g., internal‑only vs. public‑facing) while still delivering a unified documentation site? Share your thoughts, design sketches, or code snippets in the comments below.