AI-Driven Automated Network Monitoring & Anomaly Detection — Part 1: Setting Up the Monitoring Stack with Prometheus and Node Exporter

⏱ 8 min read  |  ~1648 words

AI‑Driven Automated Network Monitoring & Anomaly Detection — Part 1: Setting Up the Monitoring Stack with Prometheus and Node Exporter

In the first two installments we covered the high‑level architecture of an AI‑enhanced observability pipeline and explored the data‑flow requirements for feeding raw network telemetry into a machine‑learning model. In this third part we roll up our sleeves, provision the core metrics collection stack, and lay the groundwork for the AI layer that will run on top of it.

Based on my technical understanding as a Lead Programmer Analyst (PHP, Perl, Python, Shell) and keeping an eye on the latest advances—Claude 4.6 Opus agentic workflows and GPT‑5.4 Pro parallel agents—I’ll walk you through a production‑ready, reproducible setup that you can spin up on a single‑node lab or scale out to a full OpenShift cluster.

Why Prometheus + Node Exporter Still Rules in 2026

• Time‑series native: Prometheus stores metrics in a highly compressed, query‑optimised format that makes real‑time analytics trivial.
• Pull‑based scraping: Guarantees that you only collect what you ask for, reducing noise and accidental data‑leaks.
• Ecosystem maturity: Over 12 000 integrations, first‑class support for Kubernetes/OpenShift, and a thriving community (see the official repo).
• AI‑ready: Prometheus metrics can be exported to remote storage (e.g., Cortex, Thanos) or streamed directly to a Python inference service via the remote_write API, a pattern that powers the AI‑driven monitoring demos highlighted by Dhinesh Kumar on LinkedIn (source).

In practice, the Node Exporter is the de‑facto agent for gathering OS‑level metrics (CPU, memory, network I/O, disk latency) from every host you want to watch. Pairing it with Prometheus gives you a solid, low‑overhead baseline that AI models can later enrich with anomaly scores, root‑cause explanations, or predictive alerts.

Prerequisites

If you are deploying on OpenShift, you can replace the Docker steps with oc new-app or Helm charts—both are covered in the “Scaling Out” sidebar later.

Step 1 – Install Docker (if not already present)

# Ubuntu example
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Add Docker’s official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
    | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify
docker version

Step 2 – Pull and Run the Node Exporter

The Node Exporter runs as a privileged container so it can read host‑level stats. In a production setting you would use a DaemonSet; for this tutorial we keep it simple.

# Create a dedicated network
docker network create monitoring

# Run Node Exporter (listens on 9100)
docker run -d \
  --name node-exporter \
  --network monitoring \
  --restart unless-stopped \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  -v "/sys:/sys:ro" \
  -v "/proc:/proc:ro" \
  -v "/etc/localtime:/etc/localtime:ro" \
  prom/node-exporter:latest \
  --path.rootfs=/host

Verify that metrics are reachable:

curl http://localhost:9100/metrics | head -n 20
# Expected: # HELP node_cpu_seconds_total ...

Step 3 – Configure Prometheus

Create a directory for Prometheus configuration and data:

mkdir -p $HOME/prometheus/{data,conf}

Now create prometheus.yml (the heart of the stack). The file includes a scrape_config for the Node Exporter and a remote_write endpoint that will feed metrics to a lightweight Flask service where Claude 4.6 or GPT‑5.4 agents can run inference.

# $HOME/prometheus/conf/prometheus.yml
global:
  scrape_interval: 15s        # Default scrape cadence
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

remote_write:
  - url: "http://ai-infer:9091/write"
    # Optional: add authentication headers if you lock the endpoint down
    # basic_auth:
    #   username: prometheus
    #   password: secret

Spin up Prometheus, mounting the config and persisting data:

docker run -d \
  --name prometheus \
  --network monitoring \
  -p 9090:9090 \
  -v $HOME/prometheus/conf/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v $HOME/prometheus/data:/prometheus \
  --restart unless-stopped \
  prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus

Open http://localhost:9090 in your browser. The “Targets” page should show the Node Exporter as UP.

Step 4 – Build a Minimal AI Inference Service (Python + FastAPI)

We’ll create a tiny HTTP endpoint that accepts the same remote‑write protobuf format Prometheus uses. For brevity we’ll decode the payload with prometheus_client utilities and then run a pre‑trained isolation‑forest model (scikit‑learn) that flags outliers. In a real deployment you could replace the model with a Claude 4.6 agentic workflow or a GPT‑5.4 parallel‑agent ensemble that enriches each sample with contextual metadata.

4.1 Install Python dependencies

python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn prometheus-client scikit-learn pandas

4.2 Train a quick baseline model (run once)

# train_model.py
import pandas as pd
from sklearn.ensemble import IsolationForest
import joblib

# Pull a few minutes of historic node metrics (you can use the Prometheus HTTP API)
# For the demo we generate synthetic data
def synthetic_data(rows=5000):
    import numpy as np
    ts = pd.date_range(end=pd.Timestamp.now(), periods=rows, freq='15s')
    df = pd.DataFrame({
        'cpu_usage': np.random.normal(0.2, 0.05, size=rows),   # 20% avg
        'mem_usage': np.random.normal(0.55, 0.07, size=rows),  # 55% avg
        'net_rx': np.random.exponential(100, size=rows),
        'net_tx': np.random.exponential(80, size=rows),
    }, index=ts)
    return df

df = synthetic_data()
model = IsolationForest(contamination=0.01, random_state=42)
model.fit(df)

joblib.dump(model, 'anomaly_model.joblib')
print('Model saved.')

Run it once:

python train_model.py

4.3 FastAPI service that receives Prometheus remote_write payloads

# ai_infer_service.py
import joblib
import pandas as pd
from fastapi import FastAPI, Request, HTTPException
from prometheus_client.parser import text_string_to_metric_families
import uvicorn
import logging

app = FastAPI()
model = joblib.load('anomaly_model.joblib')
log = logging.getLogger("uvicorn.error")

def parse_remote_write(body: bytes) -> pd.DataFrame:
    """
    Very simplified parser: Prometheus remote_write sends protobuf-encoded
    WriteRequest messages.  For a tutorial we accept the *text* format that
    the Node Exporter also exposes (via /metrics) and convert it to a DataFrame.
    """
    metrics = {}
    for fam in text_string_to_metric_families(body.decode()):
        for sample in fam.samples:
            name, labels, value = sample
            # Flatten label dict into a single key for demo purposes
            key = f"{name}{{" + ",".join(f"{k}='{v}'" for k, v in labels.items()) + "}}"
            metrics[key] = value
    # Convert to a single‑row DataFrame
    df = pd.DataFrame([metrics])
    # Keep only the features we trained on
    features = ['node_cpu_seconds_total{mode=\'idle\'}',
                'node_memory_MemAvailable_bytes',
                'node_network_receive_bytes_total',
                'node_network_transmit_bytes_total']
    # Fill missing with 0 (real service would handle NaNs more gracefully)
    df = df.reindex(columns=features, fill_value=0)
    return df

@app.post("/write")
async def remote_write(request: Request):
    try:
        payload = await request.body()
        df = parse_remote_write(payload)
        # Predict anomaly (1 = normal, -1 = outlier)
        pred = model.predict(df)[0]
        if pred == -1:
            log.warning(f"⚠️ Anomaly detected: {df.to_dict(orient='records')[0]}")
        return {"status": "ok", "anomaly": pred == -1}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=9091)

Launch the service in a container (Dockerfile shown for reproducibility):

# Dockerfile
FROM python:3.12-slim

WORKDIR /app
COPY venv /app/venv
COPY ai_infer_service.py /app/
COPY anomaly_model.joblib /app/

ENV PATH="/app/venv/bin:$PATH"

EXPOSE 9091
CMD ["uvicorn", "ai_infer_service:app", "--host", "0.0.0.0", "--port", "9091"]

Build and run:

# Build
docker build -t ai-infer:latest .

# Run (same monitoring network)
docker run -d \
  --name ai-infer \
  --network monitoring \
  -p 9091:9091 \
  --restart unless-stopped \
  ai-infer:latest

Step 5 – Verify the End‑to‑End Flow

1. Prometheus scrapes Node Exporter every 15 seconds.
2. Each scrape is forwarded via remote_write to http://ai-infer:9091/write.
3. The FastAPI service decodes the payload, runs the IsolationForest model, and logs any anomaly.

Trigger a synthetic spike to see the detector in action:

# Simulate CPU load for 30 seconds
stress --cpu 4 --timeout 30
# Wait for the next scrape; you should see a warning in the ai-infer container logs
docker logs -f ai-infer | grep "Anomaly"

If everything is wired correctly you’ll see a line similar to:

⚠️ Anomaly detected: {'node_cpu_seconds_total{mode='idle'}': 12.34, 'node_memory_MemAvailable_bytes': 1.2e+09, 'node_network_receive_bytes_total': 5.6e+07, 'node_network_transmit_bytes_total': 2.1e+07}

Scaling Out: From a Single Node to OpenShift

When you move beyond a lab, the same components become DaemonSets (Node Exporter) and a StatefulSet (Prometheus) on OpenShift. The official Helm chart (kube‑prometheus‑stack) auto‑generates the ServiceMonitor objects that tell Prometheus to scrape every pod that carries the prometheus.io/scrape: "true" annotation.

For the AI inference side, you can deploy the FastAPI container as a Deployment with a ClusterIP service, then configure Prometheus’ remote_write to point to http://ai-infer.<namespace>.svc:9091/write. Both Claude 4.6 and GPT‑5.4 support parallel execution via the openai SDK; you can replace the IsolationForest with a ClaudeAgent that runs a prompt‑engineered anomaly detection chain, or spin up a GPTParallelExecutor that evaluates multiple model slices concurrently for higher throughput.

Best‑Practice Checklist

Putting It All Together – A Minimal Alerting Rule

Let’s expose the anomaly flag as a custom metric from the inference service. Update ai_infer_service.py to emit a Prometheus gauge via the prometheus_client library:

from prometheus_client import Gauge, start_http_server

# Define a gauge that will be scraped by Prometheus
anomaly_gauge = Gauge('ai_anomaly_detected', '1 if an anomaly was detected in the last batch')

@app.on_event("startup")
def start_metrics():
    # Expose metrics on port 8000 (different from write endpoint)
    start_http_server(8000)

@app.post("/write")
async def remote_write(request: Request):
    ...
    if pred == -1:
        anomaly_gauge.set(1)
    else:
        anomaly_gauge.set(0)
    ...

Re‑build and redeploy the container, then add a new scrape job to prometheus.yml:

  - job_name: 'ai-infer'
    static_configs:
      - targets: ['ai-infer:8000']

Finally, create an alert rule (e.g., alerts.yml) and load it via the --web.enable-admin-api flag or via the Prometheus UI:

groups:
  - name: ai_anomaly_alerts
    rules:
      - alert: NetworkAnomalyDetected
        expr: ai_anomaly_detected == 1
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Anomaly detected on {{ $labels.instance }}"
          description: "AI model flagged a metric outlier. Check node metrics for root cause."

🔗 You Might Also Like

• AI Safety Spotlight: EU’s New Requirements for Explainability in High‑Stakes AI Systems – Part 1
• AI Agents in Supply‑Chain Resilience: Deploying Autonomous Negotiators for Real‑Time Procurement
• Prompt Engineering for Zero‑Shot Code Generation: Techniques to Maximize LLM Efficiency