AI-Driven Automated Network Monitoring & Anomaly Detection — Part 2: Collecting Network Metrics via Bash Scripts and Exporters

⏱ 8 min read  |  ~1582 words

AI-Driven Automated Network Monitoring & Anomaly Detection — Part 2: Collecting Network Metrics via Bash Scripts and Exporters

Based on my technical understanding as a Lead Programmer Analyst (PHP, Perl, Python, Shell), this tutorial walks you through the practical steps of pulling raw network telemetry with Bash, shaping it for Prometheus, and laying the groundwork for AI‑enhanced anomaly detection.

Quick Recap of Part 1

In the first installment we defined the end‑to‑end architecture: edge devices push NetFlow/IPFIX to a collector, Prometheus scrapes time‑series data, and an AI layer (Claude 4.6 Opus agentic workflows or GPT‑5.4 Pro parallel agents) scores anomalies in real time. We also explored why Prometheus is the de‑facto choice for real‑time metric scraping, citing the Medium article on AIOps Real‑Time Anomaly Detection in Network Operations Using AIOps.

Why Bash and Exporters Still Matter in 2026

Even with sophisticated AI agents, the data‑ingestion layer must be reliable, low‑latency, and easy to audit. Bash scripts excel at:

• Running on any POSIX‑compatible host without extra runtimes.
• Co‑ordinating native OS utilities (e.g., ip, ethtool, nvidia-smi) to fetch hardware‑level health.
• Exporting metrics in the plain‑text Prometheus exposition format, which any Prometheus server can scrape.

When paired with node_exporter or a custom textfile collector, these scripts become the bridge between raw telemetry and the AI‑driven analytics pipeline.

Setting Up the Prometheus Scrape Target

First, ensure Prometheus is running with a prometheus.yml that includes a textfile job. This is the same pattern highlighted in the Medium AIOps post.

global:
  scrape_interval: 15s
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'network_metrics'
    static_configs:
      - targets: ['localhost:9100']
    metrics_path: /metrics
    file_sd_configs:
      - files:
        - /etc/prometheus/file_sd/network_metrics.yml

The file_sd stanza tells Prometheus to look for a list of files that contain the textfile metrics. Each file will be generated by the Bash exporters we build next.

1. Collecting Interface Statistics

Network interface counters (bytes, packets, errors) are the foundation for bandwidth utilization and packet loss detection. Below is a compact Bash script that reads /sys/class/net, formats the data, and writes it to a Prometheus‑compatible file.

#!/usr/bin/env bash
#
# network_if_exporter.sh – Export per‑interface stats for Prometheus
# Author: Vijay Vinoth (Lead Programmer Analyst)
# ---------------------------------------------------------

# Directory where Prometheus textfile collector reads files
OUT_DIR="/var/lib/prometheus/node_exporter/textfile_collector"
OUT_FILE="${OUT_DIR}/network_if.prom"

# Ensure the output directory exists
mkdir -p "${OUT_DIR}"

# Helper to emit a metric line
emit_metric() {
  local metric=$1
  local labels=$2
  local value=$3
  echo "${metric}${labels} ${value}"
}

# Header for Prometheus (optional but nice)
{
  echo "# HELP net_if_bytes_total Total bytes transmitted/received on the interface"
  echo "# TYPE net_if_bytes_total counter"
  echo "# HELP net_if_packets_total Total packets transmitted/received on the interface"
  echo "# TYPE net_if_packets_total counter"
  echo "# HELP net_if_errors_total Total error counters on the interface"
  echo "# TYPE net_if_errors_total counter"
} > "${OUT_FILE}"

# Loop over each physical interface (skip lo)
for IFACE in $(ls /sys/class/net | grep -v '^lo$'); do
  # Read counters from sysfs
  RX_BYTES=$(cat /sys/class/net/${IFACE}/statistics/rx_bytes)
  TX_BYTES=$(cat /sys/class/net/${IFACE}/statistics/tx_bytes)
  RX_PACKETS=$(cat /sys/class/net/${IFACE}/statistics/rx_packets)
  TX_PACKETS=$(cat /sys/class/net/${IFACE}/statistics/tx_packets)
  RX_ERRORS=$(cat /sys/class/net/${IFACE}/statistics/rx_errors)
  TX_ERRORS=$(cat /sys/class/net/${IFACE}/statistics/tx_errors)

  # Emit metrics with interface label
  emit_metric "net_if_bytes_total" "{interface=\"${IFACE}\",direction=\"rx\"}" "${RX_BYTES}" >> "${OUT_FILE}"
  emit_metric "net_if_bytes_total" "{interface=\"${IFACE}\",direction=\"tx\"}" "${TX_BYTES}" >> "${OUT_FILE}"
  emit_metric "net_if_packets_total" "{interface=\"${IFACE}\",direction=\"rx\"}" "${RX_PACKETS}" >> "${OUT_FILE}"
  emit_metric "net_if_packets_total" "{interface=\"${IFACE}\",direction=\"tx\"}" "${TX_PACKETS}" >> "${OUT_FILE}"
  emit_metric "net_if_errors_total" "{interface=\"${IFACE}\",direction=\"rx\"}" "${RX_ERRORS}" >> "${OUT_FILE}"
  emit_metric "net_if_errors_total" "{interface=\"${IFACE}\",direction=\"tx\"}" "${TX_ERRORS}" >> "${OUT_FILE}"
done

Schedule the script with systemd or cron to run every 15 seconds (matching Prometheus’ scrape interval). Example systemd unit:

[Unit]
Description=Network Interface Prometheus Exporter
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/network_if_exporter.sh

[Install]
WantedBy=multi-user.target

Enable a timer that fires every 15 seconds:

[Unit]
Description=Run network_if_exporter every 15 seconds

[Timer]
OnBootSec=5sec
OnUnitActiveSec=15sec
Unit=network_if_exporter.service

[Install]
WantedBy=timers.target

2. Exporting Router/Switch SNMP Metrics

While the host‑level script covers NICs, most production networks need per‑port health from switches and routers. The snmpwalk utility (part of net-snmp) can be wrapped in Bash to expose ifHCInOctets, ifHCOutOctets, and ifOperStatus.

#!/usr/bin/env bash
#
# snmp_if_exporter.sh – Pull SNMP interface counters and expose to Prometheus
# Requires: net-snmp, jq (for JSON conversion)
# ---------------------------------------------------------

# Configuration – replace with your own device list
DEVICES=(
  "router01|public|10.0.0.1"
  "switch02|public|10.0.0.2"
)

OUT_DIR="/var/lib/prometheus/node_exporter/textfile_collector"
OUT_FILE="${OUT_DIR}/snmp_if.prom"

mkdir -p "${OUT_DIR}"
{
  echo "# HELP snmp_if_bytes_total Total bytes on SNMP‑managed interfaces"
  echo "# TYPE snmp_if_bytes_total counter"
  echo "# HELP snmp_if_oper_status Current operational status (1=up,2=down,3=testing)"
  echo "# TYPE snmp_if_oper_status gauge"
} > "${OUT_FILE}"

# OIDs we care about
OID_IN_OCTETS="IF-MIB::ifHCInOctets"
OID_OUT_OCTETS="IF-MIB::ifHCOutOctets"
OID_OPER_STATUS="IF-MIB::ifOperStatus"

for ENTRY in "${DEVICES[@]}"; do
  IFS='|' read -r NAME COMMUNITY HOST << EOF
$ENTRY
EOF

  # Pull data via SNMP bulk walk
  SNMP_DATA=$(snmpbulkwalk -v2c -c "${COMMUNITY}" "${HOST}" "${OID_IN_OCTETS}" "${OID_OUT_OCTETS}" "${OID_OPER_STATUS}" 2>/dev/null)

  # Parse each line
  while IFS= read -r line; do
    # Example line: IF-MIB::ifHCInOctets.1 = Counter64: 123456789
    if [[ $line =~ ^([A-Za-z0-9-]+)::([A-Za-z0-9]+)\.([0-9]+)\ =\ (.+):\ (.+)$ ]]; then
      MIB="${BASH_REMATCH[1]}"
      METRIC="${BASH_REMATCH[2]}"
      IFINDEX="${BASH_REMATCH[3]}"
      TYPE="${BASH_REMATCH[4]}"
      VALUE="${BASH_REMATCH[5]}"

      case "$METRIC" in
        ifHCInOctets)
          echo "snmp_if_bytes_total{device=\"${NAME}\",interface=\"${IFINDEX}\",direction=\"rx\"} ${VALUE}" >> "${OUT_FILE}"
          ;;
        ifHCOutOctets)
          echo "snmp_if_bytes_total{device=\"${NAME}\",interface=\"${IFINDEX}\",direction=\"tx\"} ${VALUE}" >> "${OUT_FILE}"
          ;;
        ifOperStatus)
          echo "snmp_if_oper_status{device=\"${NAME}\",interface=\"${IFINDEX}\"} ${VALUE}" >> "${OUT_FILE}"
          ;;
      esac
    fi
  done <<< "$SNMP_DATA"
done

This script can be extended to pull CPU load, memory, or temperature via standard MIBs. The output file snmp_if.prom lives alongside the NIC exporter, and Prometheus scrapes them in the same job.

3. GPU Health Exporter (Day‑2 AI Ops)

Many modern network appliances run AI inference on GPUs (e.g., packet‑level DDoS classifiers). Monitoring GPU health is crucial for the “Day 2” AI‑network‑operations scenario shown in the Aviz demo Day 2 AI network operations, monitoring and anomaly detection with Aviz. The NVIDIA nvidia-smi CLI provides JSON output that we can translate.

#!/usr/bin/env bash
#
# gpu_exporter.sh – Export GPU utilization, memory, and temperature
# ---------------------------------------------------------

OUT_DIR="/var/lib/prometheus/node_exporter/textfile_collector"
OUT_FILE="${OUT_DIR}/gpu.prom"

mkdir -p "${OUT_DIR}"
{
  echo "# HELP gpu_utilization_percent GPU utilization (percent)"
  echo "# TYPE gpu_utilization_percent gauge"
  echo "# HELP gpu_memory_used_bytes GPU memory used"
  echo "# TYPE gpu_memory_used_bytes gauge"
  echo "# HELP gpu_temperature_celsius GPU temperature (Celsius)"
  echo "# TYPE gpu_temperature_celsius gauge"
} > "${OUT_FILE}"

# nvidia-smi JSON format (requires driver >= 450)
JSON=$(nvidia-smi --query-gpu=index,utilization.gpu,memory.used,temperature.gpu --format=csv,noheader,nounits)

while IFS=',' read -r IDX UTIL MEM TEMP; do
  IDX=$(echo $IDX | xargs)   # trim spaces
  UTIL=$(echo $UTIL | xargs)
  MEM=$(echo $MEM | xargs)
  TEMP=$(echo $TEMP | xargs)

  echo "gpu_utilization_percent{gpu=\"${IDX}\"} ${UTIL}" >> "${OUT_FILE}"
  echo "gpu_memory_used_bytes{gpu=\"${IDX}\"} $((MEM * 1024 * 1024))" >> "${OUT_FILE}"
  echo "gpu_temperature_celsius{gpu=\"${IDX}\"} ${TEMP}" >> "${OUT_FILE}"
done <<< "$JSON"

Run this exporter on any compute node that hosts AI inference workloads. The metrics feed directly into Prometheus, where downstream AI agents (Claude 4.6 Opus or GPT‑5.4 Pro) can consume the time‑series for anomaly scoring.

4. Wiring Exporters into Prometheus – The File‑Based Service Discovery

Prometheus needs to know which .prom files to read. Create a simple file_sd YAML file that points to the directory we used above.

# /etc/prometheus/file_sd/network_metrics.yml
- targets:
  - localhost:9100
  labels:
    __metrics_path__: /metrics
    __scrape_interval__: 15s
    __path__: /var/lib/prometheus/node_exporter/textfile_collector/*.prom

After reloading Prometheus (curl -X POST http://localhost:9090/-/reload) the new metrics become visible at http://localhost:9090/graph. Try a quick query:

sum(rate(net_if_bytes_total{direction="rx"}[1m])) by (interface)

This shows inbound bandwidth per NIC over the last minute – the exact signal you will later feed into a transformer model.

5. From Raw Metrics to AI‑Ready Feature Vectors

Now that we have a clean, time‑aligned dataset, the next step (Part 3) is to pull the series into a feature store and run them through a parallel‑agent inference pipeline. In practice you would:

1. Use promtool tsdb create-blocks-from or a remote_write endpoint to stream data into a vector database (e.g., Milvus).
2. Launch a Claude 4.6 Opus agentic workflow that:
   • Normalizes the series (z‑score per interface).
   • Feeds sliding windows (e.g., 5‑minute) into a pre‑trained LSTM or a transformer model hosted on a GPU node.
   • Generates an anomaly score (0‑1) and writes it back as network_anomaly_score metric.
3. Configure Prometheus alerting rules to fire when network_anomaly_score > 0.8 for more than two consecutive scrapes.

This architecture mirrors the AI‑driven workflow demonstrated by IBM, where “AI‑enabled workflows—many driven by agentic AI—are poised to expand from 3 % in 2024 to 25 % by 2026” IBM AI Network Monitoring. By keeping the ingestion layer simple and observable (Bash + Prometheus), you give the AI agents a reliable foundation to reason over.

6. Adding Alerting & Incident Automation

Prometheus alone can fire alerts, but modern NOC stacks push those alerts into ticketing or chat tools. The Medium AIOps post outlines integration with ZenDesk, ServiceNow, and Slack. Below is a minimal alertmanager.yml snippet that routes high‑severity network anomalies to Slack.

global:
  resolve_timeout: 5m

route:
  receiver: 'slack-critical'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - match:
        severity: critical
      receiver: slack-critical

receivers:
  - name: 'slack-critical'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
        channel: '#network-ops'
        title: '{{ .CommonAnnotations.summary }}'
        text: |
          *Alert:* {{ .CommonLabels.alertname }}
          *Instance:* {{ .CommonLabels.instance }}
          *Description:* {{ .CommonAnnotations.description }}

Corresponding Prometheus rule (saved as network_anomaly.rules.yml):

groups:
  - name: network_anomaly
    rules:
      - alert: NetworkAnomalyDetected
        expr: network_anomaly_score > 0.8
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Potential network anomaly on {{ $labels.instance }}"
          description: "Anomaly score is {{ $value }} for the last two minutes."

When the AI model spikes the network_anomaly_score, the alert fires, Slack notifies the team, and a parallel GPT‑5.4 Pro agent can automatically open a ServiceNow ticket, attach the last 10 minutes of metric graphs, and suggest a remediation playbook.

7. Testing the Exporters – A Quick Lab

To verify everything works before you push to production, spin up a Docker‑Compose stack that includes Prometheus, node_exporter, and a tiny bash container that runs the scripts.

version: '3.8'

services:

  prometheus:

    image: prom/prometheus:latest

    volumes:

      - ./prometheus.yml:/etc/prometheus/prometheus.yml

      - ./file_sd:/etc/prometheus/file_sd

      - prom_data:/prometheus

    ports:

      - "9090:9090"

    command:

      - '--config.file=/etc/prometheus/prometheus.yml'

  node_exporter:

    image: prom/node-exporter:latest

    ports:

      - "9100:9100"

    volumes:

      - /proc:/host/proc:ro

      - /sys:/host/sys:ro

      - /:/rootfs:ro

    command:

      - '--path.procfs=/host/proc'

      - '--path.sysfs=/host/sys'

      - '--path.rootfs=/rootfs'
  exporter_host:

    image: alpine:3.19

    depends_on:

      - node_exporter

    volumes:

      - ./exporters:/usr/local/bin

      - ./node_exporter_text:/var/lib/prometheus/node_exporter/textfile

🔗 You Might Also Like

AI Safety Spotlight: EU’s New Requirements for Explainability in High‑Stakes AI Systems – Part 1
AI Agents in Supply‑Chain Resilience: Deploying Autonomous Negotiators for Real‑Time Procurement
Prompt Engineering for Zero‑Shot Code Generation: Techniques to Maximize LLM Efficiency




✍️ About the Author
Vijay Vinoth — Lead Programmer Analyst with expertise in PHP, Perl, Python, and Shell scripting. Passionate about AI, automation, and building scalable systems. Writing to share practical insights from real-world engineering experience.



        Note: This technical analysis reflects my independent understanding as a Lead Programmer Analyst as of April 2026.

        As AI ecosystems like Claude 4.6 Opus evolve, actual implementation may vary. Refer to official documentation for final specs.