Building a Production AI Agent for CI/CD: Wiring Python, Qdrant, and GitHub Actions to Audit Pull Requests

Automating license compliance in CI/CD pipelines sounds straightforward: parse package.json, inspect package names, and block the Pull Request if someone smuggles in a Copyleft license (like GPL-2.0 or GPL-3.0). In practice, when dealing with messy npm metadata, registry lookups, and the runaway cost of continuous LLM API calls, traditional static scripts quickly fall apart.

In Part 1 we scaled Sentinel AI horizontally with LangGraph Map-Reduce. In Part 2 we locked down verdict output with Pydantic and grammar-based sampling on local Ollama. This article covers the production layer: a headless Python CLI, GitHub Actions orchestration, Qdrant-backed verdict caching, and Groq-hosted LLMs for CI runners that have no GPU.

Local Dev vs. CI: Same Graph, Different Backends

Sentinel AI runs the same LangGraph pipeline in both environments:

guardrail → scout → parallel lawyer/critic subgraphs → judge

The difference is infrastructure, not logic:

Local dev stays free and deterministic at the token level. CI trades that for speed and zero GPU setup on ephemeral runners.

System Architecture: Python Engine and JS Ecosystem

The system bridges two distinct tech stacks:

• Target domain: JavaScript / Node.js applications containing package.json.
• Analysis engine (Python): A CLI that loads the manifest, runs the LangGraph audit, and posts results back to GitHub.
• State layer: Qdrant stores prior verdicts keyed by (package_name, license_name) so repeat audits skip the LLM entirely.

The GitHub Actions Workflow

The workflow lives at .github/workflows/sentinel-test.yml. It triggers on pull requests targeting main or master, spins up Qdrant as a sidecar, and runs the audit CLI headlessly:

name: "Sentinel AI Audit"

on:
  pull_request:
    branches: [ main, master ]

jobs:
  audit:
    runs-on: ubuntu-latest
    services:
      qdrant:
        image: qdrant/qdrant:latest
        ports:
          - 6333:6333

    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -r backend/requirements.txt || pip install -r requirements.txt          

      - name: Run Sentinel AI Audit
        run: |
          python -m backend.app.cli --package-json package.json || python -m app.cli --package-json package.json          
        env:
          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
          LLM_PROVIDER: "groq"
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Two details matter here:

1. pull-requests: write is required. Without it, the CLI can audit dependencies but cannot post the Markdown report as a PR comment.
2. LLM_PROVIDER=groq switches the model registry from Ollama to Groq cloud models. The workflow reads GROQ_API_KEY (synced internally to LLM_API_KEY at startup).

Qdrant connects via ModelConfig.QDRANT_HOST, which defaults to http://localhost:6333. That matches the service container port mapping, so no extra host configuration is needed on the runner.

Challenge #1: Ephemeral Runners vs. Stateful Verdict Memory

GitHub Actions runners are disposable. Every job starts from scratch. Without a cache, every PR re-audits every package through Groq, burning tokens on dependencies you already evaluated yesterday.

The fix: run Qdrant as a service container on the same runner network. The verdict_cache collection stores prior lawyer verdicts as payload records. Lookups use a payload filter on package_name and license_name, not semantic vector search:

records, _ = await client.scroll(
    collection_name="verdict_cache",
    scroll_filter=Filter(
        must=[
            FieldCondition(key="package_name", match=MatchValue(value=package_name)),
            FieldCondition(key="license_name", match=MatchValue(value=license_name)),
        ]
    ),
    limit=1,
    with_payload=True,
    with_vectors=False,
)

Qdrant requires a vector field, so each point carries a dummy [0.0] vector. The cache behaves like a keyed store; the vector DB is infrastructure convenience, not RAG.

On a cache hit, the graph skips the entire lawyer/critic subgraph:

cached = await get_cached_verdict(package_name, license_name)
if cached:
    print(f"[CACHE HIT/{tier_label}] Bypassing LLM for {package_name}")
    cached_result["verdict"] = cached.get("verdict", "")
    cached_result["reasoning"] = cached.get("reasoning", "")
    return {"analyzed_dependencies": [cached_result]}

Within a single workflow run, repeated packages hit Qdrant instantly. Across runs, the cache resets with the ephemeral container. For cross-run persistence, point QDRANT_HOST at a managed Qdrant Cloud instance instead.

Challenge #2: From package.json to npm Registry Evidence

The scout node does more than read top-level keys. It merges dependencies and devDependencies, then fetches each package from the npm registry to resolve the actual SPDX license string:

merged = {**dependencies, **devDependencies}

for package_name, version_spec in merged.items():
    url = f"https://registry.npmjs.org/{package_name}/latest"
    license_value, version = await _fetch_package_metadata(client, package_name)

The _parse_npm_license() helper handles the three common npm metadata shapes: a plain string, a { "type": "MIT" } object, or a legacy licenses[] array. Packages the registry cannot resolve still enter the audit queue with license: "UNKNOWN".

Packages above 1500 tokens of context route to the HEAVY lawyer tier (llama-3.3-70b-versatile on Groq). Everything else uses the STANDARD tier (llama-3.1-8b-instant).

Challenge #3: Hybrid Determinism and Non-Determinism

Good AI engineering means not burning tokens on tasks that can be solved deterministically in O(1) time.

Deterministic layer: Parse package.json, resolve licenses from npm, check verdict_cache by exact (package_name, license_name) match.

Non-deterministic layer: On cache miss, the lawyer node calls Groq with Pydantic-typed structured output:

class LawyerAuditResponse(BaseModel):
    verdict: Literal["SAFE", "FORBIDDEN", "REVIEW_REQUIRED"]
    reasoning: str

structured_llm = llm.with_structured_output(LawyerAuditResponse, method="json_mode")
response = await structured_llm.ainvoke(messages)

If structured output validation fails, the lawyer fails closed with verdict="FORBIDDEN" rather than letting a malformed response through.

Corporate policy comes from .sentinel.yml (or conservative in-repo defaults listing MIT, Apache-2.0 as allowed and GPL/AGPL as forbidden).

Posting Results Back to the Pull Request

After the graph completes, the CLI builds a Markdown table report and publishes it in two places:

1. GitHub step summary via $GITHUB_STEP_SUMMARY
2. PR comment via the REST API

def post_pull_request_comment(markdown_report: str) -> None:
    pr_number = _read_pull_request_number()  # parsed from GITHUB_EVENT_PATH
    url = f"https://api.github.com/repos/{repository}/issues/{pr_number}/comments"
    requests.post(url, headers={"Authorization": f"Bearer {token}"}, json={"body": markdown_report})

Comment posting is best-effort. If the token is missing or the API returns an error, the CLI logs a warning and continues. The job still fails on policy violations via resolve_exit_code():

for dep in final_state.get("analyzed_dependencies") or []:
    if dep.get("verdict") == "FORBIDDEN":
        return 1
return 0

This gives you both human-readable feedback on the PR and a hard gate that blocks merges when copyleft dependencies slip in.

Hard-Earned Production Lessons

CLI Parameter Hygiene

The CLI accepts explicit flags, not positional arguments:

python -m app.cli --package-json package.json --output audit-report.md

Default is package.json in the working directory. The --output flag writes a local Markdown copy alongside the GitHub publish step.

Git History Isolation

The workflow must exist on the default branch before it runs against PRs. If .github/workflows/sentinel-test.yml only lives on a feature branch, GitHub will not trigger it. Merge the workflow to main first, then open PRs that modify package.json.

Graceful Degradation When Qdrant Is Down

Cache lookups wrap all errors in a warning and return None, letting the audit proceed without cache:

except Exception as err:
    logger.warning("Verdict cache lookup failed for %s (%s): %s", package_name, license_name, err)
    return None

The pipeline never crashes because Qdrant hiccuped. It just pays the LLM cost for that run.

For fully stateless CI runs without any Qdrant dependency, set SENTINEL_CI=true. That activates in-memory MemorySaver checkpointing and bypasses all cache reads/writes via set_ci_mode(True).

Eval Suite Stays Local

The eval harness (scripts/run_evals.py) with five ground-truth license scenarios runs against a local Qdrant stack. It is not wired into the GitHub Actions workflow yet. CI currently audits the repo’s own test manifest:

{
  "name": "sentinel-test",
  "dependencies": {
    "gpl-bad-pkg": "1.0.0",
    "potrace": "^2.1.8"
  }
}

gpl-bad-pkg is a deliberate copyleft trap. potrace is a real package with a GPL-2.0 license that exercises the full scout → lawyer → judge path against live npm metadata.

Conclusion: Prototype AI vs. Enterprise Software

Modern AI engineering is 80% software architecture and infrastructure, and 20% prompt design. Wiring a verdict cache into ephemeral CI/CD runners, swapping Ollama for Groq without rewriting the graph, and posting structured audit reports directly on PRs is where prototype AI turns into deployable enterprise software.

The Sentinel AI series so far:

• Part 1: horizontal scaling with LangGraph Map-Reduce
• Part 2: guaranteed structured outputs with Pydantic and GBNF on Ollama
• Part 3 (this article): production CI/CD with GitHub Actions, Qdrant, and Groq