marker/guarddog-nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9b98a32216ceb69dee6a2547d0b93042a51e9e69
guarddog-nexus/AGENTS.md

9.8 KiB
Raw Blame History

AGENTS.md — GuardDog Nexus

Project overview

GuardDog Nexus integrates GuardDog with Sonatype Nexus Repository Manager. It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis.

Stack: Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose.

Package ecosystems: PyPI, Go (proxy.golang.org), npm (registry.npmjs.org).

GuardDog binary: installed inside the Docker image via uv pip install --system guarddog. Supports pypi, go, npm subcommands.

Quick start

cp .env.example .env
# edit .env to set LLM vars if needed
make docker-up
# → guarddog-nexus :8080, Nexus :8081

For local development without Docker:

make install dev
export $(cat .env | xargs)
python -m guarddog_nexus.main
make test     # 135 tests
make lint     # ruff
make format   # ruff format + fix

Architecture

guarddog_nexus/
├── core/              # Business logic
│   ├── scanner.py     # subprocess guarddog CLI
│   ├── harvester.py   # download → sha256 → scan → store
│   ├── nexus.py       # httpx client + pypi/go/npm path extractors
│   └── llm.py         # OpenAI-compatible analysis client
├── db/                # Persistence
│   ├── engine.py      # async SQLAlchemy + auto-migration
│   ├── models.py      # Scan, Finding ORM
│   └── queries.py     # shared query builders
├── routes/            # HTTP layer
│   ├── webhooks.py    # POST /webhooks/nexus
│   ├── api_scans.py   # REST /api/v1/scans
│   ├── api_packages.py
│   ├── api_findings.py
│   ├── metrics.py     # GET /metrics (Prometheus)
│   └── web.py         # HTML UI (Jinja2 + htmx)
├── web/               # Static assets
│   ├── templates/     # Jinja2 templates
│   └── static/        # CSS, JS
├── schemas.py         # Pydantic models + serialize_finding helper
├── config.py           # env-var configuration dataclass
├── constants.py       # all magic strings/limits
├── i18n.py            # RU/EN translation dictionaries
├── logging_setup.py   # JSON logging + syslog
└── main.py            # FastAPI app, middleware, lifepan

Data flow:

  1. Nexus sends UPDATED webhook → POST /webhooks/nexus
  2. webhooks.py validates signature, detects ecosystem, rejects unknown ecosystems
  3. harvester.py downloads file (async via asyncio.to_thread), validates URL against NEXUS_ALLOWED_HOSTS (SSRF protection), computes SHA256, deduplicates
  4. scanner.py runs guarddog <ecosystem> scan <file> --output-format json
  5. Findings stored in SQLite (scans + findings tables)
  6. If LLM_ENABLED=1 and LLM_AUTO_ANALYZE=1, llm.py sends each finding to the configured model with retry logic. finding.report state machine: None → {"status": "analyzing"} → {verdict, summary, analysis, severity_rating} or None on failure. LLM response validated with defaults for missing fields.

Key conventions

  • Python ≥ 3.10 — type hints with | None syntax, no Optional
  • Imports: absolute from guarddog_nexus.<module>; relative (..) within subpackages
  • Line length: 100 (ruff)
  • Lint: ruff check guarddog_nexus tests (E/F/I/W rules)
  • Format: ruff format guarddog_nexus tests
  • Tests: pytest -v (135 tests, pytest-asyncio auto mode)
  • Commits: Russian descriptions, prefix convention: feat:, fix:, refactor:, docs:, ui:
  • No comments in code unless explicitly requested
  • Async I/O: file reads/writes wrapped in asyncio.to_thread() — never raw open() in async context

Configuration

All via environment variables, defined in config.py. Key ones:

Variable Default Notes
NEXUS_URL http://localhost:8081
NEXUS_ALLOWED_HOSTS host from NEXUS_URL comma-separated, SSRF protection
WEBHOOK_SECRET "" HMAC-SHA256 validation
MAX_CONCURRENT_SCANS 4 asyncio.Semaphore for guarddog processes
LLM_ENABLED 0 1 to enable analysis
LLM_AUTO_ANALYZE 0 1 to auto-trigger after scan; 0 = manual via UI button
LLM_API_KEY "" OpenAI-compatible key
LLM_MODEL gpt-4o-mini
LLM_MAX_CONCURRENT_ANALYSES 2 Semaphore for LLM calls
DATABASE_PATH data/guarddog.db

Full list in config.py.

Database

  • SQLite via aiosqlite async driver
  • Tables: scans, findings
  • Auto-migration in db/engine.py — _migrate() adds missing columns on startup
  • Indexes created in _ensure_indexes(): scans.status, scans.sha256, scans.package_name, scans.package_version, scans.flagged, scans.nexus_asset_url, findings.scan_id
  • Scan fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip
  • Finding fields: id, scan_id, data (JSON), report (JSON, nullable), created_at

LLM analysis

finding.report drives UI state:

Value UI
None Show "Analyze with LLM" button (manual mode only)
{"status": "analyzing"} Show spinner
{verdict:, summary:, ...} Show report + "Retry" link

Auto mode (LLM_AUTO_ANALYZE=1): analysis runs immediately after scan; button hidden. Manual mode (LLM_AUTO_ANALYZE=0): user clicks button; button visible for each finding.

Per-finding asyncio.Lock in web.py prevents concurrent analysis of the same finding. Retry passes ?retry=1 to bypass the idempotency guard.

Webhooks

Only UPDATED action is accepted (not CREATED). Format field in asset data determines ecosystem: pypi, go, npm. Unknown ecosystems are rejected explicitly (no silent fallback to pypi).

Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content.

Templates (htmx)

  • Fragment templates prefixed with _ are returned for HX-Request requests
  • Filter-bar lives outside htmx target — never replaced
  • Sortable columns use hx-get with all filter params in URL
  • Language persists via cookie lang, set by middleware
  • Shared includes: _status_badge.html (scan status), _pagination.html (page nav), _llm_spinner.html (LLM progress)

Docker

docker compose up -d --build   # build + start
docker compose down             # stop
docker compose down -v          # stop + destroy volumes (make docker-destroy)
docker compose logs -f          # tail logs

The Dockerfile uses uv pip install . --system to install the package and all dependencies from pyproject.toml. GuardDog is installed as a separate uv pip install step.

Testing

  • make test runs pytest -v
  • Tests use in-memory SQLite (:memory:)
  • conftest.py sets up os.environ before importing the app
  • Mock guarddog output via fixtures — no real CLI execution
  • 135 tests covering: API, webhooks, harvester, scanner, web UI, i18n, metrics, LLM analysis, e2e flows
  • E2E tests in tests/e2e/ cover full webhook-to-scan pipeline, API filtering/pagination, LLM analysis, and error handling

When adding features:

  • Always python3 -m pytest -v before committing
  • Always ruff check guarddog_nexus tests
  • Add tests for new extractors, endpoints, edge cases

Common tasks

Add a new ecosystem:

  1. Add extractor in core/nexus.py → EXTRACTORS dict
  2. Add format handler in routes/webhooks.py → _detect_ecosystem()
  3. Create proxy repo in scripts/setup-nexus.sh
  4. Add test fixture in tests/conftest.py
  5. Add tests in tests/test_nexus.py

Add a new env var:

  1. config.py → field in Config dataclass
  2. .env.example → add with default
  3. docker-compose.yml → add to environment if needed in Docker
  4. README.md → add to env table

Add a UI string (i18n):

  1. i18n.py → add to _STRINGS dict
  2. Template → {{ t('key', request.state.lang) }}

Run a manual webhook test:

curl -X POST http://localhost:8080/webhooks/nexus \
  -H "Content-Type: application/json" \
  -d '{"action":"UPDATED","repositoryName":"pypi-proxy",
       "asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz",
       "downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}'

Notes

  • AI-generated code: all code in this repository was generated by an AI assistant (Claude). Review carefully before production use.
  • No Nexus Pro required: the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins.
  • GuardDog deadlocks: GuardDog is CPU-intensive. Use MAX_CONCURRENT_SCANS to avoid resource exhaustion.
  • LLM may be slow: increase LLM_TIMEOUT_SECONDS for large models. Set LLM_MAX_CONCURRENT_ANALYSES to limit parallel requests.

Workflow — MANDATORY after completing a feature or session

Before responding to the user, you MUST complete ALL of:

  1. Lint — ruff check guarddog_nexus tests (must pass) + ruff format guarddog_nexus tests
  2. Test — python3 -m pytest -v (must pass 100%)
  3. Commit — git add -A && git commit -m "prefix: description" using the existing prefix convention (feat:, fix:, refactor:, docs:, ui:)
  4. Rebuild — docker compose up -d --build
  5. Document — update AGENTS.md if the change introduces a new concept, env var, endpoint, or workflow

If you skip any of these, the user will need to do them manually. Do NOT skip commit and rebuild.

These steps must be executed sequentially — lint before test, test before commit, commit before rebuild.