marker/guarddog-nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6ea5c85a4be58401e8798cd1d3059a637a385fc7
guarddog-nexus/AGENTS.md

14 KiB
Raw Blame History

AGENTS.md — GuardDog Nexus

Project overview

GuardDog Nexus integrates GuardDog with Sonatype Nexus Repository Manager. It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis.

Stack: Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose.

Package ecosystems: PyPI, Go (proxy.golang.org), npm (registry.npmjs.org).

GuardDog binary: installed inside the Docker image via uv pip install --system guarddog. Supports pypi, go, npm subcommands.

Quick start

cp .env.example .env
# edit .env to set LLM vars if needed
make docker-up
# → guarddog-nexus :8080, Nexus :8081

For local development without Docker:

make install dev
export $(cat .env | xargs)
python -m guarddog_nexus.main
make test     # 168 tests
make lint     # ruff
make format   # ruff format + fix

Architecture

guarddog_nexus/
├── core/              # Business logic
│   ├── scanner.py     # subprocess guarddog CLI
│   ├── harvester.py   # download → sha256 → scan → store
│   ├── nexus.py       # httpx client + pypi/go/npm path extractors
│   └── llm.py         # OpenAI-compatible analysis client
├── db/                # Persistence
│   ├── engine.py      # async SQLAlchemy + auto-migration
│   ├── models.py      # Scan, Finding ORM
│   └── queries.py     # shared query builders
├── routes/            # HTTP layer
│   ├── webhooks.py    # POST /webhooks/nexus
│   ├── api_scans.py   # REST /api/v1/scans
│   ├── api_packages.py
│   ├── api_findings.py
│   ├── metrics.py     # GET /metrics (Prometheus)
│   └── web.py         # HTML UI (Jinja2 + htmx)
├── web/               # Static assets
│   ├── templates/     # Jinja2 templates
│   └── static/        # CSS, JS
├── schemas.py         # Pydantic models + serialize_finding() helper
├── config.py          # env-var configuration dataclass + _env_int()
├── constants.py       # all magic strings/limits + SUPPORTED_ECOSYSTEMS
├── i18n.py            # RU/EN translation dictionaries
├── logging_setup.py   # JSON logging + syslog
└── main.py            # FastAPI app, middleware, lifespan, /health/dependencies

Data flow:

  1. Nexus sends UPDATED webhook → POST /webhooks/nexus
  2. webhooks.py validates signature, detects ecosystem, rejects unknown ecosystems
  3. harvester.py downloads file (async via asyncio.to_thread), validates URL against NEXUS_ALLOWED_HOSTS (SSRF protection), computes SHA256, deduplicates
  4. scanner.py runs guarddog <ecosystem> scan <file> --output-format json
  5. Findings stored in SQLite (scans + findings tables)
  6. If LLM_ENABLED=1 and LLM_AUTO_ANALYZE=1, llm.py sends findings to the configured model in parallel via asyncio.gather (respects LLM_MAX_CONCURRENT_ANALYSES). Retry logic with exponential backoff (2s, 4s, 8s, max 3 attempts). finding.report state machine: None → {"status": "analyzing"} → {verdict, summary, analysis, severity_rating} or None on failure. LLM response validated via _validate_report() which applies defaults for missing fields (verdict→unknown, severity_rating→unknown, etc.). Progress tracked via Prometheus counters: guarddog_llm_analyzed_total and guarddog_llm_pending_total.

Key conventions

  • Python ≥ 3.10 — type hints with | None syntax, no Optional
  • Imports: absolute from guarddog_nexus.<module>; relative (..) within subpackages
  • Line length: 100 (ruff)
  • Lint: ruff check guarddog_nexus tests (E/F/I/W rules)
  • Format: ruff format guarddog_nexus tests
  • Tests: pytest -v (168 tests, pytest-asyncio auto mode)
  • Commits: Russian descriptions, prefix convention: feat:, fix:, refactor:, docs:, ui:
  • No comments in code unless explicitly requested
  • Async I/O: file reads/writes wrapped in asyncio.to_thread() — never raw open() in async context
  • Config validation: _env_int logs a warning on invalid values instead of crashing
  • Type checking: make typecheck runs mypy guarddog_nexus (strict mode)
  • Pre-commit: .pre-commit-config.yaml with ruff, ruff-format, trailing-whitespace, end-of-file-fixer, check-yaml, check-toml, check-added-large-files, detect-private-key
  • CSV export: _csv_safe() in api_scans.py prepends ' to values starting with =, +, -, @ — blocks formula injection when opening CSV in spreadsheet apps

Configuration

All via environment variables, defined in config.py. Key ones:

Variable Default Notes
NEXUS_URL http://localhost:8081
NEXUS_ALLOWED_HOSTS host from NEXUS_URL comma-separated, SSRF protection
WEBHOOK_SECRET "" HMAC-SHA256 validation
MAX_CONCURRENT_SCANS 4 asyncio.Semaphore for guarddog processes
SCAN_TIMEOUT_SECONDS 300 per-package scan timeout
GUARDDOG_BINARY guarddog path to GuardDog CLI
NEXUS_DOWNLOAD_TIMEOUT_SECONDS 120 download timeout from Nexus
NEXUS_API_TIMEOUT_SECONDS 30 Nexus REST API timeout
LLM_ENABLED 0 1 to enable analysis
LLM_AUTO_ANALYZE 0 1 to auto-trigger after scan; 0 = manual via UI button
LLM_API_KEY "" OpenAI-compatible key
LLM_API_BASE https://api.openai.com/v1 OpenAI-compatible base URL
LLM_MODEL gpt-4o-mini
LLM_TIMEOUT_SECONDS 30 LLM request timeout
LLM_MAX_CONCURRENT_ANALYSES 2 Semaphore for LLM calls
DATABASE_PATH data/guarddog.db

Full list in config.py.

Database

  • SQLite via aiosqlite async driver
  • Tables: scans, findings
  • Auto-migration in db/engine.py — _migrate() adds missing columns on startup
  • Indexes created in _ensure_indexes(): scans.status, scans.sha256, scans.package_name, scans.package_version, scans.flagged, scans.nexus_asset_url, findings.scan_id
  • Scan fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip
  • Finding fields: id, scan_id, data (JSON), report (JSON, nullable), created_at

LLM analysis

finding.report drives UI state:

Value UI
None Show "Analyze with LLM" button (manual mode only)
{"status": "analyzing"} Show spinner (auto-polls via HTMX GET every 2s)
{verdict:, summary:, ...} Show report + "Retry" link

Auto mode (LLM_AUTO_ANALYZE=1): analysis runs immediately after scan; button hidden. Manual mode (LLM_AUTO_ANALYZE=0): user clicks button; button visible for each finding.

Per-finding asyncio.Lock in web.py prevents concurrent analysis of the same finding. Retry passes ?retry=1 to bypass the idempotency guard. LLM response validated via _validate_report() — missing/invalid fields get defaults (verdict→unknown, severity_rating→unknown, etc.). Retry with exponential backoff: 2s, 4s, 8s (max 3 attempts). Reports can also be unwrapped from markdown code fences (json ... ).

Webhooks

Only UPDATED action is accepted (not CREATED). Format field in asset data determines ecosystem: pypi, go, npm. Unknown ecosystems are rejected explicitly (no silent fallback to pypi).

Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content.

Templates (htmx)

  • Fragment templates prefixed with _ are returned for HX-Request requests
  • Filter-bar lives outside htmx target — never replaced
  • Sortable columns use hx-get with all filter params in URL
  • Language persists via cookie lang, set by middleware
  • Shared includes: _status_badge.html (scan status), _pagination.html (page nav), _llm_spinner.html (LLM progress)

Docker

docker compose up -d --build   # build + start
docker compose down             # stop
docker compose down -v          # stop + destroy volumes (make docker-destroy)
docker compose logs -f          # tail logs

The Dockerfile uses uv pip install . --system to install the package and all dependencies from pyproject.toml. GuardDog is installed as a separate uv pip install --system "guarddog>=2.10.0" step. Dependencies are installed before source code COPY for efficient layer caching. A .dockerignore excludes cache dirs, tests, and examples. Docker HEALTHCHECK at /health runs every 30 seconds.

Logging driver configured as json-file with rotation (max-size: 10m, max-file: 3) for both guarddog-nexus and nexus services. Nexus service also has a HEALTHCHECK (curl to /service/rest/v1/status).

Makefile targets

Command Description
make install Install project dependencies
make dev Install dev dependencies
make test Run pytest -v
make lint Ruff check
make format Ruff format + fix
make typecheck mypy strict mode
make check lint + format + typecheck + test
make run Start the app via python -m guarddog_nexus.main
make setup-env Copy .env.example → .env if missing
make docker-build Build Docker image
make docker-up Build + start stack (up -d --build)
make docker-down Stop stack
make docker-destroy Stop + destroy volumes (-v)
make docker-rebuild Down + up --build
make docker-logs Tail logs
make docker-ps docker compose ps
make docker-shell Exec bash in guarddog-nexus container
make docker-restart Restart guarddog-nexus service
make clean Remove build artifacts

Testing

  • make test runs pytest -v
  • Tests use in-memory SQLite (:memory:)
  • conftest.py sets up os.environ before importing the app
  • Mock guarddog output via fixtures — no real CLI execution
  • 168 tests covering: API, webhooks, harvester, scanner, web UI, i18n, metrics, LLM analysis, config, schemas, engine, e2e flows
  • E2E tests in tests/e2e/ cover full webhook-to-scan pipeline, API filtering/pagination, LLM analysis, and error handling

When adding features:

  • Always python3 -m pytest -v before committing
  • Always ruff check guarddog_nexus tests
  • Add tests for new extractors, endpoints, edge cases

Common tasks

Add a new ecosystem:

  1. Add extractor in core/nexus.py → EXTRACTORS dict
  2. Add format handler in routes/webhooks.py → _detect_ecosystem()
  3. Create proxy repo in scripts/setup-nexus.sh
  4. Add test fixture in tests/conftest.py
  5. Add tests in tests/test_nexus.py

Add a new env var:

  1. config.py → field in Config dataclass
  2. .env.example → add with default
  3. docker-compose.yml → add to environment if needed in Docker
  4. README.md → add to env table

Add a UI string (i18n):

  1. i18n.py → add to _STRINGS dict
  2. Template → {{ t('key', request.state.lang) }}

Run a manual webhook test:

curl -X POST http://localhost:8080/webhooks/nexus \
  -H "Content-Type: application/json" \
  -d '{"action":"UPDATED","repositoryName":"pypi-proxy",
       "asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz",
       "downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}'

Notes

  • AI-generated code: all code in this repository was generated by an AI assistant (Claude). Review carefully before production use.
  • No Nexus Pro required: the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins.
  • Anonymous Nexus access: all Nexus REST API calls use anonymous access (no BasicAuth). Ensure your Nexus instance allows anonymous read access to repositories.
  • GuardDog deadlocks: GuardDog is CPU-intensive. Use MAX_CONCURRENT_SCANS to avoid resource exhaustion.
  • LLM may be slow: increase LLM_TIMEOUT_SECONDS for large models. Set LLM_MAX_CONCURRENT_ANALYSES to limit parallel requests.
  • Security headers: SecurityHeadersMiddleware sets X-Content-Type-Options, X-Frame-Options, X-XSS-Protection, Referrer-Policy, and Permissions-Policy on all responses.
  • Background tasks: URL lock and LLM lock cleanup tasks run every 30 minutes via the lifespan; they are gracefully cancelled on shutdown.
  • serialize_finding() helper in schemas.py prevents **f.data field injection in API responses by extracting only known fields.
  • SUPPORTED_ECOSYSTEMS constant in constants.py defines the accepted ecosystem set (pypi, go, npm).
  • /_health/dependencies endpoint checks database connectivity and Nexus API reachability.

Workflow — MANDATORY after completing a feature or session

Before responding to the user, you MUST complete ALL of:

  1. Lint — ruff check guarddog_nexus tests (must pass) + ruff format guarddog_nexus tests
  2. Test — python3 -m pytest -v (must pass 100%)
  3. Commit — git add -A && git commit -m "prefix: description" using the existing prefix convention (feat:, fix:, refactor:, docs:, ui:)
  4. Rebuild — docker compose up -d --build
  5. Document — update AGENTS.md if the change introduces a new concept, env var, endpoint, or workflow

If you skip any of these, the user will need to do them manually. Do NOT skip commit and rebuild.

These steps must be executed sequentially — lint before test, test before commit, commit before rebuild.