283 lines
14 KiB
Markdown
283 lines
14 KiB
Markdown
# AGENTS.md — GuardDog Nexus
|
|
|
|
## Project overview
|
|
|
|
GuardDog Nexus integrates [GuardDog](https://github.com/DataDog/guarddog) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis.
|
|
|
|
**Stack:** Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose.
|
|
|
|
**Package ecosystems:** PyPI, Go (proxy.golang.org), npm (registry.npmjs.org).
|
|
|
|
**GuardDog binary:** installed inside the Docker image via `uv pip install --system guarddog`. Supports `pypi`, `go`, `npm` subcommands.
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# edit .env to set LLM vars if needed
|
|
make docker-up
|
|
# → guarddog-nexus :8080, Nexus :8081
|
|
```
|
|
|
|
For local development without Docker:
|
|
```bash
|
|
make install dev
|
|
export $(cat .env | xargs)
|
|
python -m guarddog_nexus.main
|
|
make test # 168 tests
|
|
make lint # ruff
|
|
make format # ruff format + fix
|
|
```
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
guarddog_nexus/
|
|
├── core/ # Business logic
|
|
│ ├── scanner.py # subprocess guarddog CLI
|
|
│ ├── harvester.py # download → sha256 → scan → store
|
|
│ ├── nexus.py # httpx client + pypi/go/npm path extractors
|
|
│ └── llm.py # OpenAI-compatible analysis client
|
|
├── db/ # Persistence
|
|
│ ├── engine.py # async SQLAlchemy + auto-migration
|
|
│ ├── models.py # Scan, Finding ORM
|
|
│ └── queries.py # shared query builders
|
|
├── routes/ # HTTP layer
|
|
│ ├── webhooks.py # POST /webhooks/nexus
|
|
│ ├── api_scans.py # REST /api/v1/scans
|
|
│ ├── api_packages.py
|
|
│ ├── api_findings.py
|
|
│ ├── metrics.py # GET /metrics (Prometheus)
|
|
│ └── web.py # HTML UI (Jinja2 + htmx)
|
|
├── web/ # Static assets
|
|
│ ├── templates/ # Jinja2 templates
|
|
│ └── static/ # CSS, JS
|
|
├── schemas.py # Pydantic models + serialize_finding() helper
|
|
├── config.py # env-var configuration dataclass + _env_int()
|
|
├── constants.py # all magic strings/limits + SUPPORTED_ECOSYSTEMS
|
|
├── i18n.py # RU/EN translation dictionaries
|
|
├── logging_setup.py # JSON logging + syslog
|
|
└── main.py # FastAPI app, middleware, lifespan, /health/dependencies
|
|
```
|
|
|
|
**Data flow:**
|
|
1. Nexus sends `UPDATED` webhook → `POST /webhooks/nexus`
|
|
2. `webhooks.py` validates signature, detects ecosystem, rejects unknown ecosystems
|
|
3. `harvester.py` downloads file (async via `asyncio.to_thread`), validates URL against `NEXUS_ALLOWED_HOSTS` (SSRF protection), computes SHA256, deduplicates
|
|
4. `scanner.py` runs `guarddog <ecosystem> scan <file> --output-format json`
|
|
5. Findings stored in SQLite (`scans` + `findings` tables)
|
|
6. If `LLM_ENABLED=1` and `LLM_AUTO_ANALYZE=1`, `llm.py` sends findings to the configured model in parallel via `asyncio.gather` (respects `LLM_MAX_CONCURRENT_ANALYSES`). Retry logic with exponential backoff (2s, 4s, 8s, max 3 attempts). `finding.report` state machine: `None` → `{"status": "analyzing"}` → `{verdict, summary, analysis, severity_rating}` or `None` on failure. LLM response validated via `_validate_report()` which applies defaults for missing fields (`verdict→unknown`, `severity_rating→unknown`, etc.). Progress tracked via Prometheus counters: `guarddog_llm_analyzed_total` and `guarddog_llm_pending_total`.
|
|
|
|
---
|
|
|
|
## Key conventions
|
|
|
|
- **Python ≥ 3.10** — type hints with `| None` syntax, no `Optional`
|
|
- **Imports:** absolute from `guarddog_nexus.<module>`; relative (`..`) within subpackages
|
|
- **Line length:** 100 (ruff)
|
|
- **Lint:** `ruff check guarddog_nexus tests` (E/F/I/W rules)
|
|
- **Format:** `ruff format guarddog_nexus tests`
|
|
- **Tests:** `pytest -v` (168 tests, pytest-asyncio auto mode)
|
|
- **Commits:** Russian descriptions, prefix convention: `feat:`, `fix:`, `refactor:`, `docs:`, `ui:`
|
|
- **No comments** in code unless explicitly requested
|
|
- **Async I/O:** file reads/writes wrapped in `asyncio.to_thread()` — never raw `open()` in async context
|
|
- **Config validation:** `_env_int` logs a warning on invalid values instead of crashing
|
|
- **Type checking:** `make typecheck` runs `mypy guarddog_nexus` (strict mode)
|
|
- **Pre-commit:** `.pre-commit-config.yaml` with ruff, ruff-format, trailing-whitespace, end-of-file-fixer, check-yaml, check-toml, check-added-large-files, detect-private-key
|
|
- **CSV export:** `_csv_safe()` in `api_scans.py` prepends `'` to values starting with `=`, `+`, `-`, `@` — blocks formula injection when opening CSV in spreadsheet apps
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
All via environment variables, defined in `config.py`. Key ones:
|
|
|
|
| Variable | Default | Notes |
|
|
|----------|---------|-------|
|
|
| `NEXUS_URL` | `http://localhost:8081` | |
|
|
| `NEXUS_ALLOWED_HOSTS` | host from `NEXUS_URL` | comma-separated, SSRF protection |
|
|
| `WEBHOOK_SECRET` | `""` | HMAC-SHA256 validation |
|
|
| `MAX_CONCURRENT_SCANS` | `4` | asyncio.Semaphore for guarddog processes |
|
|
| `SCAN_TIMEOUT_SECONDS` | `300` | per-package scan timeout |
|
|
| `GUARDDOG_BINARY` | `guarddog` | path to GuardDog CLI |
|
|
| `NEXUS_DOWNLOAD_TIMEOUT_SECONDS` | `120` | download timeout from Nexus |
|
|
| `NEXUS_API_TIMEOUT_SECONDS` | `30` | Nexus REST API timeout |
|
|
| `LLM_ENABLED` | `0` | `1` to enable analysis |
|
|
| `LLM_AUTO_ANALYZE` | `0` | `1` to auto-trigger after scan; `0` = manual via UI button |
|
|
| `LLM_API_KEY` | `""` | OpenAI-compatible key |
|
|
| `LLM_API_BASE` | `https://api.openai.com/v1` | OpenAI-compatible base URL |
|
|
| `LLM_MODEL` | `gpt-4o-mini` | |
|
|
| `LLM_TIMEOUT_SECONDS` | `30` | LLM request timeout |
|
|
| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Semaphore for LLM calls |
|
|
| `DATABASE_PATH` | `data/guarddog.db` | |
|
|
|
|
Full list in `config.py`.
|
|
|
|
---
|
|
|
|
## Database
|
|
|
|
- **SQLite** via `aiosqlite` async driver
|
|
- Tables: `scans`, `findings`
|
|
- Auto-migration in `db/engine.py` — `_migrate()` adds missing columns on startup
|
|
- Indexes created in `_ensure_indexes()`: `scans.status`, `scans.sha256`, `scans.package_name`, `scans.package_version`, `scans.flagged`, `scans.nexus_asset_url`, `findings.scan_id`
|
|
- `Scan` fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip
|
|
- `Finding` fields: id, scan_id, data (JSON), report (JSON, nullable), created_at
|
|
|
|
---
|
|
|
|
## LLM analysis
|
|
|
|
`finding.report` drives UI state:
|
|
|
|
| Value | UI |
|
|
|-------|----|
|
|
| `None` | Show "Analyze with LLM" button (manual mode only) |
|
|
| `{"status": "analyzing"}` | Show spinner (auto-polls via HTMX GET every 2s) |
|
|
| `{verdict:, summary:, ...}` | Show report + "Retry" link |
|
|
|
|
**Auto mode** (`LLM_AUTO_ANALYZE=1`): analysis runs immediately after scan; button hidden.
|
|
**Manual mode** (`LLM_AUTO_ANALYZE=0`): user clicks button; button visible for each finding.
|
|
|
|
Per-finding `asyncio.Lock` in `web.py` prevents concurrent analysis of the same finding. Retry passes `?retry=1` to bypass the idempotency guard. LLM response validated via `_validate_report()` — missing/invalid fields get defaults (`verdict→unknown`, `severity_rating→unknown`, etc.). Retry with exponential backoff: 2s, 4s, 8s (max 3 attempts). Reports can also be unwrapped from markdown code fences (```json ... ```).
|
|
|
|
---
|
|
|
|
## Webhooks
|
|
|
|
Only `UPDATED` action is accepted (not `CREATED`). Format field in asset data determines ecosystem: `pypi`, `go`, `npm`. Unknown ecosystems are rejected explicitly (no silent fallback to pypi).
|
|
|
|
Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content.
|
|
|
|
---
|
|
|
|
## Templates (htmx)
|
|
|
|
- Fragment templates prefixed with `_` are returned for `HX-Request` requests
|
|
- Filter-bar lives outside htmx target — never replaced
|
|
- Sortable columns use `hx-get` with all filter params in URL
|
|
- Language persists via cookie `lang`, set by middleware
|
|
- Shared includes: `_status_badge.html` (scan status), `_pagination.html` (page nav), `_llm_spinner.html` (LLM progress)
|
|
|
|
---
|
|
|
|
## Docker
|
|
|
|
```
|
|
docker compose up -d --build # build + start
|
|
docker compose down # stop
|
|
docker compose down -v # stop + destroy volumes (make docker-destroy)
|
|
docker compose logs -f # tail logs
|
|
```
|
|
|
|
The Dockerfile uses `uv pip install . --system` to install the package and all dependencies from `pyproject.toml`. GuardDog is installed as a separate `uv pip install --system "guarddog>=2.10.0"` step. Dependencies are installed before source code COPY for efficient layer caching. A `.dockerignore` excludes cache dirs, tests, and examples. Docker HEALTHCHECK at `/health` runs every 30 seconds.
|
|
|
|
Logging driver configured as `json-file` with rotation (max-size: 10m, max-file: 3) for both `guarddog-nexus` and `nexus` services. Nexus service also has a HEALTHCHECK (`curl` to `/service/rest/v1/status`).
|
|
|
|
---
|
|
|
|
## Makefile targets
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `make install` | Install project dependencies |
|
|
| `make dev` | Install dev dependencies |
|
|
| `make test` | Run pytest -v |
|
|
| `make lint` | Ruff check |
|
|
| `make format` | Ruff format + fix |
|
|
| `make typecheck` | mypy strict mode |
|
|
| `make check` | lint + format + typecheck + test |
|
|
| `make run` | Start the app via `python -m guarddog_nexus.main` |
|
|
| `make setup-env` | Copy `.env.example` → `.env` if missing |
|
|
| `make docker-build` | Build Docker image |
|
|
| `make docker-up` | Build + start stack (`up -d --build`) |
|
|
| `make docker-down` | Stop stack |
|
|
| `make docker-destroy` | Stop + destroy volumes (`-v`) |
|
|
| `make docker-rebuild` | Down + up --build |
|
|
| `make docker-logs` | Tail logs |
|
|
| `make docker-ps` | `docker compose ps` |
|
|
| `make docker-shell` | Exec bash in guarddog-nexus container |
|
|
| `make docker-restart` | Restart guarddog-nexus service |
|
|
| `make clean` | Remove build artifacts |
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
- `make test` runs `pytest -v`
|
|
- Tests use in-memory SQLite (`:memory:`)
|
|
- `conftest.py` sets up `os.environ` before importing the app
|
|
- Mock `guarddog` output via fixtures — no real CLI execution
|
|
- 168 tests covering: API, webhooks, harvester, scanner, web UI, i18n, metrics, LLM analysis, config, schemas, engine, e2e flows
|
|
- E2E tests in `tests/e2e/` cover full webhook-to-scan pipeline, API filtering/pagination, LLM analysis, and error handling
|
|
|
|
When adding features:
|
|
- Always `python3 -m pytest -v` before committing
|
|
- Always `ruff check guarddog_nexus tests`
|
|
- Add tests for new extractors, endpoints, edge cases
|
|
|
|
---
|
|
|
|
## Common tasks
|
|
|
|
**Add a new ecosystem:**
|
|
1. Add extractor in `core/nexus.py` → `EXTRACTORS` dict
|
|
2. Add format handler in `routes/webhooks.py` → `_detect_ecosystem()`
|
|
3. Create proxy repo in `scripts/setup-nexus.sh`
|
|
4. Add test fixture in `tests/conftest.py`
|
|
5. Add tests in `tests/test_nexus.py`
|
|
|
|
**Add a new env var:**
|
|
1. `config.py` → field in Config dataclass
|
|
2. `.env.example` → add with default
|
|
3. `docker-compose.yml` → add to environment if needed in Docker
|
|
4. `README.md` → add to env table
|
|
|
|
**Add a UI string (i18n):**
|
|
1. `i18n.py` → add to `_STRINGS` dict
|
|
2. Template → `{{ t('key', request.state.lang) }}`
|
|
|
|
**Run a manual webhook test:**
|
|
```bash
|
|
curl -X POST http://localhost:8080/webhooks/nexus \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"action":"UPDATED","repositoryName":"pypi-proxy",
|
|
"asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz",
|
|
"downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}'
|
|
```
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **AI-generated code:** all code in this repository was generated by an AI assistant (Claude). Review carefully before production use.
|
|
- **No Nexus Pro required:** the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins.
|
|
- **Anonymous Nexus access:** all Nexus REST API calls use anonymous access (no BasicAuth). Ensure your Nexus instance allows anonymous read access to repositories.
|
|
- **GuardDog deadlocks:** GuardDog is CPU-intensive. Use `MAX_CONCURRENT_SCANS` to avoid resource exhaustion.
|
|
- **LLM may be slow:** increase `LLM_TIMEOUT_SECONDS` for large models. Set `LLM_MAX_CONCURRENT_ANALYSES` to limit parallel requests.
|
|
- **Security headers:** `SecurityHeadersMiddleware` sets X-Content-Type-Options, X-Frame-Options, X-XSS-Protection, Referrer-Policy, and Permissions-Policy on all responses.
|
|
- **Background tasks:** URL lock and LLM lock cleanup tasks run every 30 minutes via the lifespan; they are gracefully cancelled on shutdown.
|
|
- **`serialize_finding()` helper** in `schemas.py` prevents `**f.data` field injection in API responses by extracting only known fields.
|
|
- **`SUPPORTED_ECOSYSTEMS`** constant in `constants.py` defines the accepted ecosystem set (`pypi`, `go`, `npm`).
|
|
- **`/_health/dependencies`** endpoint checks database connectivity and Nexus API reachability.
|
|
|
|
---
|
|
|
|
## Workflow — MANDATORY after completing a feature or session
|
|
|
|
**Before responding to the user, you MUST complete ALL of:**
|
|
|
|
1. **Lint** — `ruff check guarddog_nexus tests` (must pass) + `ruff format guarddog_nexus tests`
|
|
2. **Test** — `python3 -m pytest -v` (must pass 100%)
|
|
3. **Commit** — `git add -A && git commit -m "prefix: description"` using the existing prefix convention (`feat:`, `fix:`, `refactor:`, `docs:`, `ui:`)
|
|
4. **Rebuild** — `docker compose up -d --build`
|
|
5. **Document** — update `AGENTS.md` if the change introduces a new concept, env var, endpoint, or workflow
|
|
|
|
**If you skip any of these, the user will need to do them manually. Do NOT skip commit and rebuild.**
|
|
|
|
These steps must be executed sequentially — lint before test, test before commit, commit before rebuild.
|