From f108464828a7ec68963cef921d1e1a941ccce77c Mon Sep 17 00:00:00 2001 From: Marker689 Date: Sun, 10 May 2026 07:40:26 +0300 Subject: [PATCH] =?UTF-8?q?docs:=20AGENTS.md=20+=20AI-=D0=BD=D0=BE=D1=82?= =?UTF-8?q?=D0=B0=20=D0=B2=20README=20+=20=D0=B0=D0=BD=D0=B3=D0=BB=D0=B8?= =?UTF-8?q?=D0=B9=D1=81=D0=BA=D0=B8=D0=B9=20README.en.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - AGENTS.md: документация для разработчиков (архитектура, конвенции, common tasks, testing, Docker, webhooks) - README.md: добавлена AI-поме тка в конце - README.en.md: полная английская версия документации --- AGENTS.md | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++ README.en.md | 172 ++++++++++++++++++++++++++++++++++++++++++++ README.md | 4 ++ 3 files changed, 375 insertions(+) create mode 100644 AGENTS.md create mode 100644 README.en.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..e462658 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,199 @@ +# AGENTS.md — GuardDog Nexus + +## Project overview + +GuardDog Nexus integrates [GuardDog](https://github.com/DataDog/guarddog) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis. + +**Stack:** Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose. + +**Package ecosystems:** PyPI, Go (proxy.golang.org), npm (registry.npmjs.org). + +**GuardDog binary:** installed inside the Docker image via `uv pip install --system guarddog`. Supports `pypi`, `go`, `npm` subcommands. + +--- + +## Quick start + +```bash +cp .env.example .env +# edit .env to set NEXUS_PASSWORD, optionally LLM vars +make docker-up +# → guarddog-nexus :8080, Nexus :8081 +``` + +For local development without Docker: +```bash +make install dev +export $(cat .env | xargs) +python -m guarddog_nexus.main +make test # 50 tests +make lint # ruff +make format # ruff format + fix +``` + +--- + +## Architecture + +``` +guarddog_nexus/ +├── core/ # Business logic +│ ├── scanner.py # subprocess guarddog CLI +│ ├── harvester.py # download → sha256 → scan → store +│ ├── nexus.py # httpx client + pypi/go/npm path extractors +│ └── llm.py # OpenAI-compatible analysis client +├── db/ # Persistence +│ ├── engine.py # async SQLAlchemy + auto-migration +│ ├── models.py # Scan, Finding ORM +│ └── queries.py # shared query builders +├── routes/ # HTTP layer +│ ├── webhooks.py # POST /webhooks/nexus +│ ├── api_scans.py # REST /api/v1/scans +│ ├── api_packages.py +│ ├── api_findings.py +│ ├── metrics.py # GET /metrics (Prometheus) +│ └── web.py # HTML UI (Jinja2 + htmx) +├── web/ # Static assets +│ ├── templates/ # Jinja2 templates +│ └── static/ # CSS, JS +├── config.py # env-var configuration dataclass +├── constants.py # all magic strings/limits +├── i18n.py # RU/EN translation dictionaries +├── logging_setup.py # JSON logging + syslog +└── main.py # FastAPI app, middleware, lifepan +``` + +**Data flow:** +1. Nexus sends `UPDATED` webhook → `POST /webhooks/nexus` +2. `webhooks.py` validates signature, extracts asset info, spawns background task +3. `harvester.py` downloads file, computes SHA256, deduplicates +4. `scanner.py` runs `guarddog scan --output-format json` +5. Findings stored in SQLite (`scans` + `findings` tables) +6. If `LLM_ENABLED=1`, `llm.py` sends each finding to the configured model, stores report in `findings.report` + +--- + +## Key conventions + +- **Python ≥ 3.10** — type hints with `| None` syntax, no `Optional` +- **Imports:** absolute from `guarddog_nexus.`; relative (`..`) within subpackages +- **Line length:** 100 (ruff) +- **Lint:** `ruff check guarddog_nexus tests` (E/F/I/W rules) +- **Format:** `ruff format guarddog_nexus tests` +- **Tests:** `pytest -v` (50 tests, pytest-asyncio auto mode) +- **Commits:** Russian descriptions, prefix convention: `feat:`, `fix:`, `refactor:`, `docs:`, `ui:` +- **No comments** in code unless explicitly requested + +--- + +## Configuration + +All via environment variables, defined in `config.py`. Key ones: + +| Variable | Default | Notes | +|----------|---------|-------| +| `NEXUS_URL` | `http://localhost:8081` | | +| `NEXUS_PASSWORD` | — | Required | +| `WEBHOOK_SECRET` | `""` | HMAC-SHA256 validation | +| `MAX_CONCURRENT_SCANS` | `4` | asyncio.Semaphore for guarddog processes | +| `LLM_ENABLED` | `0` | `1` to enable analysis | +| `LLM_API_KEY` | `""` | OpenAI-compatible key | +| `LLM_MODEL` | `gpt-4o-mini` | | +| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Semaphore for LLM calls | +| `DATABASE_PATH` | `data/guarddog.db` | | + +Full list in `config.py`. + +--- + +## Database + +- **SQLite** via `aiosqlite` async driver +- Tables: `scans`, `findings` +- Auto-migration in `db/engine.py` — `_migrate()` adds missing columns on startup +- `Scan` fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip +- `Finding` fields: id, scan_id, data (JSON), report (JSON, nullable), created_at + +--- + +## Webhooks + +Only `UPDATED` action is accepted (not `CREATED`). Format field in asset data determines ecosystem: `pypi`, `go`, `npm`. + +Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content. + +--- + +## Templates (htmx) + +- Fragment templates prefixed with `_` are returned for `HX-Request` requests +- Filter-bar lives outside htmx target — never replaced +- Sortable columns use `hx-get` with all filter params in URL +- Language persists via cookie `lang`, set by middleware + +--- + +## Docker + +``` +docker compose up -d --build # build + start +docker compose down # stop +docker compose down -v # stop + destroy volumes (make docker-destroy) +docker compose logs -f # tail logs +``` + +The Dockerfile parses `pyproject.toml` for dependency list (single source of truth). GuardDog is installed as a separate `uv pip install` step. + +--- + +## Testing + +- `make test` runs `pytest -v` +- Tests use in-memory SQLite (`:memory:`) +- `conftest.py` sets up `os.environ` before importing the app +- Mock `guarddog` output via fixtures — no real CLI execution +- 50 tests covering: API, webhooks, harvester, scanner, web UI + +When adding features: +- Always `python3 -m pytest -v` before committing +- Always `ruff check guarddog_nexus tests` +- Add tests for new extractors, endpoints, edge cases + +--- + +## Common tasks + +**Add a new ecosystem:** +1. Add extractor in `core/nexus.py` → `EXTRACTORS` dict +2. Add format handler in `routes/webhooks.py` → `_detect_ecosystem()` +3. Create proxy repo in `scripts/setup-nexus.sh` +4. Add test fixture in `tests/conftest.py` +5. Add tests in `tests/test_nexus.py` + +**Add a new env var:** +1. `config.py` → field in Config dataclass +2. `.env.example` → add with default +3. `docker-compose.yml` → add to environment if needed in Docker +4. `README.md` → add to env table + +**Add a UI string (i18n):** +1. `i18n.py` → add to `_STRINGS` dict +2. Template → `{{ t('key', request.state.lang) }}` + +**Run a manual webhook test:** +```bash +curl -X POST http://localhost:8080/webhooks/nexus \ + -H "Content-Type: application/json" \ + -d '{"action":"UPDATED","repositoryName":"pypi-proxy", + "asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz", + "downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}' +``` + +--- + +## Notes + +- **AI-generated code:** all code in this repository was generated by an AI assistant (Claude). Review carefully before production use. +- **No Nexus Pro required:** the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins. +- **GuardDog deadlocks:** GuardDog is CPU-intensive. Use `MAX_CONCURRENT_SCANS` to avoid resource exhaustion. +- **LLM may be slow:** increase `LLM_TIMEOUT_SECONDS` for large models. Set `LLM_MAX_CONCURRENT_ANALYSES` to limit parallel requests. diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..c0b4ac8 --- /dev/null +++ b/README.en.md @@ -0,0 +1,172 @@ +# GuardDog Nexus + +Integration of [GuardDog](https://github.com/DataDog/guarddog) (package vulnerability scanner) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). Automatically scans packages stored in Nexus for vulnerabilities, malware, and suspicious patterns via webhooks. + +## Features + +- **Automatic scanning** via Nexus webhooks on package cache updates +- **Multi-ecosystem support** — PyPI, Go, npm (any format via proxy repositories) +- **REST API** for scan results, findings, statistics, and CSV export +- **Web dashboard** with scan tables, filtering, and LLM-powered analysis +- **LLM analysis** — automated security analysis of each finding via OpenAI-compatible APIs (optional, configurable) +- **Deduplication** by URL and SHA256 — identical content scanned only once +- **Structured JSON logging** with optional syslog output +- **Docker Compose** — full stack deployment with Nexus in one command + +## Architecture + +``` +Nexus ──(webhook)──> GuardDog Nexus ──(REST API)──> Web UI + │ + ├──> GuardDog CLI (scanning) + ├──> LLM API (finding analysis) + ├──> SQLite (result storage) + └──> REST API (data for UI + CSV export) +``` + +## Quick Start + +### Prerequisites + +- Docker and Docker Compose +- Python 3.10+ (for local development) + +### Docker Deployment + +```bash +cp .env.example .env +# edit .env: NEXUS_PASSWORD, optionally LLM_* vars + +make docker-up +``` + +After startup: + +| Service | URL | Port | +|---------|-----|------| +| GuardDog Nexus | http://localhost:8080 | 8080 | +| Sonatype Nexus | http://localhost:8081 | 8081 | + +## Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `NEXUS_URL` | `http://localhost:8081` | Sonatype Nexus URL | +| `NEXUS_USERNAME` | `admin` | Nexus username | +| `NEXUS_PASSWORD` | _(required)_ | Nexus password | +| `DATABASE_PATH` | `data/guarddog.db` | SQLite database path | +| `HOST` | `0.0.0.0` | Listen host | +| `PORT` | `8080` | Listen port | +| `LOG_LEVEL` | `INFO` | Logging level | +| `WEBHOOK_SECRET` | _(empty)_ | HMAC-SHA256 webhook validation secret | +| `SCAN_TIMEOUT_SECONDS` | `300` | Scan timeout per package | +| `TEMP_DIR` | `/tmp/guarddog-nexus` | Temporary download directory | +| `MAX_CONCURRENT_SCANS` | `4` | Maximum simultaneous GuardDog processes | +| `LLM_ENABLED` | `0` | Set to `1` to enable LLM analysis | +| `LLM_API_KEY` | _(empty)_ | API key (OpenAI / Groq / Ollama / etc.) | +| `LLM_API_BASE` | `https://api.openai.com/v1` | OpenAI-compatible base URL | +| `LLM_MODEL` | `gpt-4o-mini` | Model name | +| `LLM_TIMEOUT_SECONDS` | `30` | LLM request timeout | +| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Max parallel LLM analysis calls | + +## Nexus Configuration + +### Proxy Repositories + +The setup script (`scripts/setup-nexus.sh`) creates three proxy repositories: + +- `pypi-proxy` → `https://pypi.org` +- `go-proxy` → `https://proxy.golang.org` +- `npm-proxy` → `https://registry.npmjs.org` + +### Webhooks + +GuardDog Nexus accepts `UPDATED` webhook events from Nexus. + +1. In Nexus admin panel: **System → Capabilities** +2. Create **Webhook: Repository** capability +3. URL: `http://guarddog-nexus:8080/webhooks/nexus` +4. Event type: **Repository → Asset → Updated** +5. Repository filter: `pypi-proxy`, `go-proxy`, `npm-proxy` +6. Set `WEBHOOK_SECRET` in `.env` for HMAC-SHA256 signature validation + +> For local testing without real webhooks, POST manually: +> ```bash +> curl -X POST http://localhost:8080/webhooks/nexus \ +> -H "Content-Type: application/json" \ +> -d '{"action":"UPDATED","repositoryName":"pypi-proxy","asset":{"format":"pypi","name":"/packages/pkg/1.0/pkg-1.0.tar.gz","downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/1.0/pkg-1.0.tar.gz"}}' +> ``` + +## REST API + +### Scans +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/v1/scans` | List scans (pagination, filters) | +| GET | `/api/v1/scans/stats` | Statistics | +| GET | `/api/v1/scans/{id}` | Scan detail with findings | +| GET | `/api/v1/scans/export` | Export scans to CSV | + +### Packages +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/v1/packages` | List aggregated packages | +| GET | `/api/v1/packages/{name}/{version}` | Package detail with scans and findings | +| GET | `/api/v1/packages/export` | Export packages to CSV | + +### Findings +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/v1/findings` | List findings (filter by rule, severity, scan_id) | +| POST | `/api/v1/findings/{id}/analyze` | Trigger LLM analysis for a finding | + +### Other +| Method | Path | Description | +|--------|------|-------------| +| GET | `/health` | Health check | +| GET | `/metrics` | Prometheus-compatible metrics | + +## LLM Analysis + +GuardDog Nexus can automatically analyze each finding through an LLM. When enabled (`LLM_ENABLED=1`), every flagged scan gets an AI breakdown: threat assessment, code analysis, and recommendations. + +**Auto mode:** after a flagged scan completes, each finding is sent to the LLM. Reports are saved to the database and included in JSON log output. + +**Manual mode:** the web UI has an "Analyze with LLM" button next to each finding — click to get an inline verdict. + +Supported providers: any OpenAI-compatible API (OpenAI, Groq, Ollama, vLLM, etc.). + +LLM response format (JSON): +- `verdict` — `safe` / `suspicious` / `malicious` +- `summary` — one-line verdict +- `analysis` — detailed breakdown (2–3 paragraphs) +- `severity_rating` — `low` / `medium` / `high` / `critical` + +## Project Structure + +``` +guarddog-nexus/ +├── guarddog_nexus/ # Main package +│ ├── core/ # Business logic (scanner, harvester, nexus, llm) +│ ├── db/ # Database (engine, models, queries) +│ ├── routes/ # HTTP layer (webhooks, api, web, metrics) +│ ├── web/ # Static assets (templates, CSS, JS) +│ ├── config.py # Environment configuration +│ ├── constants.py # Centralized constants +│ ├── i18n.py # RU/EN translations +│ ├── logging_setup.py # JSON structured logging +│ └── main.py # FastAPI app entry point +├── tests/ # pytest tests (50+) +├── scripts/ # Setup scripts +├── docker-compose.yml +├── Dockerfile +└── pyproject.toml +``` + +## License + +MIT + +--- + +> **⚠ All code in this repository was generated by an AI assistant (Claude).** Review carefully before using in production. diff --git a/README.md b/README.md index 3b2bf2e..395d8f4 100644 --- a/README.md +++ b/README.md @@ -271,3 +271,7 @@ LLM возвращает JSON с полями: ## Лицензия MIT + +--- + +> **⚠ Весь код в этом репозитории сгенерирован AI-ассистентом (Claude).** Тщательно проверяйте код перед использованием в production-окружении.