docs: AGENTS.md + AI-нота в README + английский README.en.md

- AGENTS.md: документация для разработчиков (архитектура, конвенции, common tasks, testing, Docker, webhooks) - README.md: добавлена AI-поме тка в конце - README.en.md: полная английская версия документации
2026-05-10 07:40:26 +03:00
parent d33411b719
commit f108464828
3 changed files with 375 additions and 0 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -0,0 +1,199 @@
+# AGENTS.md — GuardDog Nexus
+
+## Project overview
+
+GuardDog Nexus integrates [GuardDog](https://github.com/DataDog/guarddog) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis.
+
+**Stack:** Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose.
+
+**Package ecosystems:** PyPI, Go (proxy.golang.org), npm (registry.npmjs.org).
+
+**GuardDog binary:** installed inside the Docker image via `uv pip install --system guarddog`. Supports `pypi`, `go`, `npm` subcommands.
+
+---
+
+## Quick start
+
+```bash
+cp .env.example .env
+# edit .env to set NEXUS_PASSWORD, optionally LLM vars
+make docker-up
+# → guarddog-nexus :8080, Nexus :8081
+```
+
+For local development without Docker:
+```bash
+make install dev
+export $(cat .env | xargs)
+python -m guarddog_nexus.main
+make test     # 50 tests
+make lint     # ruff
+make format   # ruff format + fix
+```
+
+---
+
+## Architecture
+
+```
+guarddog_nexus/
+├── core/              # Business logic
+│   ├── scanner.py     # subprocess guarddog CLI
+│   ├── harvester.py   # download → sha256 → scan → store
+│   ├── nexus.py       # httpx client + pypi/go/npm path extractors
+│   └── llm.py         # OpenAI-compatible analysis client
+├── db/                # Persistence
+│   ├── engine.py      # async SQLAlchemy + auto-migration
+│   ├── models.py      # Scan, Finding ORM
+│   └── queries.py     # shared query builders
+├── routes/            # HTTP layer
+│   ├── webhooks.py    # POST /webhooks/nexus
+│   ├── api_scans.py   # REST /api/v1/scans
+│   ├── api_packages.py
+│   ├── api_findings.py
+│   ├── metrics.py     # GET /metrics (Prometheus)
+│   └── web.py         # HTML UI (Jinja2 + htmx)
+├── web/               # Static assets
+│   ├── templates/     # Jinja2 templates
+│   └── static/        # CSS, JS
+├── config.py          # env-var configuration dataclass
+├── constants.py       # all magic strings/limits
+├── i18n.py            # RU/EN translation dictionaries
+├── logging_setup.py   # JSON logging + syslog
+└── main.py            # FastAPI app, middleware, lifepan
+```
+
+**Data flow:**
+1. Nexus sends `UPDATED` webhook → `POST /webhooks/nexus`
+2. `webhooks.py` validates signature, extracts asset info, spawns background task
+3. `harvester.py` downloads file, computes SHA256, deduplicates
+4. `scanner.py` runs `guarddog <ecosystem> scan <file> --output-format json`
+5. Findings stored in SQLite (`scans` + `findings` tables)
+6. If `LLM_ENABLED=1`, `llm.py` sends each finding to the configured model, stores report in `findings.report`
+
+---
+
+## Key conventions
+
+- **Python ≥ 3.10** — type hints with `| None` syntax, no `Optional`
+- **Imports:** absolute from `guarddog_nexus.<module>`; relative (`..`) within subpackages
+- **Line length:** 100 (ruff)
+- **Lint:** `ruff check guarddog_nexus tests` (E/F/I/W rules)
+- **Format:** `ruff format guarddog_nexus tests`
+- **Tests:** `pytest -v` (50 tests, pytest-asyncio auto mode)
+- **Commits:** Russian descriptions, prefix convention: `feat:`, `fix:`, `refactor:`, `docs:`, `ui:`
+- **No comments** in code unless explicitly requested
+
+---
+
+## Configuration
+
+All via environment variables, defined in `config.py`. Key ones:
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `NEXUS_URL` | `http://localhost:8081` | |
+| `NEXUS_PASSWORD` | — | Required |
+| `WEBHOOK_SECRET` | `""` | HMAC-SHA256 validation |
+| `MAX_CONCURRENT_SCANS` | `4` | asyncio.Semaphore for guarddog processes |
+| `LLM_ENABLED` | `0` | `1` to enable analysis |
+| `LLM_API_KEY` | `""` | OpenAI-compatible key |
+| `LLM_MODEL` | `gpt-4o-mini` | |
+| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Semaphore for LLM calls |
+| `DATABASE_PATH` | `data/guarddog.db` | |
+
+Full list in `config.py`.
+
+---
+
+## Database
+
+- **SQLite** via `aiosqlite` async driver
+- Tables: `scans`, `findings`
+- Auto-migration in `db/engine.py` — `_migrate()` adds missing columns on startup
+- `Scan` fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip
+- `Finding` fields: id, scan_id, data (JSON), report (JSON, nullable), created_at
+
+---
+
+## Webhooks
+
+Only `UPDATED` action is accepted (not `CREATED`). Format field in asset data determines ecosystem: `pypi`, `go`, `npm`.
+
+Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content.
+
+---
+
+## Templates (htmx)
+
+- Fragment templates prefixed with `_` are returned for `HX-Request` requests
+- Filter-bar lives outside htmx target — never replaced
+- Sortable columns use `hx-get` with all filter params in URL
+- Language persists via cookie `lang`, set by middleware
+
+---
+
+## Docker
+
+```
+docker compose up -d --build   # build + start
+docker compose down             # stop
+docker compose down -v          # stop + destroy volumes (make docker-destroy)
+docker compose logs -f          # tail logs
+```
+
+The Dockerfile parses `pyproject.toml` for dependency list (single source of truth). GuardDog is installed as a separate `uv pip install` step.
+
+---
+
+## Testing
+
+- `make test` runs `pytest -v`
+- Tests use in-memory SQLite (`:memory:`)
+- `conftest.py` sets up `os.environ` before importing the app
+- Mock `guarddog` output via fixtures — no real CLI execution
+- 50 tests covering: API, webhooks, harvester, scanner, web UI
+
+When adding features:
+- Always `python3 -m pytest -v` before committing
+- Always `ruff check guarddog_nexus tests`
+- Add tests for new extractors, endpoints, edge cases
+
+---
+
+## Common tasks
+
+**Add a new ecosystem:**
+1. Add extractor in `core/nexus.py` → `EXTRACTORS` dict
+2. Add format handler in `routes/webhooks.py` → `_detect_ecosystem()`
+3. Create proxy repo in `scripts/setup-nexus.sh`
+4. Add test fixture in `tests/conftest.py`
+5. Add tests in `tests/test_nexus.py`
+
+**Add a new env var:**
+1. `config.py` → field in Config dataclass
+2. `.env.example` → add with default
+3. `docker-compose.yml` → add to environment if needed in Docker
+4. `README.md` → add to env table
+
+**Add a UI string (i18n):**
+1. `i18n.py` → add to `_STRINGS` dict
+2. Template → `{{ t('key', request.state.lang) }}`
+
+**Run a manual webhook test:**
+```bash
+curl -X POST http://localhost:8080/webhooks/nexus \
+  -H "Content-Type: application/json" \
+  -d '{"action":"UPDATED","repositoryName":"pypi-proxy",
+       "asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz",
+       "downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}'
+```
+
+---
+
+## Notes
+
+- **AI-generated code:** all code in this repository was generated by an AI assistant (Claude). Review carefully before production use.
+- **No Nexus Pro required:** the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins.
+- **GuardDog deadlocks:** GuardDog is CPU-intensive. Use `MAX_CONCURRENT_SCANS` to avoid resource exhaustion.
+- **LLM may be slow:** increase `LLM_TIMEOUT_SECONDS` for large models. Set `LLM_MAX_CONCURRENT_ANALYSES` to limit parallel requests.
--- a/README.en.md
+++ b/README.en.md
@@ -0,0 +1,172 @@
+# GuardDog Nexus
+
+Integration of [GuardDog](https://github.com/DataDog/guarddog) (package vulnerability scanner) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). Automatically scans packages stored in Nexus for vulnerabilities, malware, and suspicious patterns via webhooks.
+
+## Features
+
+- **Automatic scanning** via Nexus webhooks on package cache updates
+- **Multi-ecosystem support** — PyPI, Go, npm (any format via proxy repositories)
+- **REST API** for scan results, findings, statistics, and CSV export
+- **Web dashboard** with scan tables, filtering, and LLM-powered analysis
+- **LLM analysis** — automated security analysis of each finding via OpenAI-compatible APIs (optional, configurable)
+- **Deduplication** by URL and SHA256 — identical content scanned only once
+- **Structured JSON logging** with optional syslog output
+- **Docker Compose** — full stack deployment with Nexus in one command
+
+## Architecture
+
+```
+Nexus ──(webhook)──> GuardDog Nexus ──(REST API)──> Web UI
+                         │
+                         ├──> GuardDog CLI (scanning)
+                         ├──> LLM API (finding analysis)
+                         ├──> SQLite (result storage)
+                         └──> REST API (data for UI + CSV export)
+```
+
+## Quick Start
+
+### Prerequisites
+
+- Docker and Docker Compose
+- Python 3.10+ (for local development)
+
+### Docker Deployment
+
+```bash
+cp .env.example .env
+# edit .env: NEXUS_PASSWORD, optionally LLM_* vars
+
+make docker-up
+```
+
+After startup:
+
+| Service | URL | Port |
+|---------|-----|------|
+| GuardDog Nexus | http://localhost:8080 | 8080 |
+| Sonatype Nexus | http://localhost:8081 | 8081 |
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `NEXUS_URL` | `http://localhost:8081` | Sonatype Nexus URL |
+| `NEXUS_USERNAME` | `admin` | Nexus username |
+| `NEXUS_PASSWORD` | _(required)_ | Nexus password |
+| `DATABASE_PATH` | `data/guarddog.db` | SQLite database path |
+| `HOST` | `0.0.0.0` | Listen host |
+| `PORT` | `8080` | Listen port |
+| `LOG_LEVEL` | `INFO` | Logging level |
+| `WEBHOOK_SECRET` | _(empty)_ | HMAC-SHA256 webhook validation secret |
+| `SCAN_TIMEOUT_SECONDS` | `300` | Scan timeout per package |
+| `TEMP_DIR` | `/tmp/guarddog-nexus` | Temporary download directory |
+| `MAX_CONCURRENT_SCANS` | `4` | Maximum simultaneous GuardDog processes |
+| `LLM_ENABLED` | `0` | Set to `1` to enable LLM analysis |
+| `LLM_API_KEY` | _(empty)_ | API key (OpenAI / Groq / Ollama / etc.) |
+| `LLM_API_BASE` | `https://api.openai.com/v1` | OpenAI-compatible base URL |
+| `LLM_MODEL` | `gpt-4o-mini` | Model name |
+| `LLM_TIMEOUT_SECONDS` | `30` | LLM request timeout |
+| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Max parallel LLM analysis calls |
+
+## Nexus Configuration
+
+### Proxy Repositories
+
+The setup script (`scripts/setup-nexus.sh`) creates three proxy repositories:
+
+- `pypi-proxy` → `https://pypi.org`
+- `go-proxy` → `https://proxy.golang.org`
+- `npm-proxy` → `https://registry.npmjs.org`
+
+### Webhooks
+
+GuardDog Nexus accepts `UPDATED` webhook events from Nexus.
+
+1. In Nexus admin panel: **System → Capabilities**
+2. Create **Webhook: Repository** capability
+3. URL: `http://guarddog-nexus:8080/webhooks/nexus`
+4. Event type: **Repository → Asset → Updated**
+5. Repository filter: `pypi-proxy`, `go-proxy`, `npm-proxy`
+6. Set `WEBHOOK_SECRET` in `.env` for HMAC-SHA256 signature validation
+
+> For local testing without real webhooks, POST manually:
+> ```bash
+> curl -X POST http://localhost:8080/webhooks/nexus \
+>   -H "Content-Type: application/json" \
+>   -d '{"action":"UPDATED","repositoryName":"pypi-proxy","asset":{"format":"pypi","name":"/packages/pkg/1.0/pkg-1.0.tar.gz","downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/1.0/pkg-1.0.tar.gz"}}'
+> ```
+
+## REST API
+
+### Scans
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/v1/scans` | List scans (pagination, filters) |
+| GET | `/api/v1/scans/stats` | Statistics |
+| GET | `/api/v1/scans/{id}` | Scan detail with findings |
+| GET | `/api/v1/scans/export` | Export scans to CSV |
+
+### Packages
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/v1/packages` | List aggregated packages |
+| GET | `/api/v1/packages/{name}/{version}` | Package detail with scans and findings |
+| GET | `/api/v1/packages/export` | Export packages to CSV |
+
+### Findings
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/v1/findings` | List findings (filter by rule, severity, scan_id) |
+| POST | `/api/v1/findings/{id}/analyze` | Trigger LLM analysis for a finding |
+
+### Other
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/health` | Health check |
+| GET | `/metrics` | Prometheus-compatible metrics |
+
+## LLM Analysis
+
+GuardDog Nexus can automatically analyze each finding through an LLM. When enabled (`LLM_ENABLED=1`), every flagged scan gets an AI breakdown: threat assessment, code analysis, and recommendations.
+
+**Auto mode:** after a flagged scan completes, each finding is sent to the LLM. Reports are saved to the database and included in JSON log output.
+
+**Manual mode:** the web UI has an "Analyze with LLM" button next to each finding — click to get an inline verdict.
+
+Supported providers: any OpenAI-compatible API (OpenAI, Groq, Ollama, vLLM, etc.).
+
+LLM response format (JSON):
+- `verdict` — `safe` / `suspicious` / `malicious`
+- `summary` — one-line verdict
+- `analysis` — detailed breakdown (2–3 paragraphs)
+- `severity_rating` — `low` / `medium` / `high` / `critical`
+
+## Project Structure
+
+```
+guarddog-nexus/
+├── guarddog_nexus/           # Main package
+│   ├── core/                 # Business logic (scanner, harvester, nexus, llm)
+│   ├── db/                   # Database (engine, models, queries)
+│   ├── routes/               # HTTP layer (webhooks, api, web, metrics)
+│   ├── web/                  # Static assets (templates, CSS, JS)
+│   ├── config.py             # Environment configuration
+│   ├── constants.py          # Centralized constants
+│   ├── i18n.py               # RU/EN translations
+│   ├── logging_setup.py      # JSON structured logging
+│   └── main.py               # FastAPI app entry point
+├── tests/                    # pytest tests (50+)
+├── scripts/                  # Setup scripts
+├── docker-compose.yml
+├── Dockerfile
+└── pyproject.toml
+```
+
+## License
+
+MIT
+
+---
+
+> **⚠ All code in this repository was generated by an AI assistant (Claude).** Review carefully before using in production.
--- a/README.md
+++ b/README.md
@@ -271,3 +271,7 @@ LLM возвращает JSON с полями:
 ## Лицензия

 MIT
+
+---
+
+> **⚠ Весь код в этом репозитории сгенерирован AI-ассистентом (Claude).** Тщательно проверяйте код перед использованием в production-окружении.