marker/guarddog-nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: AGENTS.md + AI-нота в README + английский README.en.md

Browse Source 
- AGENTS.md: документация для разработчиков (архитектура, конвенции,
  common tasks, testing, Docker, webhooks)
- README.md: добавлена AI-поме тка в конце
- README.en.md: полная английская версия документации
This commit is contained in:
Marker689
2026-05-10 07:40:26 +03:00
parent d33411b719
commit f108464828
3 changed files with 375 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

199
AGENTS.md Normal file
View File

@@ -0,0 +1,199 @@
# AGENTS.md — GuardDog Nexus
## Project overview
GuardDog Nexus integrates [GuardDog](https://github.com/DataDog/guarddog) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). It receives webhooks from Nexus, downloads packages from proxied repositories, scans them with GuardDog CLI, and stores results in SQLite. A web dashboard built with FastAPI + Jinja2 + htmx displays findings with optional LLM-based analysis.
**Stack:** Python 3.12, FastAPI, SQLAlchemy (async), aiosqlite, Jinja2, htmx, Docker Compose.
**Package ecosystems:** PyPI, Go (proxy.golang.org), npm (registry.npmjs.org).
**GuardDog binary:** installed inside the Docker image via `uv pip install --system guarddog`. Supports `pypi`, `go`, `npm` subcommands.
---
## Quick start
```bash
cp .env.example .env
# edit .env to set NEXUS_PASSWORD, optionally LLM vars
make docker-up
# → guarddog-nexus :8080, Nexus :8081
```
For local development without Docker:
```bash
make install dev
export $(cat .env | xargs)
python -m guarddog_nexus.main
make test # 50 tests
make lint # ruff
make format # ruff format + fix
```
---
## Architecture
```
guarddog_nexus/
├── core/ # Business logic
│ ├── scanner.py # subprocess guarddog CLI
│ ├── harvester.py # download → sha256 → scan → store
│ ├── nexus.py # httpx client + pypi/go/npm path extractors
│ └── llm.py # OpenAI-compatible analysis client
├── db/ # Persistence
│ ├── engine.py # async SQLAlchemy + auto-migration
│ ├── models.py # Scan, Finding ORM
│ └── queries.py # shared query builders
├── routes/ # HTTP layer
│ ├── webhooks.py # POST /webhooks/nexus
│ ├── api_scans.py # REST /api/v1/scans
│ ├── api_packages.py
│ ├── api_findings.py
│ ├── metrics.py # GET /metrics (Prometheus)
│ └── web.py # HTML UI (Jinja2 + htmx)
├── web/ # Static assets
│ ├── templates/ # Jinja2 templates
│ └── static/ # CSS, JS
├── config.py # env-var configuration dataclass
├── constants.py # all magic strings/limits
├── i18n.py # RU/EN translation dictionaries
├── logging_setup.py # JSON logging + syslog
└── main.py # FastAPI app, middleware, lifepan
```
**Data flow:**
1. Nexus sends `UPDATED` webhook → `POST /webhooks/nexus`
2. `webhooks.py` validates signature, extracts asset info, spawns background task
3. `harvester.py` downloads file, computes SHA256, deduplicates
4. `scanner.py` runs `guarddog <ecosystem> scan <file> --output-format json`
5. Findings stored in SQLite (`scans` + `findings` tables)
6. If `LLM_ENABLED=1`, `llm.py` sends each finding to the configured model, stores report in `findings.report`
---
## Key conventions
- **Python ≥ 3.10** — type hints with `| None` syntax, no `Optional`
- **Imports:** absolute from `guarddog_nexus.<module>`; relative (`..`) within subpackages
- **Line length:** 100 (ruff)
- **Lint:** `ruff check guarddog_nexus tests` (E/F/I/W rules)
- **Format:** `ruff format guarddog_nexus tests`
- **Tests:** `pytest -v` (50 tests, pytest-asyncio auto mode)
- **Commits:** Russian descriptions, prefix convention: `feat:`, `fix:`, `refactor:`, `docs:`, `ui:`
- **No comments** in code unless explicitly requested
---
## Configuration
All via environment variables, defined in `config.py`. Key ones:
| Variable | Default | Notes |
|----------|---------|-------|
| `NEXUS_URL` | `http://localhost:8081` | |
| `NEXUS_PASSWORD` | — | Required |
| `WEBHOOK_SECRET` | `""` | HMAC-SHA256 validation |
| `MAX_CONCURRENT_SCANS` | `4` | asyncio.Semaphore for guarddog processes |
| `LLM_ENABLED` | `0` | `1` to enable analysis |
| `LLM_API_KEY` | `""` | OpenAI-compatible key |
| `LLM_MODEL` | `gpt-4o-mini` | |
| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Semaphore for LLM calls |
| `DATABASE_PATH` | `data/guarddog.db` | |
Full list in `config.py`.
---
## Database
- **SQLite** via `aiosqlite` async driver
- Tables: `scans`, `findings`
- Auto-migration in `db/engine.py` — `_migrate()` adds missing columns on startup
- `Scan` fields: id, package_name, package_version, ecosystem, repository, nexus_asset_url, sha256, status, total_findings, flagged, started_at, finished_at, error_message, initiator, source_ip
- `Finding` fields: id, scan_id, data (JSON), report (JSON, nullable), created_at
---
## Webhooks
Only `UPDATED` action is accepted (not `CREATED`). Format field in asset data determines ecosystem: `pypi`, `go`, `npm`.
Per-URL locking (asyncio.Lock) prevents parallel scans of the same asset. SHA256 dedup prevents re-scanning identical file content.
---
## Templates (htmx)
- Fragment templates prefixed with `_` are returned for `HX-Request` requests
- Filter-bar lives outside htmx target — never replaced
- Sortable columns use `hx-get` with all filter params in URL
- Language persists via cookie `lang`, set by middleware
---
## Docker
```
docker compose up -d --build # build + start
docker compose down # stop
docker compose down -v # stop + destroy volumes (make docker-destroy)
docker compose logs -f # tail logs
```
The Dockerfile parses `pyproject.toml` for dependency list (single source of truth). GuardDog is installed as a separate `uv pip install` step.
---
## Testing
- `make test` runs `pytest -v`
- Tests use in-memory SQLite (`:memory:`)
- `conftest.py` sets up `os.environ` before importing the app
- Mock `guarddog` output via fixtures — no real CLI execution
- 50 tests covering: API, webhooks, harvester, scanner, web UI
When adding features:
- Always `python3 -m pytest -v` before committing
- Always `ruff check guarddog_nexus tests`
- Add tests for new extractors, endpoints, edge cases
---
## Common tasks
**Add a new ecosystem:**
1. Add extractor in `core/nexus.py` → `EXTRACTORS` dict
2. Add format handler in `routes/webhooks.py` → `_detect_ecosystem()`
3. Create proxy repo in `scripts/setup-nexus.sh`
4. Add test fixture in `tests/conftest.py`
5. Add tests in `tests/test_nexus.py`
**Add a new env var:**
1. `config.py` → field in Config dataclass
2. `.env.example` → add with default
3. `docker-compose.yml` → add to environment if needed in Docker
4. `README.md` → add to env table
**Add a UI string (i18n):**
1. `i18n.py` → add to `_STRINGS` dict
2. Template → `{{ t('key', request.state.lang) }}`
**Run a manual webhook test:**
```bash
curl -X POST http://localhost:8080/webhooks/nexus \
-H "Content-Type: application/json" \
-d '{"action":"UPDATED","repositoryName":"pypi-proxy",
"asset":{"format":"pypi","name":"/packages/pkg/ver/pkg-ver.tar.gz",
"downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/ver/pkg-ver.tar.gz"}}'
```
---
## Notes
- **AI-generated code:** all code in this repository was generated by an AI assistant (Claude). Review carefully before production use.
- **No Nexus Pro required:** the system works with Nexus OSS. Webhooks can be triggered manually or via community plugins.
- **GuardDog deadlocks:** GuardDog is CPU-intensive. Use `MAX_CONCURRENT_SCANS` to avoid resource exhaustion.
- **LLM may be slow:** increase `LLM_TIMEOUT_SECONDS` for large models. Set `LLM_MAX_CONCURRENT_ANALYSES` to limit parallel requests.

172
README.en.md Normal file
View File

@@ -0,0 +1,172 @@
# GuardDog Nexus
Integration of [GuardDog](https://github.com/DataDog/guarddog) (package vulnerability scanner) with [Sonatype Nexus Repository Manager](https://www.sonatype.com/products/nexus-repository). Automatically scans packages stored in Nexus for vulnerabilities, malware, and suspicious patterns via webhooks.
## Features
- **Automatic scanning** via Nexus webhooks on package cache updates
- **Multi-ecosystem support** — PyPI, Go, npm (any format via proxy repositories)
- **REST API** for scan results, findings, statistics, and CSV export
- **Web dashboard** with scan tables, filtering, and LLM-powered analysis
- **LLM analysis** — automated security analysis of each finding via OpenAI-compatible APIs (optional, configurable)
- **Deduplication** by URL and SHA256 — identical content scanned only once
- **Structured JSON logging** with optional syslog output
- **Docker Compose** — full stack deployment with Nexus in one command
## Architecture
```
Nexus ──(webhook)──> GuardDog Nexus ──(REST API)──> Web UI
│
├──> GuardDog CLI (scanning)
├──> LLM API (finding analysis)
├──> SQLite (result storage)
└──> REST API (data for UI + CSV export)
```
## Quick Start
### Prerequisites
- Docker and Docker Compose
- Python 3.10+ (for local development)
### Docker Deployment
```bash
cp .env.example .env
# edit .env: NEXUS_PASSWORD, optionally LLM_* vars
make docker-up
```
After startup:
| Service | URL | Port |
|---------|-----|------|
| GuardDog Nexus | http://localhost:8080 | 8080 |
| Sonatype Nexus | http://localhost:8081 | 8081 |
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NEXUS_URL` | `http://localhost:8081` | Sonatype Nexus URL |
| `NEXUS_USERNAME` | `admin` | Nexus username |
| `NEXUS_PASSWORD` | _(required)_ | Nexus password |
| `DATABASE_PATH` | `data/guarddog.db` | SQLite database path |
| `HOST` | `0.0.0.0` | Listen host |
| `PORT` | `8080` | Listen port |
| `LOG_LEVEL` | `INFO` | Logging level |
| `WEBHOOK_SECRET` | _(empty)_ | HMAC-SHA256 webhook validation secret |
| `SCAN_TIMEOUT_SECONDS` | `300` | Scan timeout per package |
| `TEMP_DIR` | `/tmp/guarddog-nexus` | Temporary download directory |
| `MAX_CONCURRENT_SCANS` | `4` | Maximum simultaneous GuardDog processes |
| `LLM_ENABLED` | `0` | Set to `1` to enable LLM analysis |
| `LLM_API_KEY` | _(empty)_ | API key (OpenAI / Groq / Ollama / etc.) |
| `LLM_API_BASE` | `https://api.openai.com/v1` | OpenAI-compatible base URL |
| `LLM_MODEL` | `gpt-4o-mini` | Model name |
| `LLM_TIMEOUT_SECONDS` | `30` | LLM request timeout |
| `LLM_MAX_CONCURRENT_ANALYSES` | `2` | Max parallel LLM analysis calls |
## Nexus Configuration
### Proxy Repositories
The setup script (`scripts/setup-nexus.sh`) creates three proxy repositories:
- `pypi-proxy` → `https://pypi.org`
- `go-proxy` → `https://proxy.golang.org`
- `npm-proxy` → `https://registry.npmjs.org`
### Webhooks
GuardDog Nexus accepts `UPDATED` webhook events from Nexus.
1. In Nexus admin panel: **System → Capabilities**
2. Create **Webhook: Repository** capability
3. URL: `http://guarddog-nexus:8080/webhooks/nexus`
4. Event type: **Repository → Asset → Updated**
5. Repository filter: `pypi-proxy`, `go-proxy`, `npm-proxy`
6. Set `WEBHOOK_SECRET` in `.env` for HMAC-SHA256 signature validation
> For local testing without real webhooks, POST manually:
> ```bash
> curl -X POST http://localhost:8080/webhooks/nexus \
> -H "Content-Type: application/json" \
> -d '{"action":"UPDATED","repositoryName":"pypi-proxy","asset":{"format":"pypi","name":"/packages/pkg/1.0/pkg-1.0.tar.gz","downloadUrl":"http://nexus:8081/repository/pypi-proxy/packages/pkg/1.0/pkg-1.0.tar.gz"}}'
> ```
## REST API
### Scans
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/scans` | List scans (pagination, filters) |
| GET | `/api/v1/scans/stats` | Statistics |
| GET | `/api/v1/scans/{id}` | Scan detail with findings |
| GET | `/api/v1/scans/export` | Export scans to CSV |
### Packages
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/packages` | List aggregated packages |
| GET | `/api/v1/packages/{name}/{version}` | Package detail with scans and findings |
| GET | `/api/v1/packages/export` | Export packages to CSV |
### Findings
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/findings` | List findings (filter by rule, severity, scan_id) |
| POST | `/api/v1/findings/{id}/analyze` | Trigger LLM analysis for a finding |
### Other
| Method | Path | Description |
|--------|------|-------------|
| GET | `/health` | Health check |
| GET | `/metrics` | Prometheus-compatible metrics |
## LLM Analysis
GuardDog Nexus can automatically analyze each finding through an LLM. When enabled (`LLM_ENABLED=1`), every flagged scan gets an AI breakdown: threat assessment, code analysis, and recommendations.
**Auto mode:** after a flagged scan completes, each finding is sent to the LLM. Reports are saved to the database and included in JSON log output.
**Manual mode:** the web UI has an "Analyze with LLM" button next to each finding — click to get an inline verdict.
Supported providers: any OpenAI-compatible API (OpenAI, Groq, Ollama, vLLM, etc.).
LLM response format (JSON):
- `verdict` — `safe` / `suspicious` / `malicious`
- `summary` — one-line verdict
- `analysis` — detailed breakdown (2–3 paragraphs)
- `severity_rating` — `low` / `medium` / `high` / `critical`
## Project Structure
```
guarddog-nexus/
├── guarddog_nexus/ # Main package
│ ├── core/ # Business logic (scanner, harvester, nexus, llm)
│ ├── db/ # Database (engine, models, queries)
│ ├── routes/ # HTTP layer (webhooks, api, web, metrics)
│ ├── web/ # Static assets (templates, CSS, JS)
│ ├── config.py # Environment configuration
│ ├── constants.py # Centralized constants
│ ├── i18n.py # RU/EN translations
│ ├── logging_setup.py # JSON structured logging
│ └── main.py # FastAPI app entry point
├── tests/ # pytest tests (50+)
├── scripts/ # Setup scripts
├── docker-compose.yml
├── Dockerfile
└── pyproject.toml
```
## License
MIT
---
> **⚠ All code in this repository was generated by an AI assistant (Claude).** Review carefully before using in production.

4
README.md
View File

@@ -271,3 +271,7 @@ LLM возвращает JSON с полями:
## Лицензия ## Лицензия
MIT MIT
---
> **⚠ Весь код в этом репозитории сгенерирован AI-ассистентом (Claude).** Тщательно проверяйте код перед использованием в production-окружении.