diff --git a/.env b/.env index 821a69cb..7e073bd9 100644 --- a/.env +++ b/.env @@ -8,7 +8,7 @@ SHUFFLE_DOWNLOAD_WORKFLOW_USERNAME= SHUFFLE_DOWNLOAD_WORKFLOW_PASSWORD= SHUFFLE_DOWNLOAD_WORKFLOW_BRANCH= -SHUFFLE_APP_DOWNLOAD_LOCATION=https://github.com/frikky/shuffle-apps +SHUFFLE_APP_DOWNLOAD_LOCATION=https://github.com/frikky/shuffle-apps SHUFFLE_DOWNLOAD_AUTH_USERNAME= SHUFFLE_DOWNLOAD_AUTH_PASSWORD= SHUFFLE_DOWNLOAD_AUTH_BRANCH= @@ -40,7 +40,6 @@ OUTER_HOSTNAME=shuffle-backend DB_LOCATION=./shuffle-database DOCKER_API_VERSION=1.40 - # Proxy configurations. SHUFFLE_PASS_WORKER_PROXY must be FALSE to not pass the proxy information to sub-apps. # PS: It will skip proxy for HTTP_PROXY= diff --git a/.github/install-guide.html b/.github/install-guide.html new file mode 100644 index 00000000..45bf03e2 --- /dev/null +++ b/.github/install-guide.html @@ -0,0 +1,126 @@ + + + + + +

Installation guide

+

Installation of Shuffle is currently only available in docker. Looking for how to update Shuffle? Check the updating guide

+

Docker

+

The Docker setup is done with docker-compose and is a single command to get set up.

+

PS: if you're setting up Shuffle on Windows, go to the next step (Windows Docker setup)

+
    +
  1. Make sure you have Docker and docker-compose installed.
    2. +
  2. Adapt the .env file to your convenience.
    3. +
  3. Run docker-compose.
    git clone https://github.com/frikky/Shuffle
+cd Shuffle
+docker-compose up -d
    4. +
+

When you're done, skip to the "After installation" step below.

+

Windows Docker setup

+

This step is for setting up with Docker on windows from scratch.

+
    +
  1. Make sure you have Docker and docker-compose installed. WSL2 may be required.
    2. +
  2. Go to https://github.com/frikky/shuffle/releases and download the latest .zip release (or install git)
    3. +
  3. Unzip the folder and enter it
    4. +
  4. Open the .env file and change the line with "OUTER_HOSTNAME" to contain your IP:
    OUTER_HOSTNAME=YOUR.IP.HERE
    5. +
  5. Run docker-compose
    docker-compose up -d
    6. +
+

Configurations (proxies, default users etc.)

+

https://shuffler.io/docs/configuration

+

After installation

+
    +

  1. After installation, go to http://localhost:3001/adminsetup (or your servername)

    +
    2. +

  2. Now set up your admin account (username & password). Shuffle doesn't have a default username and password.

    +
    3. +
  3. Check out https://shuffler.io/docs/configuration as it has a lot of useful information to get started
    4. +
+

Admin account setup

+

Useful info

+ +

Execution problems

+

If you have problems with your first execution (hello world), you might need to set the correct Docker API version. Here's how:

+
    +
  1. Find your API version by running "docker version" +` +$ docker version
    2. +
+

Client: + Version: 17.09.1-ce + API version: 1.32 # <-- this one + Go version: go1.8.3 + Git commit: 19e2cf6 + Built: Thu Dec 7 22:24:16 2017 + OS/Arch: linux/amd64

+

Server: + Version: 17.09.1-ce + API version: 1.32 (minimum version 1.12) + Go version: go1.8.3 + Git commit: 19e2cf6 + Built: Thu Dec 7 22:22:56 2017 + OS/Arch: linux/amd64 + Experimental: false

+

+2. Open docker-compose.yml and change the line with "DOCKER_API_VERSION" to your version.
+3. Restart docker-compose

docker-compose down +docker-compose up

+

+Related issue: #47
+
+# Local development installation
+Local development is pretty straight forward with **ReactJS** and **Golang**. This part is intended to help you run the code for development purposes.
+
+**PS: You have to stop the Backend Docker container to get this one working**
+**PPS: Use the "Launch" branch when developing to get it set up easier**
+
+## Frontend - ReactJS /w cytoscape
+http://localhost:3000 - Requires [npm](https://nodejs.org/en/download/)/[yarn](https://yarnpkg.com/lang/en/docs/install/#debian-stable)/your preferred manager. Runs independently from backend.
+```bash
+cd frontend
+npm i
+npm start

Backend - Golang

+

http://localhost:5001 - REST API - requires >=go1.13

+
export DATASTORE_EMULATOR_HOST=0.0.0.0:8000
+cd backend/go-app
+go run *.go
+

WINDOWS USERS: You'll have to to add the "export" part as an environment variable.

+

Database - Datastore

+

Based on Google datastore

+
docker run -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=shuffle --host-port 0.0.0.0:8000 --no-store-on-disk

Orborus

+

Execution of Workflows: +PS: This requires some specific environment variables

+
cd functions/onprem/orborus
+go run orborus.go

Environments (modify for Windows):

+
export ORG_ID=Shuffle
+export ENVIRONMENT_NAME=Shuffle
+export BASE_URL=http://YOUR-IP:5001
+export DOCKER_API_VERSION=1.40

WINDOWS USERS: You'll have to to add the "export" part as an environment variable.

+

AND THAT's it - hopefully it worked. If it didn't please email frikky@shuffler.io

+
+ +
+ \ No newline at end of file diff --git a/.github/install-guide.md b/.github/install-guide.md index fa3d1030..d1dfcf76 100644 --- a/.github/install-guide.md +++ b/.github/install-guide.md @@ -66,6 +66,7 @@ https://shuffler.io/docs/configuration * Default database location is in the same folder: ./shuffle-database # Local development installation + Local development is pretty straight forward with **ReactJS** and **Golang**. This part is intended to help you run the code for development purposes. We recommend having Shuffle running with the Docker-compose, then manually running the portion that you want to test and/or edit. **PS: You have to stop the Backend Docker container to get this one working** diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 727aa8d9..b82be082 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -13,10 +13,14 @@ name: "CodeQL" on: push: - branches: [ master ] + branches: + - master + - launch pull_request: # The branches below must be a subset of the branches above - branches: [ master ] + branches: + - master + - launch schedule: - cron: '38 16 * * 4' diff --git a/.github/workflows/snyk-container-analysis.yml b/.github/workflows/snyk-container-analysis.yml new file mode 100644 index 00000000..b3a78761 --- /dev/null +++ b/.github/workflows/snyk-container-analysis.yml @@ -0,0 +1,47 @@ +# A sample workflow which checks out the code, builds a container +# image using Docker and scans that image for vulnerabilities using +# Snyk. The results are then uploaded to GitHub Security Code Scanning +# +# For more examples, including how to limit scans to only high-severity +# issues, monitor images for newly disclosed vulnerabilities in Snyk and +# fail PR checks for new vulnerabilities, see https://github.com/snyk/actions/ + +name: Snyk Container + +on: + push: + branches: + - master + - launch + pull_request: + # The branches below must be a subset of the branches above + branches: + - master + - launch + schedule: + - cron: '18 4 * * 3' + +jobs: + snyk: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Build a Docker image + run: docker build -t your/image-to-test . + - name: Run Snyk to check Docker image for vulnerabilities + # Snyk can be used to break the build when it detects vulnerabilities. + # In this case we want to upload the issues to GitHub Code Scanning + continue-on-error: true + uses: snyk/actions/docker@master + env: + # In order to use the Snyk Action you will need to have a Snyk API token. + # More details in https://github.com/snyk/actions#getting-your-snyk-token + # or you can signup for free at https://snyk.io/login + SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} + with: + image: your/image-to-test + args: --file=Dockerfile + - name: Upload result to GitHub Code Scanning + uses: github/codeql-action/upload-sarif@v1 + with: + sarif_file: snyk.sarif diff --git a/.github/workflows/snyk-infrastructure-analysis.yml b/.github/workflows/snyk-infrastructure-analysis.yml new file mode 100644 index 00000000..4402487d --- /dev/null +++ b/.github/workflows/snyk-infrastructure-analysis.yml @@ -0,0 +1,46 @@ +# A sample workflow which checks out your Infrastructure as Code Configuration files, +# such as Kubernetes, Helm & Terraform and scans them for any security issues. +# The results are then uploaded to GitHub Security Code Scanning +# +# For more examples, including how to limit scans to only high-severity issues +# and fail PR checks, see https://github.com/snyk/actions/ + +name: Snyk Infrastructure as Code + +on: + push: + branches: + - master + - launch + pull_request: + # The branches below must be a subset of the branches above + branches: + - master + - launch + schedule: + - cron: '41 16 * * 2' + +jobs: + snyk: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Run Snyk to check configuration files for security issues + # Snyk can be used to break the build when it detects security issues. + # In this case we want to upload the issues to GitHub Code Scanning + continue-on-error: true + uses: snyk/actions/iac@master + env: + # In order to use the Snyk Action you will need to have a Snyk API token. + # More details in https://github.com/snyk/actions#getting-your-snyk-token + # or you can signup for free at https://snyk.io/login + SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} + with: + # Add the path to the configuration file that you would like to test. + # For example `deployment.yaml` for a Kubernetes deployment manifest + # or `main.tf` for a Terraform configuration file + file: your-file-to-test.yaml + - name: Upload result to GitHub Code Scanning + uses: github/codeql-action/upload-sarif@v1 + with: + sarif_file: snyk.sarif diff --git a/backend/app_sdk/Dockerfile b/backend/app_sdk/Dockerfile index 2bb4e4d3..25c9aa40 100644 --- a/backend/app_sdk/Dockerfile +++ b/backend/app_sdk/Dockerfile @@ -1,5 +1,6 @@ #FROM python:3.9.1-alpine as base FROM python:3.10.0-alpine as base + FROM base as builder RUN apk --no-cache add --update alpine-sdk libffi libffi-dev musl-dev openssl-dev tzdata coreutils diff --git a/docker-compose.yml b/docker-compose.yml index 069fedad..e804bb4f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,7 +2,7 @@ version: '3' services: frontend: #build: ./frontend - image: ghcr.io/frikky/shuffle-frontend:nightly + image: ghcr.io/frikky/shuffle-frontend:latest container_name: shuffle-frontend hostname: shuffle-frontend ports: @@ -17,7 +17,7 @@ services: - backend backend: #build: ./backend - image: ghcr.io/frikky/shuffle-backend:nightly + image: ghcr.io/frikky/shuffle-backend:latest container_name: shuffle-backend hostname: ${BACKEND_HOSTNAME} # Here for debugging: @@ -40,7 +40,7 @@ services: #- database orborus: #build: ./functions/onprem/orborus - image: ghcr.io/frikky/shuffle-orborus:nightly + image: ghcr.io/frikky/shuffle-orborus:latest container_name: shuffle-orborus hostname: shuffle-orborus networks: @@ -49,7 +49,7 @@ services: - /var/run/docker.sock:/var/run/docker.sock environment: - SHUFFLE_APP_SDK_VERSION=0.8.97 - - SHUFFLE_WORKER_VERSION=nightly + - SHUFFLE_WORKER_VERSION=latest - ORG_ID=${ORG_ID} - ENVIRONMENT_NAME=${ENVIRONMENT_NAME} - BASE_URL=http://${OUTER_HOSTNAME}:${BACKEND_PORT} diff --git a/functions/extensions/cortex-responders/Shuffle/requirements.txt b/functions/extensions/cortex-responders/Shuffle/requirements.txt new file mode 100644 index 00000000..6aabc3cf --- /dev/null +++ b/functions/extensions/cortex-responders/Shuffle/requirements.txt @@ -0,0 +1,2 @@ +cortexutils +requests diff --git a/functions/extensions/cortex-responders/Shuffle/shuffle.json b/functions/extensions/cortex-responders/Shuffle/shuffle.json index ef2610dd..db755b49 100644 --- a/functions/extensions/cortex-responders/Shuffle/shuffle.json +++ b/functions/extensions/cortex-responders/Shuffle/shuffle.json @@ -5,7 +5,7 @@ "url": "https://github.com/frikky/shuffle", "license": "AGPL-V3", "description": "Execute a workflow in Shuffle", - "dataTypeList": ["thehive:case", "thehive:alert"], + "dataTypeList": ["thehive:case", "thehive:alert", "thehive:case_artifact"], "command": "Shuffle/shuffle.py", "baseConfig": "Shuffle", "configurationItems": [ @@ -24,6 +24,14 @@ "multi": false, "required": true }, + { + "name": "verifyssl", + "description": "Verify SSL certificate", + "type": "boolean", + "multi": false, + "required": true, + "defaultValue": true + }, { "name": "workflow_id", "description": "The ID of the workflow to execute", diff --git a/functions/extensions/cortex-responders/Shuffle/shuffle.py b/functions/extensions/cortex-responders/Shuffle/shuffle.py index 0816ca53..db343230 100644 --- a/functions/extensions/cortex-responders/Shuffle/shuffle.py +++ b/functions/extensions/cortex-responders/Shuffle/shuffle.py @@ -1,28 +1,27 @@ - -#!/usr/bin/env python -# encoding: utf-8 +#!/usr/bin/env python3 +#encoding: utf-8 from cortexutils.responder import Responder import requests class Shuffle(Responder): - def __init__(self): - Responder.__init__(self) - self.api_key = self.get_param("config.api_key", "") - self.url = self.get_param("config.url", "") - self.workflow_id = self.get_param("config.workflow_id", "") + def __init__(self): + Responder.__init__(self) + self.api_key = self.get_param("config.api_key", "") + self.url = self.get_param("config.url", "") + self.workflow_id = self.get_param("config.workflow_id", "") + self.verify = self.get_param('config.verifyssl', True, None) - def run(self): - Responder.run(self) + def run(self): + Responder.run(self) + parsed_url = "%s/api/v1/workflows/%s/execute" % (self.url, self.workflow_id) + headers = { + "Authorization": "Bearer %s" % self.api_key, + "User-Agent": "Cortex-Analyzer" + } + requests.post(parsed_url, headers=headers,verify=self.verify) - parsed_url = "%s/api/v1/workflows/%s/execute" % (self.url, self.workflow_id) - headers = { - "Authorization": "Bearer %s" % self.api_key - } - requests.post(parsed_url, headers=headers) - - self.report({'message': 'message sent'}) + self.report({'message': 'message sent'}) if __name__ == '__main__': Shuffle().run() - diff --git a/functions/extensions/cortex-responders/Shuffle_Webhook/requirements.txt b/functions/extensions/cortex-responders/Shuffle_Webhook/requirements.txt new file mode 100644 index 00000000..6aabc3cf --- /dev/null +++ b/functions/extensions/cortex-responders/Shuffle_Webhook/requirements.txt @@ -0,0 +1,2 @@ +cortexutils +requests diff --git a/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.json b/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.json new file mode 100644 index 00000000..a6d3e9d8 --- /dev/null +++ b/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.json @@ -0,0 +1,28 @@ +{ + "name": "Shuffle_webhook", + "version": "1.0", + "author": "@azgaviperr", + "url": "https://github.com/frikky/shuffle", + "license": "AGPL-V3", + "description": "Execute a webhook in Shuffle", + "dataTypeList": ["thehive:case", "thehive:alert", "thehive:case_artifact"], + "command": "Shuffle_Webhook/shuffle_webhook.py", + "baseConfig": "Shuffle_Webhook", + "configurationItems": [ + { + "name": "webhook_url", + "description": "The URL to your shuffle instance", + "type": "string", + "multi": false, + "required": true + }, + { + "name": "verifyssl", + "description": "Verify SSL certificate", + "type": "boolean", + "multi": false, + "required": true, + "defaultValue": true + } + ] +} diff --git a/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.py b/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.py new file mode 100644 index 00000000..daed501c --- /dev/null +++ b/functions/extensions/cortex-responders/Shuffle_Webhook/shuffle_webhook.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +#encoding: utf-8 + +from cortexutils.responder import Responder +import requests + +class Shuffle(Responder): + def __init__(self): + Responder.__init__(self) + self.api_key = self.get_param("config.api_key", "") + self.webhook_url = self.get_param("config.webhook_url", "") + self.webhook_id = self.get_param("config.webhook_id", "") + self.verify = self.get_param('config.verifyssl', True, None) + self.data = self.get_param('data') + + def run(self): + Responder.run(self) + headers = { + "Content-Type": "application/json", + "Accept": "application/json", + "User-Agent": "Cortex-Analyzer" + } + requests.post(self.webhook_url, headers=headers,verify=self.verify, json=self.data) + + self.report({'message': 'message sent'}) + +if __name__ == '__main__': + Shuffle().run()