diff --git a/README.md b/README.md index 8f1b8a4b..04fefd6e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Shuffle -[Shuffle](https://shuffler.io) is an automation platform for your security stack (SOAR). It uses Docker for scaling and OpenAPI for integrations. Shuffle has the possibility to run across multiple isolated environments and tracks progress for you. +[Shuffle](https://shuffler.io) is the platform for automation needs (SOAR). It has thousands of premade integrations and uses open frameworks like OpenAPI to ease migration. The workflow editor is based on a no-code thought process to empower non-developer, and the app creator makes you able to inegrate any platform in minutes. -**It's in BETA**, but if you want to implement Shuffle - [get in touch](https://shuffler.io/contact), send a mail to [frikky@shuffler.io](mailto:frikky@shuffler.io) or on twitter [@frikkylikeme](https://twitter.com/frikkylikeme) +**It's in BETA** - [Get in touch](https://shuffler.io/contact), send a mail to [frikky@shuffler.io](mailto:frikky@shuffler.io) or poke me on twitter [@frikkylikeme](https://twitter.com/frikkylikeme) ![Example Shuffle webhook integration](shuffle_webhook.png) @@ -12,8 +12,8 @@ ## Related repositories * Apps: https://github.com/frikky/shuffle-apps * Workflows: https://github.com/frikky/shuffle-workflows (empty) +* Security OpenAPI apps: https://github.com/frikky/OpenAPI-security-definitions * Documentation: https://github.com/frikky/shuffle-docs -* OpenAPI apps: https://github.com/frikky/OpenAPI-security-definitions ## Documentation Documentation can be found on https://shuffler.io/docs/about or in your own instance. Currently lacking: @@ -21,113 +21,42 @@ Documentation can be found on https://shuffler.io/docs/about or in your own inst * Updates after migrating from SaaS to open source ## Features -* Premade workflows for TheHive and MISP +* Simple workflow automation editor * Premade apps for a number of security tools -* Simple workflow editor * App creator for [OpenAPI](https://github.com/frikky/OpenAPI-security-definitions) * Easy to learn Python library for custom apps ## In the works +* Premade workflows for security professionals * Dashboard - Statistics are implemented -* Debug view for manual executions * App versioning ## Support Open an issue on Github, or [join the gitter chat](https://gitter.im/Shuffle-SOAR/community). For other / private requests: [frikky@shuffler.io](mailto:frikky@shuffler.io) -### Setup - Local development -Frontend - requires [npm](https://nodejs.org/en/download/)/[yarn](https://yarnpkg.com/lang/en/docs/install/#debian-stable)/your preferred manager. Runs independently from backend - edit frontend/src/App.yaml (line 44~) from window.location.origin to http://YOUR IP:5001 -```bash -cd frontend -npm i -npm start -``` +## Website +https://shuffler.io -Backend - API calls - requires [>=go1.13](https://golang.org/dl/) -```bash -export DATASTORE_EMULATOR_HOST=0.0.0.0:8000 -cd backend/go-app -go build -go run *.go -``` +## License +All modular information related to Shuffle will be under MIT (anyone can use it for whatever purpose), with Shuffle itself using AGPLv3. -Database - Datastore: -``` -docker run -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=shuffle --host-port 0.0.0.0:8000 --no-store-on-disk -``` +Apps & App SDK: MIT +Shuffle backend: AGPLv3 -Orborus - Execution of Workflows: -PS: This requires some specific environment variables. -``` -cd functions/onprem/orborus -go run orborus.go -``` - - -### Project overview +### Project overview Below is the folder structure with a short explanation ```bash -├── README.md # :) -├── backend # Contains directly backend related code. Go with sh tests +├── README.md # What you're reading right now +├── backend # Contains backend related code. +│   ├── go-app # The backend golang webserver +│   ├── app_gen # Code for app generation outside the Shuffle platform +│ └── app_sdk # The SDK used for apps ├── frontend # Contains frontend code. ReactJS and cytoscape. Horrible code :) -├── app_gen # Contains code generation projects for OpenAPI or PythonLib -> Shuffler app ├── functions # Contains google cloud function code mainly. │   ├── static_baseline.py # Static code used by stitcher.go to generate code │   ├── stitcher.go # Attempts to stitch together an app - part of backend now │   ├── onprem # Code for onprem solutions │  │   ├── Orborus # Distributes execution locations │  │   ├── Worker # Runs a workflow -│ │   └── App_SDK # Backend of individual apps -│   └── triggers # Custom triggers used in https://shuffler.io/workflows └ docker-compose.yml # Used for deployments ``` - -# Technology -GCP was chosen because why not use the best thingies. "Serverless" \o/ -```bash -├── languages -│   ├── Go # I like go, which is why go. -│   ├── Python3.7 # 3.7 specifically because of f-strings and 2.7 deprecation in 2020 -│   ├── Javascript # Frontend stuff. Uses ReactJS + Cytoscape for visualization -│   ├── sh/Bash # Basic testing and some deployments -├── gcloud -│   ├── datastore # TODO: Move away from this -├── onprem -│   ├── Docker # Runs the same cloud functions. I didn't like the thought of proxies -``` - -# Migration -Shuffle was initially built for cloud and SaaS, and a lot broke when it was moved to local execution. -There will be a major overhaul to the backend specifically. I'm currently moving and updating the following: -- Create dockerfiles and a single runscript -- * App creator - (Cloud function -> Docker) -- * Workflows - Run workflows locally -- * App list - IMPORT EXISTING APPS -- * Dockerfiles - Load the ones that are in workflows with a new version -- * Docker-compose- Frontend, backend, db & orborus -- * Configuration - Write setup documentation - Did for docker -- * Remove orborus? Can deploy straight, but that would be weird - Won't do this yet -- * Full OpenAPI support with authentication schemes in App creator (not Oauth2 yet) -- * Change workflow name -- * User run statistics -- Extended result data usage, build json with answers, not just "from previous action" -- Workflows - IMPORT DEFAULT WORKFLOWS - Create some towards e.g. TheHive & MISP. -- Documentation - General documentation /docs rewrite -- API doc - 1. In Shuffle. 2. In e.g. python -- Fix scheduler -- Use variables in e.g. JSON body -- Add random secret to orborus - -``` -# 1. export DATASTORE_EMULATOR_HOST=0.0.0.0:8000 -# 2. docker run -p 8000:8000 google/cloud-sdk gcloud beta emulators datastore start --project=shuffle --host-port 0.0.0.0:8000 --no-store-on-disk -``` -* Mail: Use appengine and connect to sendmail - -# Update for users: -* Build from scratch (docker-compose) -* Push stable builds to Dockerhub - -### Migration issues: -* Some workflows where items have multiple parents don't work. -* Fix dummy.json (GCP config) - bypass this somehow. diff --git a/backend/app_sdk/app_base.py b/backend/app_sdk/app_base.py index 03a71e01..33fa9314 100644 --- a/backend/app_sdk/app_base.py +++ b/backend/app_sdk/app_base.py @@ -114,18 +114,29 @@ class AppBase: # 1. Find the action baseresult = "" + actionname_lower = actionname.lower() try: - if actionname.lower() == "exec": + if actionname_lower == "exec": baseresult = execution_data["execution_argument"] else: for result in execution_data["results"]: resultlabel = result["action"]["label"].replace(" ", "_", -1).lower() - if resultlabel.lower() == actionname.lower(): + if resultlabel.lower() == actionname_lower: baseresult = result["result"] break + + print("BEFORE VARIABLES!") + if len(baseresult) == 0: + print("Variables: %s" % execution_data["workflow"]["workflow_variables"]) + for variable in execution_data["workflow"]["workflow_variables"]: + variablename = variable["name"].replace(" ", "_", -1).lower() + + if variablename.lower() == actionname_lower: + baseresult = variable["value"] + break except KeyError as error: - print(f"Error: {error}") + print(f"KeyError in JSON: {error}") print(f"After first trycatch") @@ -187,7 +198,7 @@ class AppBase: self.logger.debug(f"\n\nHandle static data with JSON: {data}\n\n") actualitem = re.findall(match, data, re.MULTILINE) - self.logger.info("PARSED: %s" % actualitem) + self.logger.info("STATIC PARSED: %s" % actualitem) if len(actualitem) > 0: for replace in actualitem: try: @@ -238,7 +249,7 @@ class AppBase: self.logger.info("Fullname: %s" % fullname) actualitem = re.findall(match, fullname, re.MULTILINE) - self.logger.info("PARSED: %s" % actualitem) + self.logger.info("ACTION PARSED: %s" % actualitem) if len(actualitem) > 0: for replace in actualitem: try: diff --git a/backend/go-app/README.md b/backend/go-app/README.md deleted file mode 100644 index c882ced5..00000000 --- a/backend/go-app/README.md +++ /dev/null @@ -1,6 +0,0 @@ -## RUN -```bash -cd shaffuru/ -dev_appserver.py --port=5001 --host=0.0.0.0 --enable_host_checking=false . -``` - diff --git a/backend/go-app/codegen.go b/backend/go-app/codegen.go index d1cf6ac0..e995afea 100644 --- a/backend/go-app/codegen.go +++ b/backend/go-app/codegen.go @@ -909,7 +909,7 @@ func handleGet(swagger *openapi3.Swagger, api WorkflowApp, extraParameters []Wor } if param.Value.In == "path" { - //log.Printf("PATH!: %s", param.Value.Name) + log.Printf("PATH!: %s", param.Value.Name) parameters = append(parameters, param.Value.Name) //baseUrl = fmt.Sprintf("%s%s", baseUrl) } else if param.Value.In == "query" { diff --git a/backend/go-app/main.go b/backend/go-app/main.go index bac7e31a..0bfeeda9 100644 --- a/backend/go-app/main.go +++ b/backend/go-app/main.go @@ -2158,22 +2158,18 @@ func setUser(ctx context.Context, data *User) error { return nil } +// Used for testing only. Shouldn't impact production. func handleCors(resp http.ResponseWriter, request *http.Request) bool { + allowedOrigins := "http://localhost:3000" - // FIXME - this is to handle multiple frontends in test rofl - origin := request.Header["Origin"] resp.Header().Set("Vary", "Origin") - if len(origin) > 0 { - resp.Header().Set("Access-Control-Allow-Origin", origin[0]) - } else { - resp.Header().Set("Access-Control-Allow-Origin", "http://localhost:4201") - } - //resp.Header().Set("Access-Control-Allow-Origin", "http://localhost:8000") resp.Header().Set("Access-Control-Allow-Headers", "Content-Type, Accept, X-Requested-With, remember-me") resp.Header().Set("Access-Control-Allow-Methods", "POST, GET, PUT, DELETE") resp.Header().Set("Access-Control-Allow-Credentials", "true") + resp.Header().Set("Access-Control-Allow-Origin", allowedOrigins) if request.Method == "OPTIONS" { + resp.WriteHeader(200) resp.Write([]byte("OK")) return true diff --git a/backend/go-app/walkoff.go b/backend/go-app/walkoff.go index 5ba6c7f5..b9fde0ba 100644 --- a/backend/go-app/walkoff.go +++ b/backend/go-app/walkoff.go @@ -1182,7 +1182,6 @@ func saveWorkflow(resp http.ResponseWriter, request *http.Request) { //Actions []Action `json:"actions" datastore:"actions,noindex"` - log.Printf("Hello") body, err := ioutil.ReadAll(request.Body) if err != nil { log.Printf("Failed hook unmarshaling: %s", err) @@ -1191,7 +1190,6 @@ func saveWorkflow(resp http.ResponseWriter, request *http.Request) { return } - log.Printf("Hello2") var workflow Workflow err = json.Unmarshal([]byte(body), &workflow) //log.Printf(string(body)) @@ -1216,12 +1214,23 @@ func saveWorkflow(resp http.ResponseWriter, request *http.Request) { log.Println("Pre") for _, action := range workflow.Actions { allNodes = append(allNodes, action.ID) + if action.Environment == "" { resp.WriteHeader(401) resp.Write([]byte(fmt.Sprintf(`{"success": false, "reason": "An environment for %s is required"}`, action.Label))) return action.IsValid = true } + + // FIXME: Have a good way of tracking errors. ID's or similar. + if !action.IsValid { + resp.WriteHeader(401) + resp.Write([]byte(fmt.Sprintf(`{"success": false, "reason": "Node %s is invalid and needs to be remade."}`, action.Label))) + return + action.IsValid = true + action.Errors = []string{} + } + newActions = append(newActions, action) } @@ -2653,6 +2662,65 @@ func deleteWorkflowApp(resp http.ResponseWriter, request *http.Request) { private = true } + q := datastore.NewQuery("workflow") + var workflows []Workflow + _, err = dbclient.GetAll(ctx, q, &workflows) + if err != nil { + resp.WriteHeader(401) + resp.Write([]byte(`{"success": false, "reason": "}`)) + return + } + + for _, workflow := range workflows { + found := false + + newActions := []Action{} + for _, action := range workflow.Actions { + if action.AppName == app.Name && action.AppVersion == app.AppVersion { + found = true + action.Errors = append(action.Errors, "App has been deleted") + action.IsValid = false + } + + newActions = append(newActions, action) + } + + if found { + workflow.IsValid = false + workflow.Errors = append(workflow.Errors, fmt.Sprintf("App %s_%s has been deleted", app.Name, app.AppVersion)) + workflow.Actions = newActions + + for _, trigger := range workflow.Triggers { + log.Printf("TRIGGER: %#v", trigger) + //err = deleteSchedule(ctx, scheduleId) + //if err != nil { + // if strings.Contains(err.Error(), "Job not found") { + // resp.WriteHeader(200) + // resp.Write([]byte(fmt.Sprintf(`{"success": true}`))) + // } else { + // resp.WriteHeader(401) + // resp.Write([]byte(fmt.Sprintf(`{"success": false, "reason": "Failed stopping schedule"}`))) + // } + // return + //} + } + + err = setWorkflow(ctx, workflow, workflow.ID) + if err != nil { + log.Printf("Failed setting workflow when deleting app: %s", err) + continue + } else { + log.Printf("Set %s (%s) to have errors", workflow.ID, workflow.Name) + } + + } + + } + + //resp.WriteHeader(200) + //resp.Write([]byte(`{"success": true}`)) + //return + // Not really deleting it, just removing from user cache if private { log.Printf("Deleting private app") @@ -2675,6 +2743,7 @@ func deleteWorkflowApp(resp http.ResponseWriter, request *http.Request) { return } } else { + log.Printf("Deleting public app") err = DeleteKey(ctx, "workflowapp", fileId) if err != nil { diff --git a/backend/go-app/webapp b/backend/go-app/webapp deleted file mode 100755 index 400c09ac..00000000 Binary files a/backend/go-app/webapp and /dev/null differ diff --git a/docker-compose.yml b/docker-compose.yml index 0fbc7c41..72dccb0c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,7 @@ version: '3' services: frontend: - #build: ./frontend + build: ./frontend image: frikky/shuffle:frontend container_name: shuffle-frontend hostname: shuffle-frontend @@ -26,7 +26,7 @@ services: volumes: - ${DB_LOCATION}:/etc/shuffle backend: - #build: ./backend + build: ./backend image: frikky/shuffle:backend container_name: shuffle-backend hostname: ${BACKEND_HOSTNAME} diff --git a/frontend/src/Admin.js b/frontend/src/Admin.js index 5d2e8158..a0a1fa74 100644 --- a/frontend/src/Admin.js +++ b/frontend/src/Admin.js @@ -371,7 +371,7 @@ const Admin = (props) => { diff --git a/frontend/src/AngularWorkflow.js b/frontend/src/AngularWorkflow.js index 86fb1b6c..c1701c69 100644 --- a/frontend/src/AngularWorkflow.js +++ b/frontend/src/AngularWorkflow.js @@ -172,7 +172,11 @@ const AngularWorkflow = (props) => { const [workflowExecutions, setWorkflowExecutions] = React.useState([]); const unloadText = 'Are you sure you want to leave?' - useBeforeunload(() => unloadText) + useBeforeunload(() => { + if (!lastSaved) { + return unloadText + } + }) const [elements, setElements] = useState([]) const { start, stop } = useInterval({ @@ -810,7 +814,7 @@ const AngularWorkflow = (props) => { //console.log("ACTION: ", selectedAction) //console.log("APP: ", selectedApp) setSelectedAction({}) - //setSelectedApp({}) + setSelectedApp({}) //setSelectedTrigger({}) //setSelectedEdge({}) @@ -828,6 +832,7 @@ const AngularWorkflow = (props) => { const onEdgeSelect = (event) => { setRightSideBarOpen(true) + setLastSaved(false) const triggercheck = workflow.triggers.find(trigger => trigger.id === event.target.data()["source"]) if (triggercheck === undefined) { @@ -843,6 +848,8 @@ const AngularWorkflow = (props) => { const onNodeSelect = (event) => { const data = event.target.data() + console.log("NODE: ", data) + setLastSaved(false) //console.log(data) if (data.type === "ACTION") { @@ -949,7 +956,7 @@ const AngularWorkflow = (props) => { } const onNodeAdded = (event) => { - //setLastSaved(false) + setLastSaved(false) const node = event.target if (node.isNode() && cy.nodes().size() === 1) { @@ -960,6 +967,7 @@ const AngularWorkflow = (props) => { } const onEdgeRemoved = (event) => { + setLastSaved(false) const edge = event.target workflow.branches = workflow.branches.filter(a => a.id !== edge.data().id) @@ -980,6 +988,7 @@ const AngularWorkflow = (props) => { const onNodeRemoved = (event) => { const node = event.target const data = node.data() + setLastSaved(false) //var currentnode = cy.getElementById(data.id) //if (currentnode.length === 0) { @@ -2556,10 +2565,16 @@ const AngularWorkflow = (props) => { const appApiView = Object.getOwnPropertyNames(selectedAction).length > 0 && Object.getOwnPropertyNames(selectedApp).length > 0 ?
-
-
-

{selectedAction.app_name}

+
+
+

{selectedAction.app_name}

What are apps? + {selectedAction.errors !== null && selectedAction.errors.length > 0 ? +
+ Errors: {selectedAction.errors.join("\n")} +
+ : null + }
+ + + + + : null + const appsModalLoad = loadAppsModalOpen ? { {appView} {modalView} {appsModalLoad} + {deleteModal}
:
diff --git a/frontend/src/SettingsPage.js b/frontend/src/SettingsPage.js index dad3c89f..f4baa76e 100644 --- a/frontend/src/SettingsPage.js +++ b/frontend/src/SettingsPage.js @@ -3,6 +3,7 @@ import React, {useState, useEffect} from 'react'; import Paper from '@material-ui/core/Paper'; import Button from '@material-ui/core/Button'; import Divider from '@material-ui/core/Divider'; +import {Link} from 'react-router-dom'; import TextField from '@material-ui/core/TextField'; @@ -51,6 +52,7 @@ const Settings = (props) => { const boxStyle = { flex: "1", + color: "white", marginLeft: "10px", marginRight: "10px", paddingLeft: "30px", @@ -181,6 +183,7 @@ const Settings = (props) => {

APIKEY

+ What is the API key used for? { const alert = useAlert() + var upload = "" + const [file, setFile] = React.useState(""); + const [workflows, setWorkflows] = React.useState([]); const [selectedWorkflow, setSelectedWorkflow] = React.useState({}); const [selectedExecution, setSelectedExecution] = React.useState({}); @@ -314,7 +319,7 @@ const Workflows = (props) => { // dropdown with copy etc I guess const WorkflowPaper = (props) => { - const { data } = props; + const { data } = props; const [open, setOpen] = React.useState(false); const [anchorEl, setAnchorEl] = React.useState(null); @@ -695,10 +700,8 @@ const Workflows = (props) => { ) } - const setNewWorkflow = () => { - if (newWorkflowName.length === 0) { - return - } + // Can create and set workflows + const setNewWorkflow = (name, description, editingWorkflow, redirect) => { var method = "POST" var extraData = "" @@ -710,12 +713,12 @@ const Workflows = (props) => { workflowdata = editingWorkflow } - workflowdata["name"] = newWorkflowName - workflowdata["description"] = newWorkflowDescription + workflowdata["name"] = name + workflowdata["description"] = description //console.log(workflowdata) //return - fetch(globalUrl+"/api/v1/workflows"+extraData, { + return fetch(globalUrl+"/api/v1/workflows"+extraData, { method: method, headers: { 'Content-Type': 'application/json', @@ -732,17 +735,71 @@ const Workflows = (props) => { return response.json() }) .then((responseJson) => { - if (method === "POST") { + if (method === "POST" && redirect) { window.location.pathname = "/workflows/"+responseJson["id"] + } else if (!redirect) { + // Update :) + getAvailableWorkflows() } else { alert.info("Successfully changed basic info for workflow") } + + return responseJson }) .catch(error => { alert.error(error.toString()) }); } + + const importFiles = (event) => { + const file = event.target.value + if (event.target.files.length > 0) { + for (var key in event.target.files) { + const file = event.target.files[key] + if (file.type !== "application/json") { + //alert.error("File has to contain json.") + continue + } + + const reader = new FileReader() + // Waits for the read + reader.addEventListener('load', (event) => { + var data = reader.result + try { + data = JSON.parse(reader.result) + } catch (e) { + alert.error("Invalid JSON: "+e) + return + } + + // Initialize the workflow itself + const ret = setNewWorkflow(data.name, data.description, {}, false) + .then((response) => { + if (response !== undefined) { + // SET THE FULL THING + data.id = response.id + + // Actually create it + const ret = setNewWorkflow(data.name, data.description, data, false) + .then((response) => { + if (response !== undefined) { + alert.success("Successfully created "+data.name) + } + }) + } + }) + .catch(error => { + alert.error("Import error: "+error.toString()) + }); + }) + + // Actually reads + reader.readAsText(file) + } + } + } + const modalView = modalOpen ? { Cancel -
-
- -
+
+ + + + + + + upload = ref} onChange={importFiles} />
@@ -850,7 +910,7 @@ const Workflows = (props) => {

Executions: {selectedWorkflow.name}

-