Merge pull request #1266 from Shuffle/main

Syncing 1.4.0 with main

.github/CONTRIBUTING.md

@@ -19,10 +19,10 @@ Workflows are where the magic of Shuffle automation happens. Our current ones [a
 Documentation is essential to any product, and Shuffle is no exception. Documentation in Shuffle uses markdown and is located in the [shuffle-docs](https://github.com/frikky/shuffle-docs/tree/master/docs) repository. These are then loaded into Shuffle when someone visits [https://shuffler/docs/about](https://shuffler/docs/about), then cached for later use. If you make an edit, expect it on our website in about an hour. 
 
 #### Frontend (ReactJS)
-The frontend of Shuffle is what everyone sees when they log in. Our goal here is to make it easy to get started and keep going with Shuffle - removing any blockers from the point of accessibility. If you'd like to get started, find [an issue](https://github.com/frikky/Shuffle/issues) and check the [installation guide](https://github.com/frikky/Shuffle/blob/master/install-guide.md#local-development-installation) for setting it up locally without Docker. 
+The frontend of Shuffle is what everyone sees when they log in. Our goal here is to make it easy to get started and keep going with Shuffle - removing any blockers from the point of accessibility. If you'd like to get started, find [an issue](https://github.com/frikky/Shuffle/issues) and check the [installation guide](https://github.com/Shuffle/Shuffle/blob/main/.github/install-guide.md) for setting it up locally without Docker. 
 
 #### Backend (Golang)
-The backend of Shuffle is our REST API Server that runs in the background, handling all the API-calls in general, whether from users or apps. If you'd like to get started, find [an issue](https://github.com/frikky/Shuffle/issues) and check the [installation guide](https://github.com/frikky/Shuffle/blob/master/install-guide.md#local-development-installation) for setting it up locally without Docker. 
+The backend of Shuffle is our REST API Server that runs in the background, handling all the API-calls in general, whether from users or apps. If you'd like to get started, find [an issue](https://github.com/frikky/Shuffle/issues) and check the [installation guide](https://github.com/Shuffle/Shuffle/blob/main/.github/install-guide.md) for setting it up locally without Docker. 
 
 #### Scaling (Golang & Python)
 Shuffle runs using Docker, and is built to scale. There are many areas that may revolve around scaling, but the main issues come down to how we use Docker in our [architecture](https://shuffler.io/docs/architecture). If you want to help by submitting Helm charts (K8s), Docker swarm configurations, blogposts, or talk about code changes that would help scaling - please reach out (or just start building!), and we can discuss the possibilities. Make sure to read about the architecture first :)

.github/install-guide.md

@@ -1,7 +1,7 @@
 # Installation guide
 Installation of Shuffle is currently only available in docker. Looking for how to update Shuffle? Check the [updating guide](https://shuffler.io/docs/configuration#updating_shuffle)
 
-This document outlines a an introduction environment which is not scalable. [Read here](https://shuffler.io/docs/configuration#production_readiness) for information on production readiness. This also includes system requirements and configurations for Swarm or Kubernetes. 
+This document outlines an introduction environment which is not scalable. [Read here](https://shuffler.io/docs/configuration#production_readiness) for information on production readiness. This also includes system requirements and configurations for Swarm or Kubernetes. 
 
 # Docker - *nix
 The Docker setup is done with docker-compose 
@@ -17,9 +17,10 @@ cd Shuffle
 
 3. Fix prerequisites for the Opensearch database (Elasticsearch): 
 ```bash
-mkdir shuffle-database
+mkdir shuffle-database                    # Create a database folder
-sudo chown -R 1000:1000 shuffle-database
+sudo chown -R 1000:1000 shuffle-database  # IF you get an error using 'chown', add the user first with 'sudo useradd opensearch'
-# IF you get an error using 'chown', add the user first with 'sudo useradd opensearch'
+
+sudo swapoff -a                           # Disable swap
 ```
 
 4. Run docker-compose.
@@ -78,14 +79,14 @@ Local development is pretty straight forward with **ReactJS** and **Golang**. Th
 
 **PS: You have to stop the Backend Docker container to get this one working**
 
-**PPS: Use the "Launch" branch when developing to get it set up easier**
+**PPS: Use the "main" branch when developing to get it set up easier**
 
 ## Frontend - ReactJS /w cytoscape
 http://localhost:3000 - Requires [npm](https://nodejs.org/en/download/)/[yarn](https://yarnpkg.com/lang/en/docs/install/#debian-stable)/your preferred manager. Runs independently from backend.
 ```bash
 cd frontend
-npm i
+yarn install
-npm start
+yarn start
 ```
 
 ## Backend - Golang
@@ -106,13 +107,18 @@ Large portions of the backend is written in another repository - [shuffle-shared
 2. Open the Shuffle backend's go.mod file (./shuffle/backend/go.mod)  (**NOT** in shuffle-shared)
 3. Change the following line to point to your directory AFTER the =>
 ```
-//replace github.com/frikky/shuffle-shared => ../../../../git/shuffle-shared
+//replace github.com/frikky/shuffle-shared => ../../shuffle-shared
 ```
 4. Make the changes you want, then restart the backend server!
 5. With your changes made, make a pull request :fire:
 
 ## Database - Opensearch 
-Make sure this is running through the docker-compose, and that the backend points to it with SHUFFLE_OPENSEARCH_URL defined
+Make sure this is running through the docker-compose, and that the backend points to it with SHUFFLE_OPENSEARCH_URL defined.
+
+So essentially, what that means is:
+1. Make sure you have docker-compose installed
+2. Make sure you have the docker-compose.yml file from this repository
+3. Run `docker-compose up opensearch -d`
 
 ## Orborus
 Execution of Workflows:

.github/workflows/dockerbuild.yaml

@@ -2,7 +2,13 @@ name: dockerbuild
 
 on:
   push:
-    branches: [1.3.0] 
+    branches:
+      - main
+    paths:
+      - "**"
+      - "!.github/**"
+      - "!**.md"
+      - "!docker-compose.yml"
 jobs:
   main:
     runs-on: ubuntu-latest
@@ -13,23 +19,23 @@ jobs:
         include:
           - app: frontend
             path: frontend
-           version: nightly
+            version: 1.3.0
             experimental: true
           - app: backend
             path: backend
-           version: nightly
+            version: 1.3.0
             experimental: true
           - app: app_sdk
             path: backend/app_sdk
-           version: nightly
+            version: 1.3.0
             experimental: true
           - app: orborus
             path: functions/onprem/orborus
-           version: nightly
+            version: 1.3.0
             experimental: true
           - app: worker
             path: functions/onprem/worker
-           version: nightly
+            version: 1.3.0
             experimental: true
     steps:
       - name: Checkout
@@ -41,7 +47,7 @@ jobs:
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
         with:
-          platforms: 'amd64,arm64,arm'
+          platforms: "amd64,arm64,arm"
 
       - name: Login to DockerHub
         uses: docker/login-action@v3
@@ -71,7 +77,9 @@ jobs:
           cache-to: type=local,dest=/tmp/.buildx-cache
           tags: |
             ghcr.io/shuffle/shuffle-${{ matrix.app }}:${{ matrix.version }}
+            ghcr.io/shuffle/shuffle-${{ matrix.app }}:latest
             ${{ secrets.DOCKERHUB_USERNAME }}/shuffle-${{ matrix.app }}:${{ matrix.version }}
+            ${{ secrets.DOCKERHUB_USERNAME }}/shuffle-${{ matrix.app }}:latest
 
       - name: Image digest
         run: echo ${{ steps.docker_build.outputs.digest }}

.gitignore

@@ -21,6 +21,7 @@ backend/onprem/app_sdk/apps
 
 *.exe
 *debug*
+!**/RuntimeDebugger.jsx
 
 shuffle-database/batch_metrics_enabled.conf
 shuffle-database/logging_enabled.conf

README.md

@@ -45,6 +45,10 @@ Please consider [sponsoring](https://github.com/sponsors/frikky) the project if
 ## Documentation
 [Documentation](https://shuffler.io/docs) can be found on [https://shuffler.io/docs](https://shuffler.io/docs) and is written here: [https://github.com/frikky/shuffle-docs](https://github.com/frikky/shuffle-docs).
 
+### Setting up a local development environment
+
+Please follow the steps mentioned [here](https://github.com/Shuffle/Shuffle/blob/main/.github/install-guide.md#local-development-installation)!
+
 ## Related repositories
 * OpenAPI apps: [https://github.com/frikky/security-openapis](https://github.com/frikky/security-openapis)
 * Documentation: [https://github.com/frikky/shuffle-docs](https://github.com/frikky/shuffle-docs)
@@ -75,7 +79,7 @@ These are the main areas to contribute in:
 Contributing guidelines are outlined [here](https://github.com/frikky/Shuffle/blob/master/.github/CONTRIBUTING.md).
 
 ## Contributors 
-![ICPL logo](https://github.com/frikky/Shuffle/blob/launch/frontend/src/assets/img/icpl_logo.png)
+![ICPL logo](https://github.com/Shuffle/Shuffle/blob/main/frontend/src/assets/img/icpl_logo.png)
 
 **Shuffle**
 <a href="https://github.com/frikky/shuffle/graphs/contributors">
@@ -113,4 +117,4 @@ Below is the folder structure with a short explanation
 └ docker-compose.yml 	# Used for deployments
 ```
 
-**It's in BETA** - [Get in touch](https://shuffler.io/contact), send a mail to [frikky@shuffler.io](mailto:frikky@shuffler.io) or poke me on twitter [@frikkylikeme](https://twitter.com/frikkylikeme)
+[Get in touch](https://shuffler.io/contact), send a mail to [frikky@shuffler.io](mailto:frikky@shuffler.io) or poke me on twitter [@frikkylikeme](https://twitter.com/frikkylikeme)

backend/README.md

@@ -12,7 +12,7 @@ This folder has all parts necessary for the backend to run locally and in Docker
 Shuffle's backend is written in Go, with apps being python (for now). More about local development can be seen in the main README.
 
 Running the backend:
-```
+```bash
 cd go-app
-go run *.go
+go run main.go docker.go walkoff.go
 ```

backend/app_sdk/app_base.py

@@ -3519,6 +3519,12 @@ class AppBase:
                                     if self.action["app_name"].lower() == "shuffle tools":
                                         timeout = 55
 
+                                    timeout_env = os.getenv("SHUFFLE_APP_SDK_TIMEOUT", timeout)
+                                    try:
+                                        timeout = int(timeout_env)
+                                    except Exception as e:
+                                        self.logger.info(f"[WARNING] Failed parsing timeout to int: {e}")
+
                                     #timeout = 30 
 
                                     try:

backend/app_sdk/requirements.txt

@@ -1,4 +1,4 @@
-urllib3==1.26.5
+urllib3==1.26.18
 requests==2.31.0
 MarkupSafe==2.0.1
 liquidpy==0.8.1

backend/go-app/go.mod

@@ -6,7 +6,6 @@ go 1.19
 
 require (
 	cloud.google.com/go/datastore v1.11.0
-	cloud.google.com/go/pubsub v1.31.0
 	cloud.google.com/go/storage v1.30.1
 	github.com/basgys/goxml2json v1.1.0
 	github.com/carlescere/scheduler v0.0.0-20170109141437-ee74d2f83d82
@@ -22,7 +21,6 @@ require (
 	github.com/shuffle/shuffle-shared v0.4.79
 	golang.org/x/crypto v0.9.0
 	google.golang.org/api v0.125.0
-	google.golang.org/appengine v1.6.7
 	google.golang.org/grpc v1.55.0
 	gopkg.in/src-d/go-git.v4 v4.13.1
 	gopkg.in/yaml.v3 v3.0.1
@@ -98,15 +96,15 @@ require (
 	go.opencensus.io v0.24.0 // indirect
 	go4.org v0.0.0-20201209231011-d4a079459e60 // indirect
 	golang.org/x/mod v0.8.0 // indirect
-	golang.org/x/net v0.10.0 // indirect
+	golang.org/x/net v0.17.0 // indirect
 	golang.org/x/oauth2 v0.8.0 // indirect
-	golang.org/x/sync v0.2.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
-	golang.org/x/sys v0.8.0 // indirect
+	golang.org/x/term v0.13.0 // indirect
-	golang.org/x/term v0.8.0 // indirect
+	golang.org/x/text v0.13.0 // indirect
-	golang.org/x/text v0.9.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	golang.org/x/tools v0.6.0 // indirect
 	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
+	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect

backend/go-app/go.sum

@@ -39,8 +39,6 @@ cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2k
 cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
 cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
 cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
-cloud.google.com/go/pubsub v1.31.0 h1:aXdyyJz90kA+bor9+6+xHAciMD5mj8v15WqFZ5E0sek=
-cloud.google.com/go/pubsub v1.31.0/go.mod h1:dYmJ3K97NCQ/e4OwZ20rD4Ym3Bu8Gu9m/aJdWQjdcks=
 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
 cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
 cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
@@ -50,6 +48,7 @@ cloud.google.com/go/storage v1.12.0/go.mod h1:fFLk2dp2oAhDz8QFKwqrjdJvxSp/W2g7ni
 cloud.google.com/go/storage v1.30.1 h1:uOdMxAs8HExqBlnLtnQyP0YkvbiDpdGShGKtx6U/oNM=
 cloud.google.com/go/storage v1.30.1/go.mod h1:NfxhC0UJE1aXSx7CIIbCf7y9HKT7BiccwkR7+P7gN8E=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
+github.com/AdaLogics/go-fuzz-headers v0.0.0-20210715213245-6c3934b029d8 h1:V8krnnfGj4pV65YLUm3C0/8bl7V5Nry2Pwvy3ru/wLc=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
 github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
@@ -66,6 +65,7 @@ github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF0
 github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
 github.com/Microsoft/go-winio v0.6.0 h1:slsWYD/zyx7lCXoZVlvQrj0hPTM1HI4+v1sIda2yDvg=
 github.com/Microsoft/go-winio v0.6.0/go.mod h1:cTAf44im0RAYeL23bpB+fzCyDH2MJiz2BO69KH/soAE=
+github.com/Microsoft/hcsshim v0.9.6 h1:VwnDOgLeoi2du6dAznfmspNqTiwczvjv4K7NxuY9jsY=
 github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/ProtonMail/go-crypto v0.0.0-20230518184743-7afd39499903 h1:ZK3C5DtzV2nVAQTx5S5jQvMeDqWtD1By5mOoyY/xJek=
 github.com/ProtonMail/go-crypto v0.0.0-20230518184743-7afd39499903/go.mod h1:8TI4H3IbrackdNgv+92dI+rhpCaLqM0IfpgCgenFvRE=
@@ -75,11 +75,14 @@ github.com/acomagu/bufpipe v1.0.4 h1:e3H4WUzM3npvo5uv95QuJM3cQspFNtFBzvJ2oNjKIDQ
 github.com/acomagu/bufpipe v1.0.4/go.mod h1:mxdxdup/WdsKVreO5GpW4+M/1CE2sMG4jeGJ2sYmHc4=
 github.com/adrg/strutil v0.2.3 h1:WZVn3ItPBovFmP4wMHHVXUr8luRaHrbyIuLlHt32GZQ=
 github.com/adrg/strutil v0.2.3/go.mod h1:+SNxbiH6t+O+5SZqIj5n/9i5yUjR+S3XXVrjEcN2mxg=
+github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs=
 github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs=
 github.com/algolia/algoliasearch-client-go/v3 v3.18.1 h1:FP2Xtqqs/sefR5Qluygp+jVV+juXzEdJaPrZTCDLhDQ=
 github.com/algolia/algoliasearch-client-go/v3 v3.18.1/go.mod h1:i7tLoP7TYDmHX3Q7vkIOL4syVse/k5VJ+k0i8WqFiJk=
 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
+github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/aws/aws-sdk-go v1.42.27/go.mod h1:OGr6lGMAKGlG9CVrYnWYDKIyb829c6EVBRjxqjmPepc=
@@ -98,6 +101,7 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.19.0/go.mod h1:BgQOMsg8av8jset59jely
 github.com/aws/smithy-go v1.13.5/go.mod h1:Tg+OJXh4MB2R/uN61Ko2f6hTZwB/ZYGOtib8J3gBHzA=
 github.com/basgys/goxml2json v1.1.0 h1:4ln5i4rseYfXNd86lGEB+Vi652IsIXIvggKM/BhUKVw=
 github.com/basgys/goxml2json v1.1.0/go.mod h1:wH7a5Np/Q4QoECFIU8zTQlZwZkrilY0itPfecMw41Dw=
+github.com/bitly/go-simplejson v0.5.0 h1:6IH+V8/tVMab511d5bn4M7EwGXZf9Hj6i2xSwkNEM+Y=
 github.com/bitly/go-simplejson v0.5.0/go.mod h1:cXHtHw4XUPsvGaxgjIAn8PhEWG9NfngEKAMDJEczWVA=
 github.com/bradfitz/gomemcache v0.0.0-20221031212613-62deef7fc822 h1:hjXJeBcAMS1WGENGqDpzvmgS43oECTx8UXq31UBu0Jw=
 github.com/bradfitz/gomemcache v0.0.0-20221031212613-62deef7fc822/go.mod h1:H0wQNHz2YrLsuXOZozoeDmnHXkNCRmMW0gwFWDfEZDA=
@@ -131,6 +135,7 @@ github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSV
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
+github.com/creack/pty v1.1.11 h1:07n33Z8lZxZ2qwegKbObQohDhXDQxiMMz1NOUGYlesw=
 github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/cyphar/filepath-securejoin v0.2.3/go.mod h1:aPGpWjXOXUn2NCNjFvBE6aRxGGx79pTxQpKOJNYHHl4=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -147,6 +152,7 @@ github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4
 github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
+github.com/elazarl/goproxy v0.0.0-20221015165544-a0805db90819 h1:RIB4cRk+lBqKK3Oy0r2gRX4ui7tuhiZq2SuTtTCi0/0=
 github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
 github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
@@ -163,7 +169,6 @@ github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
 github.com/form3tech-oss/jwt-go v3.2.3+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
 github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
-github.com/frikky/go-elasticsearch/v8 v8.13.1/go.mod h1:RPq0JXPQVVSFHTlPwj/go8BZ1hegRf+StaSpT2iGIoQ=
 github.com/frikky/kin-openapi v0.41.0/go.mod h1:ev9OZAw7Bv5p0w93j91++6a1ElPzGcCofst+kmrWsj4=
 github.com/frikky/kin-openapi v0.42.0 h1:d5Z6vnuQ6RnCCPIxZaDL+TH2ODLxT8abytOt+Zh+Kd0=
 github.com/frikky/kin-openapi v0.42.0/go.mod h1:ev9OZAw7Bv5p0w93j91++6a1ElPzGcCofst+kmrWsj4=
@@ -174,10 +179,12 @@ github.com/fsouza/go-dockerclient v1.9.7/go.mod h1:vx9C32kE2D15yDSOMCDaAEIARZpDQ
 github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
+github.com/gliderlabs/ssh v0.3.5 h1:OcaySEmAQJgyYcArR+gGGTHCyE7nvhEMTlYY+Dp8CpY=
 github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI=
 github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic=
 github.com/go-git/go-billy/v5 v5.4.1 h1:Uwp5tDRkPr+l/TnbHOQzp+tmJfLceOlbVucgpTz8ix4=
 github.com/go-git/go-billy/v5 v5.4.1/go.mod h1:vjbugF6Fz7JIflbVpl1hJsGjSHNltrSw45YK/ukIvQg=
+github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20230305113008-0c11038e723f h1:Pz0DHeFij3XFhoBRGUDPzSJ+w2UcK5/0JvF8DRI58r8=
 github.com/go-git/go-git/v5 v5.7.0 h1:t9AudWVLmqzlo+4bqdf7GY+46SUuRsx59SboFxkq2aE=
 github.com/go-git/go-git/v5 v5.7.0/go.mod h1:coJHKEOk5kUClpsNlXrUvPrDxY3w3gjHvhcZd8Fodw8=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
@@ -254,9 +261,11 @@ github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/
 github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
 github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
 github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
+github.com/google/martian/v3 v3.3.2 h1:IqNFLAmvJOgVlpdEBiQbDc2EwKW77amAycfTuWKdfvw=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
@@ -318,14 +327,17 @@ github.com/klauspost/compress v1.11.13 h1:eSvu8Tmq6j2psUJqJrLcWH6K3w5Dwc+qipbaA6
 github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
+github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e h1:hB2xlXdHp/pmPZq0y3QnmWAArdw9PqbmotexnWx/FU8=
 github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
+github.com/matryer/is v1.2.0 h1:92UTHpy8CDwaJ08GqLDzhhuixiBUUD1p3AU6PHddz4A=
 github.com/matryer/is v1.2.0/go.mod h1:2fLPjFQM9rhQ15aVEtbuwhJinnOqrmgXPNdZsdwlWXA=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
@@ -379,6 +391,7 @@ github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFz
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
@@ -391,10 +404,8 @@ github.com/seccomp/libseccomp-golang v0.9.2-0.20220502022130-f33da4d89646/go.mod
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
 github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
 github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
-github.com/shuffle/shuffle-shared v0.4.56 h1:Z29GZB/eipt2aA4i+6heO1V2mrUmHPdw4QN4X4b1HdU=
+github.com/shuffle/shuffle-shared v0.4.66 h1:Aw4qOp0VsVJrRzW1sJhEy4OY4fRGlFErUD5+93RXL6g=
-github.com/shuffle/shuffle-shared v0.4.56/go.mod h1:X613gbo0dT3fnYvXDRwjQZyLC+T49T2nSQOrCV5QMlI=
+github.com/shuffle/shuffle-shared v0.4.66/go.mod h1:X613gbo0dT3fnYvXDRwjQZyLC+T49T2nSQOrCV5QMlI=
-github.com/shuffle/shuffle-shared v0.4.59 h1:5Sv8aorgQJFZr3cCKltfycdXzp9v5zlF2l3GZXjrTEo=
-github.com/shuffle/shuffle-shared v0.4.59/go.mod h1:X613gbo0dT3fnYvXDRwjQZyLC+T49T2nSQOrCV5QMlI=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
 github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
 github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
@@ -406,6 +417,7 @@ github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e/go.mod h1:XV66xRDq
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/src-d/gcfg v1.4.0 h1:xXbNR5AlLSA315x2UO+fTSSAXCDf+Ar38/6oyGbDKQ4=
 github.com/src-d/gcfg v1.4.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
@@ -422,6 +434,7 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
+github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
 github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
 github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
 github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
@@ -460,8 +473,8 @@ golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0
 golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw=
 golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
-golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g=
+golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc=
-golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0=
+golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -541,8 +554,8 @@ golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
-golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
+golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
-golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
+golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -566,7 +579,6 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI=
-golang.org/x/sync v0.2.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -621,8 +633,8 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
+golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
-golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -630,8 +642,8 @@ golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuX
 golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
 golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
-golang.org/x/term v0.8.0 h1:n5xxQn2i3PC0yLAbjTpNT85q/Kgzcr2gIoX9OrJUols=
+golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek=
-golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
+golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -644,8 +656,8 @@ golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
 golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
-golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
+golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
-golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -829,12 +841,15 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/src-d/go-billy.v4 v4.3.2 h1:0SQA1pRztfTFx2miS8sA97XvooFeNOmvUenF4o0EcVg=
 gopkg.in/src-d/go-billy.v4 v4.3.2/go.mod h1:nDjArDMp+XMs1aFAESLRjfGSgfvoYN0hDfzEk0GjC98=
+gopkg.in/src-d/go-git-fixtures.v3 v3.5.0 h1:ivZFOIltbce2Mo8IjzUHAFoq/IylO9WHhNOAJK+LsJg=
 gopkg.in/src-d/go-git-fixtures.v3 v3.5.0/go.mod h1:dLBcvytrw/TYZsNTWCnkNF2DSIlzWYqTe3rJR56Ac7g=
 gopkg.in/src-d/go-git.v4 v4.13.1 h1:SRtFyV8Kxc0UP7aCHcijOMQGPxHSmMOPrzulQWolkYE=
 gopkg.in/src-d/go-git.v4 v4.13.1/go.mod h1:nx5NYcxdKxq5fpltdHnPa2Exj4Sx0EclMWZQbYDu2z8=
@@ -855,6 +870,7 @@ gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
+gotest.tools/v3 v3.0.3 h1:4AuOwCGf4lLR9u3YOe2awrHygurzhO/HeQ6laiA6Sx0=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3'
 services:
   frontend:
-    image: ghcr.io/shuffle/shuffle-frontend:nightly
+    image: ghcr.io/shuffle/shuffle-frontend:latest
     container_name: shuffle-frontend
     hostname: shuffle-frontend
     ports:
@@ -15,7 +15,7 @@ services:
     depends_on:
       - backend
   backend:
-    image: ghcr.io/shuffle/shuffle-backend:nightly
+    image: ghcr.io/shuffle/shuffle-backend:latest
     container_name: shuffle-backend
     hostname: ${BACKEND_HOSTNAME}
     # Here for debugging:
@@ -34,7 +34,7 @@ services:
       - SHUFFLE_FILE_LOCATION=/shuffle-files
     restart: unless-stopped
   orborus:
-    image: ghcr.io/shuffle/shuffle-orborus:nightly
+    image: ghcr.io/shuffle/shuffle-orborus:latest
     container_name: shuffle-orborus
     hostname: shuffle-orborus
     networks:

frontend/confd/templates/nginx.conf

@@ -70,7 +70,7 @@ http {
 			try_files $uri /index.html;
 		}
 
-		location /api/v1 {
+		location ~ /api/v(1|2) {
 			proxy_pass http://{{ getenv "BACKEND_HOSTNAME" "shuffle-backend" }}:5001;
 			proxy_buffering off;
 			proxy_http_version 1.1;
@@ -81,16 +81,6 @@ http {
 			send_timeout 900;
 		}
 
-		location /api/v2 {
-			proxy_pass http://{{ getenv "BACKEND_HOSTNAME" "shuffle-backend" }}:5001;
-			proxy_buffering off;
-			proxy_http_version 1.1;
-			
-			proxy_connect_timeout 900;
-			proxy_send_timeout 900;
-			proxy_read_timeout 900;
-			send_timeout 900;
-		}
 	}
 
 	server {
@@ -122,7 +112,7 @@ http {
 		}
 
 		# Get the hostname from environment here?
-		location /api/v1 {
+		location ~ /api/v(1|2) {
 			proxy_pass http://{{ getenv "BACKEND_HOSTNAME" "shuffle-backend" }}:5001;
 			proxy_buffering off;
 			proxy_http_version 1.1;

frontend/src/components/AppSelection.jsx

@@ -443,7 +443,7 @@ const AppSelection = props => {
                     }}
                     color="rgba(158, 158, 158, 1)"
                 >
-                    Select the apps you work with and we will connect the for you.
+                    Select the apps you work with and we will connect them for you.
                 </Typography>
                 <Grid rowSpacing={1} columnSpacing={2} container >
                     {appButtons.map((appData, index) => {

frontend/src/components/RuntimeDebugger.jsx

@@ -0,0 +1,491 @@
+import React, { useState, useEffect } from 'react';
+
+import { 
+	TextField,
+	Link,
+	Button,
+	CircularProgress,
+	Select,
+	MenuList,
+	MenuItem,
+	FormControl,
+	InputLabel,
+	Autocomplete,
+	Tooltip,
+	Typography,
+} from '@mui/material';
+
+import { toast } from "react-toastify" 
+import { makeStyles } from "@mui/styles";
+import theme from '../theme.jsx';
+import dayjs from 'dayjs';
+import { AdapterDayjs } from '@mui/x-date-pickers/AdapterDayjs'
+import Pagination from '@mui/material/Pagination';
+import { 
+	DatePicker, 
+	DateTimePicker,
+	LocalizationProvider,
+} from '@mui/x-date-pickers'
+
+import {
+	OpenInNew as OpenInNewIcon,
+} from '@mui/icons-material';
+
+import { DataGrid, GridColDef, GridValueGetterParams } from '@mui/x-data-grid'
+
+const useStyles = makeStyles({
+  notchedOutline: {
+    borderColor: "#f85a3e !important",
+  },
+});
+
+const RuntimeDebugger = (props) => {
+	const { userdata, globalUrl, } = props
+
+  	const classes = useStyles();
+
+	//const [workflowId, setWorkflowId] = useState("");
+	//const [status, setStatus] = useState("FINISHED");
+	//const [endTime, setEndTime] = useState(dayjs().subtract(0, 'day'))
+	//const [startTime, setStartTime] = useState(dayjs().subtract(30, 'day'))
+
+	const [workflowId, setWorkflowId] = useState("")
+	const [status, setStatus] = useState("")
+	const [endTime, setEndTime] = useState("")
+	const [startTime, setStartTime] = useState("")
+
+	const [workflow, setWorkflow] = useState({})
+	const [searchLoading, setSearchLoading] = useState(false)
+	const [rowCursor, setCursor] = useState("")
+	const [rowsPerPage, setRowsPerPage] = useState(10)
+	const [resultRows, setResultRows] = useState([])
+	const [workflows, setWorkflows] = useState([
+		{"id": "", "name": "All Workflows",}
+	])
+	/*
+	[
+		{id: "1", execution_id: "1", status: "FINISHED", startTimestamp: "2021-10-01 12:00:00", endTimestamp: "2021-10-01 12:00:00", workflow: {"id": "1234", "name": "what",}},
+		{id: "2", execution_id: "2", status: "WAITING", startTimestamp: "2021-10-01 12:00:00", endTimestamp: "2021-10-01 12:00:00", workflow: {"id": "1234", "name": "what",}},
+		{id: "3", execution_id: "3", status: "EXECUTING",startTimestamp: "2021-10-01 12:00:00", endTimestamp: "2021-10-01 12:00:00", workflow: {"id": "1234", "name": "what",}},
+		{id: "4", execution_id: "4", status: "ABORTED", startTimestamp: "2021-10-01 12:00:00", endTimestamp: "2021-10-01 12:00:00", workflow: {"id": "1234", "name": "what",}},
+	]);
+	*/
+
+	const submitSearch = (workflowId, status, startTime, endTime, cursor, limit) => {
+
+		setResultRows([])
+		setSearchLoading(true)
+		const fetchData = {
+			workflow_id: workflowId,
+			cursor: cursor,
+			limit: limit,
+
+			status: status.toUpperCase(),
+			start_time: startTime,
+			end_time: endTime,
+		}
+
+		fetch(`${globalUrl}/api/v1/workflows/search`, {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+			},
+			credentials: "include",
+			body: JSON.stringify(fetchData),
+		})
+		.then((response) => response.json())
+		.then((data) => {
+			setSearchLoading(false)
+			if (data.success) {
+				if (data.runs !== undefined && data.runs !== null && data.runs.length > 0) {
+					for (let key in data.runs) {
+						data.runs[key].id = data.runs[key].execution_id
+
+						// make started_at field into ISO string
+						//data.runs[key].endTimestamp = data.runs[key].ended_at.toISOString().slice(0, 19).replace('T', ' ')
+						const startTimestamp = new Date(data.runs[key].started_at*1000)
+						data.runs[key].startTimestamp = startTimestamp.toISOString().slice(0, 19).replace('T', ' ')
+						const endTimestamp = new Date(data.runs[key].completed_at*1000)
+						data.runs[key].endTimestamp = endTimestamp.toISOString().slice(0, 19).replace('T', ' ')
+					}
+
+
+					// Add 20 empty rows to the end of the resultRows array
+					// This is to make sure that the scrollbar is always visible
+					setResultRows(data.runs)
+				}
+			} else {
+				console.error("Search error: ", data.reason)
+			}
+		})
+		.catch((error) => {
+			setSearchLoading(false)
+			console.error("Error:", error);
+		})
+	}
+
+	  const getAvailableWorkflows = () => {
+		fetch(globalUrl + "/api/v1/workflows", {
+		  method: "GET",
+		  headers: {
+			"Content-Type": "application/json",
+			Accept: "application/json",
+		  },
+		  credentials: "include",
+		})
+		.then((response) => {
+		  if (response.status !== 200) {
+			console.log("Status not 200 for workflows :O!");
+			return;
+		  }
+		  return response.json();
+		})
+		.then((responseJson) => {
+		  if (responseJson !== undefined) {
+			  var foundWorkflows = [{"id": "", "name": "All Workflows",}]
+			  foundWorkflows.push(...responseJson)
+			  setWorkflows(foundWorkflows)
+		  }
+		})
+		.catch((error) => {
+		  console.log("Error getting workflows: " + error);
+		})
+	  }
+
+	useEffect(() => {
+	  	getAvailableWorkflows()
+
+		// Find workflow_id in url query
+		const urlParams = new URLSearchParams(window.location.search);
+		const workflowId = urlParams.get('workflow_id');
+		if (workflowId !== undefined && workflowId !== null && workflowId !== "" && workflowId.length === 36) {
+			setWorkflowId(workflowId)
+
+			// find the workflow in workflows and select it
+			for (let key in workflows) {
+				if (workflows[key].id === workflowId) {
+					setWorkflow(workflows[key])
+					break
+				}
+			}
+		}
+
+		const foundStatus = urlParams.get('status');
+		if (foundStatus !== undefined && foundStatus !== null && foundStatus !== "") {
+			setStatus(foundStatus)
+		}
+	}, [])
+
+	const columns: GridColDef[] = [
+		{
+			field: 'status',
+			headerName: 'Status',
+			width: 150,
+			renderCell: (params) => (
+				<span style={{cursor: "pointer", }} onClick={() => {
+					setStatus(params.row.status)
+				}}>
+					{params.row.status}
+				</span>
+			),
+	    },
+	    {
+			field: 'workflow.name',
+			headerName: 'Workflow Name',
+			width: 250,
+			renderCell: (params) => (
+				<span style={{cursor: "pointer", }} onClick={() => {
+					setWorkflowId(params.row.workflow.id)
+
+					for (let key in workflows) {
+						if (workflows[key].id === params.row.workflow.id) {
+							setWorkflow(workflows[key])
+							break
+						}
+					}
+				}}>
+					{params.row.workflow.name}
+				</span>
+			),
+		  },
+	      {
+			field: 'workflow results',
+			headerName: 'Results',
+			width: 75,
+			renderCell: (params) => {
+				//console.log("PARAMS: ", params)
+				var extraItems = 0
+				if (params.row.workflow.triggers !== null) {
+					for (let key in params.row.workflow.triggers) {
+						if (params.row.workflow.triggers[key].app_name === "User Input" || params.row.workflow.triggers[key].app_name === "Shuffle Workflow") { 
+							extraItems += 1
+						}
+					}
+				}
+				
+				const actionLength = params.row.workflow.actions === null ? 0 : params.row.workflow.actions.length+extraItems
+				const resultLength = params.row.results === null ? 0 : params.row.results.length
+
+				//console.log("actionLength: ", actionLength, " resultLength: ", resultLength)
+				return (
+					<span style={{}} onClick={() => {
+					}}>
+						{resultLength} / {actionLength} 
+					</span>
+				)
+			},
+		  },
+		{ field: 'startTimestamp', headerName: 'Start time', width: 160, },
+		{ field: 'endTimestamp', headerName: 'End time', width: 160, },
+	    {
+			field: 'id',
+			headerName: 'Explore',
+			width: 65,
+			renderCell: (params) => (
+			  <Link href={`/workflows/${params.row.workflow.id}?execution_id=${params.row.id}`} target="_blank" rel="noopener noreferrer">
+				<OpenInNewIcon fontSize="small" />
+			  </Link>
+			),
+		  },
+	]
+
+	useEffect(() => {
+		// Check if the user is currently focusing a texxtfield or not
+		// If they are, don't submit the search
+		if (document.activeElement.tagName === "INPUT") {
+			console.log("User is focusing a textfield, not submitting search")
+			return
+		}
+
+		submitSearch(workflowId, status, startTime, endTime, rowCursor, rowsPerPage)
+	}, [workflowId, status, startTime, endTime])
+
+	const textfieldStyle = {
+		marginTop: 7,
+		marginLeft: 5, 
+	}
+
+	const handleStartTimeChange = (date) => {
+		setStartTime(date)
+	}
+	
+	const handleEndTimeChange = (date) => {
+		setEndTime(date)
+	}
+
+	const handleWorkflowSelectionUpdate = (e, isUserinput) => {
+		if (e.target.value === undefined || e.target.value === null || e.target.value.id === undefined) {
+			console.log("Returning as there's no id")
+			return null
+		}
+
+		console.log("DATA: ", e.target.value)
+
+		setWorkflow(e.target.value)
+		setWorkflowId(e.target.value.id)
+
+		submitSearch(e.target.value.id, status, startTime, endTime, rowCursor, rowsPerPage)
+	}
+
+	return (
+		<div style={{minWidth: 1000, maxWidth: 1000, margin: "auto", }}>
+			<h1>Workflow Run Debugger</h1>
+			<form onSubmit={(e) => {
+				submitSearch(workflowId, status, startTime, endTime, rowCursor, rowsPerPage)
+			}} style={{display: "flex", }}>
+				<FormControl fullWidth style={{marginTop: 5, }}>
+				  <InputLabel id="status-label">Status</InputLabel>
+				  <Select
+					labelId="status-label"
+					id="status-label-select"
+					value={status}
+					label="Status"
+					onChange={(event) => {
+						console.log("Changed: ", event.target)
+						setStatus(event.target.value)
+					}}
+				  >
+					<MenuItem value="">No selection</MenuItem>
+					<MenuItem value="FINISHED">FINISHED</MenuItem>
+					<MenuItem value="EXECUTING">EXECUTING</MenuItem>
+					<MenuItem value="WAITING">WAITING</MenuItem>
+					<MenuItem value="ABORTED">ABORTED</MenuItem>
+				  </Select>
+				</FormControl>
+				{/*
+				<TextField
+					label="Workflow ID"
+					fullWidth
+					value={workflowId}
+					onChange={(e) => {
+						setWorkflowId(e.target.value)
+					}}
+					style={textfieldStyle}
+				/>
+				*/}
+				<Autocomplete
+				  id="workflow_search"
+				  value={workflow}
+				  classes={{ inputRoot: classes.inputRoot }}
+				  ListboxProps={{
+					style: {
+					  backgroundColor: theme.palette.inputColor,
+					  color: "white",
+					},
+				  }}
+				  getOptionLabel={(option) => {
+					if (
+					  option === undefined ||
+					  option === null ||
+					  option.name === undefined ||
+					  option.name === null
+					) {
+					  return "No Workflow Selected";
+					}
+
+					const newname = (
+					  option.name.charAt(0).toUpperCase() + option.name.substring(1)
+					).replaceAll("_", " ");
+					return newname;
+				  }}
+				  options={workflows}
+				  fullWidth
+				  style={{
+					backgroundColor: theme.palette.inputColor,
+					height: 50,
+					borderRadius: theme.palette.borderRadius,
+					marginTop: 5, 
+					marginLeft: 5,
+				  }}
+				  onChange={(event, newValue) => {
+					console.log("Found value: ", newValue)
+
+					var parsedinput = { target: { value: newValue } }
+
+					// For variables
+					if (typeof newValue === 'string' && newValue.startsWith("$")) {
+						parsedinput = { 
+							target: { 
+								value: {
+									"name": newValue, 
+									"id": newValue,
+									"actions": [],
+									"triggers": [],
+								} 
+							} 
+						}
+					}
+
+					handleWorkflowSelectionUpdate(parsedinput)
+				  }}
+				  renderOption={(props, data, state) => {
+					if (data.id === workflow.id) {
+					  data = workflow;
+					}
+
+					return (
+					  <Tooltip arrow placement="left" title={
+						<span style={{}}>
+						  {data.image !== undefined && data.image !== null && data.image.length > 0 ?
+							<img src={data.image} alt={data.name} style={{ backgroundColor: theme.palette.surfaceColor, maxHeight: 200, minHeigth: 200, borderRadius: theme.palette.borderRadius, }} />
+							: null}
+						  <Typography>
+							Choose {data.name}
+						  </Typography>
+						</span>
+					  } placement="bottom">
+						<MenuItem
+						  style={{
+							backgroundColor: theme.palette.inputColor,
+							color: data.id === workflow.id ? "red" : "white",
+						  }}
+						  value={data}
+						  onClick={(e) => {
+							var parsedinput = { target: { value: data } }
+							handleWorkflowSelectionUpdate(parsedinput)
+						  }}
+						>
+						  {data.name}
+						</MenuItem>
+					  </Tooltip>
+					)
+				  }}
+				  renderInput={(params) => {
+					return (
+					  <TextField
+						style={{
+						  backgroundColor: theme.palette.inputColor,
+						  borderRadius: theme.palette.borderRadius,
+						}}
+						{...params}
+						label="Workflow"
+						variant="outlined"
+					  />
+					);
+				  }}
+				/>
+				<LocalizationProvider dateAdapter={AdapterDayjs}>
+		        	<DateTimePicker
+					  sx={{
+						marginTop: 1, 
+						marginLeft: 1,
+						minWidth: 240,
+					    maxWidth: 240,
+					  }}
+					  label="Search from"
+					  format="YYYY-MM-DD HH:mm:ss"
+					  value={startTime}
+					  onChange={handleStartTimeChange}
+					  renderInput={(params) => <TextField {...params} />}
+					/>
+					<DateTimePicker
+					  sx={{
+						marginTop: 1, 
+						marginLeft: 1,
+						minWidth: 240,
+					    maxWidth: 240,
+					  }}
+					  label="Search until"
+					  format="YYYY-MM-DD HH:mm:ss"
+					  value={endTime}
+					  onChange={handleEndTimeChange}
+					  renderInput={(params) => <TextField {...params} />}
+					/>
+
+				</LocalizationProvider>
+				<Button
+					variant="contained"
+					color="primary"
+					onClick={() => {
+						submitSearch(workflowId, status, startTime, endTime, rowCursor, rowsPerPage) 
+					}}
+					disabled={searchLoading}
+					style={{height: 50, minWidth: 100, marginTop: 15, marginLeft: 10, }}
+				>
+					{searchLoading ? <CircularProgress size={30} /> : "Search"}
+				</Button>
+			</form>
+			<div style={{height: 700, padding: "10px 0px 10px 0px", }}>
+				<DataGrid
+					rows={resultRows}
+					columns={columns}
+					pageSize={rowsPerPage}
+					rowsPerPageOptions={[10, 20, 50, 100]}
+					checkboxSelection
+					disableSelectionOnClick
+					onPageSizeChange={(newPageSize) => {
+						setRowsPerPage(newPageSize)
+						submitSearch(workflowId, status, startTime, endTime, rowCursor, newPageSize) 
+					}}
+					// event for when clicking next page
+					// Hide page changer
+					onPageChange={(params) => {
+						console.log("params: ", params)
+					}}
+				  />
+			</div> 
+		</div>
+	)
+}
+
+export default RuntimeDebugger;

frontend/src/views/Docs.jsx

@@ -6,6 +6,8 @@ import { useParams, useNavigate, Link } from "react-router-dom";
 import { isMobile } from "react-device-detect";
 import theme from '../theme.jsx';
 import remarkGfm from 'remark-gfm'
+import KeyboardArrowRightIcon from '@mui/icons-material/KeyboardArrowRight';
+import ExpandMoreIcon from '@mui/icons-material/ExpandMore';
 
 import {
   Grid,
@@ -19,6 +21,9 @@ import {
   Typography,
   Paper,
   List,
+  Collapse,
+  ListItemButton,
+  ListItemText
 } from "@mui/material";
 
 import {
@@ -62,7 +67,7 @@ const Docs = (defaultprops) => {
   // Quickfix for react router 5 -> 6 
   const params = useParams();
   //var props = JSON.parse(JSON.stringify(defaultprops))
-	var props = Object.assign({selected: false}, defaultprops);
+  var props = Object.assign({ selected: false }, defaultprops);
   props.match = {}
   props.match.params = params
 
@@ -78,6 +83,7 @@ const Docs = (defaultprops) => {
   const [data, setData] = useState("");
   const [firstrequest, setFirstrequest] = useState(true);
   const [list, setList] = useState([]);
+  const [isopen, setOpen] = useState(-1);
   const [, setListLoaded] = useState(false);
   const [anchorEl, setAnchorEl] = React.useState(null);
   const [headingSet, setHeadingSet] = React.useState(false);
@@ -98,11 +104,16 @@ const Docs = (defaultprops) => {
     setAnchorEl(null);
   }
 
+  const handleCollapse = (index) => {
+    setOpen(isopen === index ? -1 : index)
+  };
+
   const SidebarPaperStyle = {
     backgroundColor: theme.palette.surfaceColor,
     overflowX: "hidden",
     position: "relative",
-    padding: 30,
+    paddingLeft: 15,
+    paddingRight: 15,
     paddingTop: 15,
     marginTop: 15,
     minHeight: "80vh",
@@ -110,8 +121,8 @@ const Docs = (defaultprops) => {
   };
 
   const SideBar = {
-    minWidth: 250,
+    minWidth: 300,
-    maxWidth: 300,
+    width: "20%",
     left: 0,
     position: "sticky",
     top: 50,
@@ -142,7 +153,7 @@ const Docs = (defaultprops) => {
         }
         setListLoaded(true);
       })
-      .catch((error) => {});
+      .catch((error) => { });
   };
 
   const fetchDocs = (docId) => {
@@ -226,7 +237,7 @@ const Docs = (defaultprops) => {
           setData("# Error\nThis page doesn't exist.");
         }
       })
-      .catch((error) => {});
+      .catch((error) => { });
   };
 
   if (firstrequest) {
@@ -445,7 +456,7 @@ const Docs = (defaultprops) => {
                   href={selectedMeta.link}
                   style={{ textDecoration: "none", color: "#f85a3e" }}
                 >
-                  <Button style={{color: "white", }} variant="outlined" color="secondary">
+                  <Button style={{ color: "white", }} variant="outlined" color="secondary">
                     <EditIcon /> &nbsp;&nbsp;Edit
                   </Button>
                 </a>
@@ -546,7 +557,7 @@ const Docs = (defaultprops) => {
 
 
   const CustomButton = (props) => {
-		const {title, icon, link} = props
+    const { title, icon, link } = props
 
     const [hover, setHover] = useState(false)
 
@@ -557,7 +568,7 @@ const Docs = (defaultprops) => {
         target="_blank"
         style={{ textDecoration: "none", color: "inherit", flex: 1, margin: 10, }}
       >
-				<div style={{cursor: hover ? "pointer" : "default", borderRadius: theme.palette.borderRadius, flex: 1, border: "1px solid rgba(255,255,255,0.3)", backgroundColor: hover ? theme.palette.surfaceColor : theme.palette.inputColor, padding: 25, }} 
+        <div style={{ cursor: hover ? "pointer" : "default", borderRadius: theme.palette.borderRadius, flex: 1, border: "1px solid rgba(255,255,255,0.3)", backgroundColor: hover ? theme.palette.surfaceColor : theme.palette.inputColor, padding: 25, }}
           onClick={(event) => {
             if (link === "" || link === undefined) {
               event.preventDefault()
@@ -588,7 +599,7 @@ const Docs = (defaultprops) => {
 
 
   const DocumentationButton = (props) => {
-		const {item, link} = props
+    const { item, link } = props
 
     const [hover, setHover] = useState(false);
 
@@ -599,7 +610,7 @@ const Docs = (defaultprops) => {
 
     return (
       <Link to={link} style={hrefStyle}>
-				<div style={{width: "100%", height: 80, cursor: hover ? "pointer" : "default", borderRadius: theme.palette.borderRadius, border: "1px solid rgba(255,255,255,0.3)", backgroundColor: hover ? theme.palette.surfaceColor : theme.palette.inputColor, }}
+        <div style={{ width: "100%", height: 80, cursor: hover ? "pointer" : "default", borderRadius: theme.palette.borderRadius, border: "1px solid rgba(255,255,255,0.3)", backgroundColor: hover ? theme.palette.surfaceColor : theme.palette.inputColor, }}
           onMouseOver={() => {
             setHover(true)
           }}
@@ -631,15 +642,15 @@ const Docs = (defaultprops) => {
       margin: "auto",
       marginTop: 50,
     }}>
-				<Typography variant="h4" style={{textAlign: "center",}}>
+      <Typography variant="h4" style={{ textAlign: "center", }}>
         Documentation
       </Typography>
-				<div style={{display: "flex", marginTop: 25, }}>
+      <div style={{ display: "flex", marginTop: 25, }}>
-					<CustomButton title="Talk to Support" icon=<img src="/images/Shuffle_logo_new.png" style={{height: 35, width: 35, border: "", borderRadius: theme.palette.borderRadius, }} /> />
+        <CustomButton title="Talk to Support" icon=<img src="/images/Shuffle_logo_new.png" style={{ height: 35, width: 35, border: "", borderRadius: theme.palette.borderRadius, }} /> />
-					<CustomButton title="Ask the community" icon=<img src="/images/social/discord.png" style={{height: 35, width: 35, border: "", borderRadius: theme.palette.borderRadius, }} /> link="https://discord.gg/B2CBzUm" />
+        <CustomButton title="Ask the community" icon=<img src="/images/social/discord.png" style={{ height: 35, width: 35, border: "", borderRadius: theme.palette.borderRadius, }} /> link="https://discord.gg/B2CBzUm" />
       </div>
 
-				<div style={{textAlign: "left"}}>
+      <div style={{ textAlign: "left" }}>
         <Typography variant="h6" style={headerStyle} >Tutorial</Typography>
         <Typography variant="body1">
           <b>Dive in.</b> Hands-on is the best approach to see how Shuffle can transform your security operations. Our set of tutorials and videos teach you how to build your skills. Check out the <Link to="/docs/getting-started" style={hrefStyle2}>getting started</Link> section to give it a go!
@@ -724,6 +735,7 @@ const Docs = (defaultprops) => {
 				*/}
     </div>
 
+  // PostDataBrowser Section
   const postDataBrowser =
     list === undefined || list === null ? null : (
       <div style={Body}>
@@ -743,50 +755,51 @@ const Docs = (defaultprops) => {
 
                 const itemMatching = props.match.params.key === undefined ? false :
                   props.match.params.key.toLowerCase() === item.toLowerCase();
-                //const [tocLines, setTocLines] = React.useState([]);
                 return (
-                  <li key={index} style={{ marginTop: 10 }}>
+                  <li key={index}>
-                    <Link
+                    <ListItemButton
+                      component={Link}
                       key={index}
                       style={hrefStyle}
                       to={path}
                       onClick={() => {
                         setTocLines([]);
                         fetchDocs(item);
+                        handleCollapse(index);
                       }}
                     >
-                      <Typography
+                      <ListItemText
                         style={{ color: itemMatching ? "#f86a3e" : "inherit" }}
                         variant="body1"
                       >
-                        <b>> {newname}</b>
+                        {newname}
-                      </Typography>
+                      </ListItemText>
-                    </Link>
+                      {isopen === index ? <ExpandMoreIcon /> : <KeyboardArrowRightIcon />}
+                    </ListItemButton>
                     {itemMatching &&
                       tocLines !== null &&
                       tocLines !== undefined &&
                       tocLines.length > 0 ? (
-                      <div style={{ marginLeft: 5 }}>
+                      <Collapse in={isopen === index} timeout="auto" unmountOnExit>
                         {tocLines.map((data, index) => {
-                          //console.log(data)
 
                           return (
-                            <Link
+                            <ListItemButton
+                              component={Link}
                               key={index}
                               style={innerHrefStyle}
                               to={data.link}
-                              onClick={() => {}}
                             >
-                              <Typography
+                              <ListItemText
                                 variant="body2"
                                 style={{ cursor: "pointer" }}
                               >
-                                - {data.text}
+                                {data.text}
-                              </Typography>
+                              </ListItemText>
-                            </Link>
+                            </ListItemButton>
                           );
                         })}
-                      </div>
+                      </Collapse>
                     ) : null}
                   </li>
                 );
@@ -794,7 +807,7 @@ const Docs = (defaultprops) => {
             </List>
           </Paper>
         </div>
-				<div style={{maxWidth: 750, minWidth: 750, margin: "auto", overflow: "hidden", marginTop: 50, }}>
+        <div style={{ width: "70%", margin: "auto", overflow: "hidden", marginTop: 50, paddingRight: 50 }}>
           {props.match.params.key === undefined ?
             mainpageInfo
             :
@@ -936,7 +949,7 @@ const Docs = (defaultprops) => {
 
   // Padding and zIndex etc set because of footer in cloud.
   const loadedCheck = (
-    <div style={{minHeight: 1000, paddingBottom: 100, zIndex: 50000, }}>
+    <div style={{ minHeight: 1000, paddingBottom: 100, zIndex: 50000, }}>
       <BrowserView>{postDataBrowser}</BrowserView>
       <MobileView>{postDataMobile}</MobileView>
     </div>

functions/kubernetes/setup_registry.sh

@@ -23,16 +23,14 @@ mkdir -p certs
 # Generate CA key
 openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/reg.key -x509 -days 365 -out certs/reg.crt -subj "/CN=$node_ip"
 
-# generate a random string
+echo "Generated certs/reg.key and certs/reg.crt!"
-random_string=$(openssl rand -hex 3)
 
-echo "Starting docker registry with name shuffle-local-registry-$random_string.."
+echo -e "Please run:\n" 
-
+echo "docker run -d -p 5000:5000 --restart=always --name "shuffle-local-registry" \
-docker run -d -p 5000:5000 --restart=always --name "shuffle-local-registry-$random_string" \
   -v $(pwd)/certs:/certs \
   -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/reg.crt \
   -e REGISTRY_HTTP_TLS_KEY=/certs/reg.key \
-  registry:2
+  registry:2"
 
-echo "Done!"
+echo -e  "\nnow to start the reigstry :)!"
 

functions/onprem/orborus/go.sum

@@ -656,7 +656,6 @@ gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
 gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

functions/onprem/orborus/orborus.go

@@ -1,7 +1,7 @@
 package main
 
 /*
-	Orborus exists to listen for new workflow executions whcih are deployed as workers.
+	Orborus exists to listen for new workflow executions which are deployed as workers.
 */
 
 //  Potential issues:
@@ -475,8 +475,8 @@ func deployServiceWorkers(image string) {
 						fmt.Sprintf("SHUFFLE_SWARM_NETWORK_NAME=%s", networkName),
 						fmt.Sprintf("SHUFFLE_APP_REPLICAS=%d", appReplicaCnt),
 						fmt.Sprintf("SHUFFLE_LOGS_DISABLED=%s", os.Getenv("SHUFFLE_LOGS_DISABLED")),
-						fmt.Sprintf("SHUFFLE_DEBUG_MEMORY=%s", os.Getenv("SHUFFLE_DEBUG_MEMORY")),
+						fmt.Sprintf("DEBUG_MEMORY=%s", os.Getenv("DEBUG_MEMORY")),
-						fmt.Sprintf("TZ=%s", timezone),
+						fmt.Sprintf("SHUFFLE_APP_SDK_TIMEOUT=%s", os.Getenv("SHUFFLE_APP_SDK_TIMEOUT")),
 					},
 					//Hosts: []string{
 					//	innerContainerName,

functions/onprem/worker/worker.go

@@ -230,6 +230,8 @@ func deployApp(cli *dockerclient.Client, image string, identifier string, env []
 	// log.Printf("image: %s", image)
 	// log.Printf("identifier: %s", identifier)
 	// log.Printf("execution: %+v", workflowExecution)
+	log.Printf("[DEBUG] Adding SHUFFLE_APP_SDK_TIMEOUT=%s", os.Getenv("SHUFFLE_APP_SDK_TIMEOUT"))
+	env = append(env, fmt.Sprintf("SHUFFLE_APP_SDK_TIMEOUT=%s", os.Getenv("SHUFFLE_APP_SDK_TIMEOUT")))
 
 	if os.Getenv("IS_KUBERNETES") == "true" {