Merge branch '1.3.0' of github.com:Shuffle/Shuffle into 1.2.0

This commit is contained in:
Aditya
2023-07-13 00:21:43 +05:30
256 changed files with 2314 additions and 2470 deletions
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
View File
+38
View File
@@ -0,0 +1,38 @@
## Install Shuffle on AWS
First, you need to create your own VPC for the same range of private IP addresses.
**To create a VPC in AWS, follow these steps**
1. Sign in to the AWS Management Console and open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
2. In the top navigation bar, choose the region in which you want to create the VPC.
3. In the navigation pane, choose Your VPCs & Choose Create VPC.
4. Enter a name for your VPC in the Name tag field & Choose VPC and more option.
5. Specify the IPv4 CIDR block for your VPC. The CIDR block is the range of IP addresses that will be available for use within your VPC. You can specify any CIDR block that is:
- Between a /16 and /28 netmask (inclusive)
- Not currently in use
6. Specify the AZs, Number of public subnets & Number of private subnets.
![image](https://user-images.githubusercontent.com/118437260/211500830-30c52dc0-0688-47f9-8ee7-eb7f9b31a9b8.png)
7. Choose Yes, Create VPC.
Your VPC will be created and will appear in the list of Your VPCs. By default, a VPC includes a default security group and a default network ACL. You can customize your VPC by adding subnets, security groups, network ACLs, and other resources.
**To create an EC2 instance in AWS, follow these steps:**
1. Sign in to the AWS Management Console and open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the top navigation bar, choose the region in which you want to create the instance.
3. In the navigation pane, choose Instances and Choose Launch Instance.
4. On the Choose an Amazon Machine Image (AMI) page, choose an AMI. An AMI is a template that contains the software configuration (operating system, application server, and applications) for your instance.
5. On the Choose an Instance Type page, choose the hardware configuration of your instance.
6. On the Select an existing key pair or create a new key pair dialog box, choose an existing key pair or create a new one.
7. On the network settings page and click on edit and select your VPC & subnet.
8. On the Configure Security Group page, configure the security group for your instance. A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.
![image](https://user-images.githubusercontent.com/118437260/211514598-1c95e459-b98a-4579-b3a7-a92bf36e9f50.png)
9. On the Add Storage page, add storage to your instance.
10. Review your instance launch details and choose Launch.
Regular → Executable
View File
Regular → Executable
+9 -3
View File
@@ -1,7 +1,7 @@
# Installation guide
Installation of Shuffle is currently only available in docker. Looking for how to update Shuffle? Check the [updating guide](https://shuffler.io/docs/configuration#updating_shuffle)
This document outlines a an introduction environment which is not scalable. [Read here](https://shuffler.io/docs/configuration#production_readiness) for information on production readiness. This also includes system requirements and configurations for Swarm or K8s.
This document outlines a an introduction environment which is not scalable. [Read here](https://shuffler.io/docs/configuration#production_readiness) for information on production readiness. This also includes system requirements and configurations for Swarm or Kubernetes.
# Docker - *nix
The Docker setup is done with docker-compose
@@ -19,6 +19,7 @@ cd Shuffle
```bash
mkdir shuffle-database
sudo chown -R 1000:1000 shuffle-database
# IF you get an error using 'chown', add the user first with 'sudo useradd opensearch'
```
4. Run docker-compose.
@@ -26,7 +27,12 @@ sudo chown -R 1000:1000 shuffle-database
docker-compose up -d
```
When you're done, skip to the "After installation" step below.
5. Recommended for Opensearch to work well
```bash
sudo sysctl -w vm.max_map_count=262144 # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
```
When you're done, skip to the [After installation](#after-installation) step below.
## Windows with WSL
This step is for setting up with Docker on windows from scratch.
@@ -57,7 +63,7 @@ https://shuffler.io/docs/configuration
3. Sign in with the same Username & Password! Go to /apps and see if you have any apps yet. If not - you may need to [configure proxies](https://shuffler.io/docs/configuration#production_readiness)
4. Check out https://shuffler.io/docs/configuration as it has a lot of useful information to get started
![Admin account setup](https://github.com/frikky/Shuffle/blob/master/frontend/src/assets/img/shuffle_adminaccount.png)
![Admin account setup](https://github.com/Shuffle/Shuffle/blob/main/frontend/src/assets/img/shuffle_adminaccount.png?raw=true)
### Useful info
* Check out [getting started](https://shuffler.io/docs/getting_started)
View File
-66
View File
@@ -1,66 +0,0 @@
name: docker-build
on:
push:
branches: launch
jobs:
main:
runs-on: ubuntu-latest
continue-on-error: ${{ matrix.experimental }}
strategy:
fail-fast: false
matrix:
include:
- app: frontend
path: frontend
version: 1.0.0
experimental: true
- app: backend
path: backend
version: 1.0.0
experimental: false
- app: orborus
path: functions/onprem/orborus
version: 1.0.0
experimental: false
- app: database
path: backend/database
version: 1.0.0
experimental: false
steps:
-
name: Checkout
uses: actions/checkout@v2
-
name: Set up QEMU
uses: docker/setup-qemu-action@v1
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Use below configuration for ghcr.io
# with:
# registry: ghcr.io
# username: ${{ github.repository_owner }}
# password: ${{ secrets.CR_PAT }}
-
name: Build and push
id: docker_build
uses: docker/build-push-action@v2
env:
BUILDX_NO_DEFAULT_LOAD: true
with:
context: ${{ matrix.path }}/
file: ${{ matrix.path }}/Dockerfile
platforms: linux/amd64,linux/arm64
#,linux/386 - no node image I guess?
push: true
tags: ${{ secrets.DOCKERHUB_USERNAME }}/${{ matrix.app }}:${{ matrix.version }}
-
name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
+7 -5
View File
@@ -2,7 +2,7 @@ name: dockerbuild
on:
push:
branches: 1.2.0
branches: [main, 1.3.0]
jobs:
main:
runs-on: ubuntu-latest
@@ -35,12 +35,14 @@ jobs:
- name: Checkout
uses: actions/checkout@v2
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
with:
platforms: 'amd64,arm64,arm'
- name: Login to DockerHub
uses: docker/login-action@v2
with:
+16
View File
@@ -0,0 +1,16 @@
name: Automation - Add all new issues to roadmap project
on:
issues:
types:
- opened
jobs:
add-to-project:
name: Add issue to project
runs-on: ubuntu-latest
steps:
- uses: actions/add-to-project@v0.5.0
with:
project-url: https://github.com/orgs/Shuffle/projects/8
github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}
View File
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
+2 -2
View File
@@ -21,7 +21,7 @@ Follow us on Twitter at [@shuffleio](https://twitter.com/shuffleio).
</h4>
![Example Shuffle webhook integration](https://github.com/frikky/Shuffle/blob/master/frontend/src/assets/img/github_shuffle_img.png)
![Example Shuffle webhook integration](https://github.com/frikky/Shuffle/blob/main/frontend/src/assets/img/github_shuffle_img.png)
## Try it
* Self-hosted: Check out the [installation guide](https://github.com/frikky/shuffle/blob/master/.github/install-guide.md)
@@ -96,7 +96,7 @@ Shuffle backend: AGPLv3
Apps, specification and App SDK: MIT
## Architecture
![Shuffle Architecture](https://github.com/frikky/Shuffle/blob/master/frontend/src/assets/img/shuffle_architecture.png)
![Shuffle Architecture](https://github.com/frikky/Shuffle/blob/main/frontend/src/assets/img/shuffle_architecture.png)
### Repository overview
Below is the folder structure with a short explanation
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
View File
View File
View File
View File
Regular → Executable
View File
View File
View File
View File
View File
View File
View File
View File
View File
Regular → Executable
View File
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
Regular → Executable
+1 -1
View File
@@ -534,7 +534,7 @@ func getDockerImage(resp http.ResponseWriter, request *http.Request) {
if len(img.ID) == 0 {
if len(img2.ID) == 0 {
workflowapps, err := shuffle.GetAllWorkflowApps(ctx, 0, 0)
//log.Printf("[INFO] Getting workflowapps for a rebuild. Got %d with err %#v", len(workflowapps), err)
log.Printf("[INFO] Getting workflowapps for a rebuild. Got %d with err %#v", len(workflowapps), err)
if err == nil {
imageName := ""
imageVersion := ""
Regular → Executable
+1 -1
View File
@@ -49,7 +49,6 @@ require (
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/frikky/go-elasticsearch/v8 v8.13.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
github.com/go-openapi/swag v0.19.5 // indirect
@@ -102,4 +101,5 @@ require (
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
)
Regular → Executable
+1
View File
@@ -5997,6 +5997,7 @@ func initHandlers() {
// New for recommendations in Shuffle
r.HandleFunc("/api/v1/recommendations/get_actions", shuffle.HandleActionRecommendation).Methods("POST", "OPTIONS")
r.HandleFunc("/api/v1/recommendations/modify", shuffle.HandleRecommendationAction).Methods("POST", "OPTIONS")
r.HandleFunc("/api/v1/workflows/{key}/revisions", shuffle.GetWorkflowRevisions).Methods("GET", "OPTIONS")
// Triggers
r.HandleFunc("/api/v1/hooks/new", shuffle.HandleNewHook).Methods("POST", "OPTIONS")
Regular → Executable
View File
Regular → Executable
+676 -2
View File
@@ -636,6 +636,7 @@ func handleWorkflowQueue(resp http.ResponseWriter, request *http.Request) {
}
//log.Printf("Actionresult unmarshal: %s", string(body))
log.Printf("[DEBUG] Got workflow result from %s of length %d", request.RemoteAddr, len(body))
ctx := context.Background()
err = shuffle.ValidateNewWorkerExecution(ctx, body)
if err == nil {
@@ -774,7 +775,6 @@ func runWorkflowExecutionTransaction(ctx context.Context, attempts int64, workfl
}
//log.Printf("BASE LENGTH: %d", len(workflowExecution.Results))
workflowExecution, dbSave, err := shuffle.ParsedExecutionResult(ctx, *workflowExecution, actionResult, false, 0)
if err != nil {
b, suberr := json.Marshal(actionResult)
@@ -1118,6 +1118,680 @@ func handleExecution(id string, workflow shuffle.Workflow, request *http.Request
err = imageCheckBuilder(execInfo.ImageNames)
if err != nil {
log.Printf("[ERROR] Failed building the required images from %#v: %s", execInfo.ImageNames, err)
return shuffle.WorkflowExecution{}, "Failed unmarshal during execution", err
}
makeNew := true
start, startok := request.URL.Query()["start"]
if request.Method == "POST" {
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Printf("[ERROR] Failed request POST read: %s", err)
return shuffle.WorkflowExecution{}, "Failed getting body", err
}
// This one doesn't really matter.
log.Printf("[INFO] Running POST execution with body of length %d for workflow %s", len(string(body)), workflowExecution.Workflow.ID)
if len(body) >= 4 {
if body[0] == 34 && body[len(body)-1] == 34 {
body = body[1 : len(body)-1]
}
if body[0] == 34 && body[len(body)-1] == 34 {
body = body[1 : len(body)-1]
}
}
sourceAuth, sourceAuthOk := request.URL.Query()["source_auth"]
if sourceAuthOk {
//log.Printf("\n\n\nSETTING SOURCE WORKFLOW AUTH TO %s!!!\n\n\n", sourceAuth[0])
workflowExecution.ExecutionSourceAuth = sourceAuth[0]
} else {
//log.Printf("Did NOT get source workflow")
}
sourceNode, sourceNodeOk := request.URL.Query()["source_node"]
if sourceNodeOk {
//log.Printf("\n\n\nSETTING SOURCE WORKFLOW NODE TO %s!!!\n\n\n", sourceNode[0])
workflowExecution.ExecutionSourceNode = sourceNode[0]
} else {
//log.Printf("Did NOT get source workflow")
}
//workflowExecution.ExecutionSource = "default"
sourceWorkflow, sourceWorkflowOk := request.URL.Query()["source_workflow"]
if sourceWorkflowOk {
//log.Printf("Got source workflow %s", sourceWorkflow)
workflowExecution.ExecutionSource = sourceWorkflow[0]
} else {
//log.Printf("Did NOT get source workflow")
}
sourceExecution, sourceExecutionOk := request.URL.Query()["source_execution"]
if sourceExecutionOk {
//log.Printf("[INFO] Got source execution%s", sourceExecution)
workflowExecution.ExecutionParent = sourceExecution[0]
} else {
//log.Printf("Did NOT get source execution")
}
if len(string(body)) < 50 {
//log.Println(body)
// String in string
//log.Println(body)
//if string(body)[0] == "\"" && string(body)[string(body)
log.Printf("[DEBUG] Body: %s", string(body))
}
var execution shuffle.ExecutionRequest
err = json.Unmarshal(body, &execution)
if err != nil {
log.Printf("[WARNING] Failed execution POST unmarshaling - continuing anyway: %s", err)
//return shuffle.WorkflowExecution{}, "", err
}
if execution.Start == "" && len(body) > 0 {
execution.ExecutionArgument = string(body)
}
// FIXME - this should have "execution_argument" from executeWorkflow frontend
//log.Printf("EXEC: %#v", execution)
if len(execution.ExecutionArgument) > 0 {
workflowExecution.ExecutionArgument = execution.ExecutionArgument
}
if len(execution.ExecutionSource) > 0 {
workflowExecution.ExecutionSource = execution.ExecutionSource
}
//log.Printf("Execution data: %#v", execution)
if len(execution.Start) == 36 && len(workflow.Actions) > 0 {
log.Printf("[INFO] Should start execution on node %s", execution.Start)
workflowExecution.Start = execution.Start
found := false
for _, action := range workflow.Actions {
if action.ID == execution.Start {
found = true
break
}
}
if !found {
log.Printf("[ERROR] Action %s was NOT found! Exiting execution.", execution.Start)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Startnode %s was not found in actions", workflow.Start), errors.New(fmt.Sprintf("Startnode %s was not found in actions", workflow.Start))
}
} else if len(execution.Start) > 0 {
//log.Printf("[INFO] !")
//log.Printf("[ERROR] START ACTION %s IS WRONG ID LENGTH %d!", execution.Start, len(execution.Start))
//return shuffle.WorkflowExecution{}, fmt.Sprintf("Startnode %s was not found in actions", execution.Start), errors.New(fmt.Sprintf("Startnode %s was not found in actions", execution.Start))
}
if len(execution.ExecutionId) == 36 {
workflowExecution.ExecutionId = execution.ExecutionId
} else {
sessionToken := uuid.NewV4()
workflowExecution.ExecutionId = sessionToken.String()
}
} else {
// Check for parameters of start and ExecutionId
// This is mostly used for user input trigger
answer, answerok := request.URL.Query()["answer"]
referenceId, referenceok := request.URL.Query()["reference_execution"]
if answerok && referenceok {
// If answer is false, reference execution with result
log.Printf("[INFO] Answer is OK AND reference is OK!")
if answer[0] == "false" {
log.Printf("Should update reference and return, no need for further execution!")
// Get the reference execution
oldExecution, err := shuffle.GetWorkflowExecution(ctx, referenceId[0])
if err != nil {
log.Printf("Failed getting execution (execution) %s: %s", referenceId[0], err)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Failed getting execution ID %s because it doesn't exist.", referenceId[0]), err
}
if oldExecution.Workflow.ID != id {
log.Println("Wrong workflowid!")
return shuffle.WorkflowExecution{}, fmt.Sprintf("Bad ID %s", referenceId), errors.New("Bad ID")
}
newResults := []shuffle.ActionResult{}
//log.Printf("%#v", oldExecution.Results)
for _, result := range oldExecution.Results {
log.Printf("%s - %s", result.Action.ID, start[0])
if result.Action.ID == start[0] {
note, noteok := request.URL.Query()["note"]
if noteok {
result.Result = fmt.Sprintf("User note: %s", note[0])
} else {
result.Result = fmt.Sprintf("User clicked %s", answer[0])
}
// Stopping the whole thing
result.CompletedAt = int64(time.Now().Unix())
result.Status = "ABORTED"
oldExecution.Status = result.Status
oldExecution.Result = result.Result
oldExecution.LastNode = result.Action.ID
}
newResults = append(newResults, result)
}
oldExecution.Results = newResults
err = shuffle.SetWorkflowExecution(ctx, *oldExecution, true)
if err != nil {
log.Printf("Error saving workflow execution actionresult setting: %s", err)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Failed setting workflowexecution actionresult in execution: %s", err), err
}
return shuffle.WorkflowExecution{}, "", nil
}
}
if referenceok {
log.Printf("Handling an old execution continuation!")
// Will use the old name, but still continue with NEW ID
oldExecution, err := shuffle.GetWorkflowExecution(ctx, referenceId[0])
if err != nil {
log.Printf("Failed getting execution (execution) %s: %s", referenceId[0], err)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Failed getting execution ID %s because it doesn't exist.", referenceId[0]), err
}
workflowExecution = *oldExecution
}
if len(workflowExecution.ExecutionId) == 0 {
sessionToken := uuid.NewV4()
workflowExecution.ExecutionId = sessionToken.String()
} else {
log.Printf("Using the same executionId as before: %s", workflowExecution.ExecutionId)
makeNew = false
}
// Don't override workflow defaults
}
if startok {
//log.Printf("\n\n[INFO] Setting start to %s based on query!\n\n", start[0])
//workflowExecution.Workflow.Start = start[0]
workflowExecution.Start = start[0]
}
// FIXME - regex uuid, and check if already exists?
if len(workflowExecution.ExecutionId) != 36 {
log.Printf("Invalid uuid: %s", workflowExecution.ExecutionId)
return shuffle.WorkflowExecution{}, "Invalid uuid", err
}
// FIXME - find owner of workflow
// FIXME - get the actual workflow itself and build the request
// MAYBE: Don't send the workflow within the pubsub, as this requires more data to be sent
// Check if a worker already exists for company, else run one with:
// locations, project IDs and subscription names
// When app is executed:
// Should update with status execution (somewhere), which will trigger the next node
// IF action.type == internal, we need the internal watcher to be running and executing
// This essentially means the WORKER has to be the responsible party for new actions in the INTERNAL landscape
// Results are ALWAYS posted back to cloud@execution_id?
if makeNew {
workflowExecution.Type = "workflow"
//workflowExecution.Stream = "tmp"
//workflowExecution.WorkflowQueue = "tmp"
//workflowExecution.SubscriptionNameNodestream = "testcompany-nodestream"
//workflowExecution.Locations = []string{"europe-west2"}
workflowExecution.ProjectId = gceProject
workflowExecution.WorkflowId = workflow.ID
workflowExecution.StartedAt = int64(time.Now().Unix())
workflowExecution.CompletedAt = 0
workflowExecution.Authorization = uuid.NewV4().String()
// Status for the entire workflow.
workflowExecution.Status = "EXECUTING"
}
if len(workflowExecution.ExecutionSource) == 0 {
log.Printf("[INFO] No execution source (trigger) specified. Setting to default")
workflowExecution.ExecutionSource = "default"
} else {
log.Printf("[INFO] Execution source is %s for execution ID %s in workflow %s", workflowExecution.ExecutionSource, workflowExecution.ExecutionId, workflowExecution.Workflow.ID)
}
workflowExecution.ExecutionVariables = workflow.ExecutionVariables
if len(workflowExecution.Start) == 0 && len(workflowExecution.Workflow.Start) > 0 {
workflowExecution.Start = workflowExecution.Workflow.Start
}
startnodeFound := false
newStartnode := ""
for _, item := range workflowExecution.Workflow.Actions {
if item.ID == workflowExecution.Start {
startnodeFound = true
}
if item.IsStartNode {
newStartnode = item.ID
}
}
if !startnodeFound {
log.Printf("[INFO] Couldn't find startnode %s. Remapping to %#v", workflowExecution.Start, newStartnode)
if len(newStartnode) > 0 {
workflowExecution.Start = newStartnode
} else {
return shuffle.WorkflowExecution{}, fmt.Sprintf("Startnode couldn't be found"), errors.New("Startnode isn't defined in this workflow..")
}
}
childNodes := shuffle.FindChildNodes(workflowExecution, workflowExecution.Start, []string{}, []string{})
//topic := "workflows"
startFound := false
// FIXME - remove this?
newActions := []shuffle.Action{}
defaultResults := []shuffle.ActionResult{}
allAuths := []shuffle.AppAuthenticationStorage{}
for _, action := range workflowExecution.Workflow.Actions {
//action.LargeImage = ""
if action.ID == workflowExecution.Start {
startFound = true
}
//log.Println(action.Environment)
if action.Environment == "" {
return shuffle.WorkflowExecution{}, fmt.Sprintf("Environment is not defined for %s", action.Name), errors.New("Environment not defined!")
}
// FIXME: Authentication parameters
if len(action.AuthenticationId) > 0 {
if len(allAuths) == 0 {
allAuths, err = shuffle.GetAllWorkflowAppAuth(ctx, workflow.ExecutingOrg.Id)
if err != nil {
log.Printf("Api authentication failed in get all app auth: %s", err)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Api authentication failed in get all app auth: %s", err), err
}
}
curAuth := shuffle.AppAuthenticationStorage{Id: ""}
for _, auth := range allAuths {
if auth.Id == action.AuthenticationId {
curAuth = auth
break
}
}
if len(curAuth.Id) == 0 {
return shuffle.WorkflowExecution{}, fmt.Sprintf("Auth ID %s doesn't exist", action.AuthenticationId), errors.New(fmt.Sprintf("Auth ID %s doesn't exist", action.AuthenticationId))
}
if curAuth.Encrypted {
setField := true
newFields := []shuffle.AuthenticationStore{}
for _, field := range curAuth.Fields {
parsedKey := fmt.Sprintf("%s_%d_%s_%s", curAuth.OrgId, curAuth.Created, curAuth.Label, field.Key)
newValue, err := shuffle.HandleKeyDecryption([]byte(field.Value), parsedKey)
if err != nil {
log.Printf("[WARNING] Failed decryption for %s: %s", field.Key, err)
setField = false
break
}
field.Value = string(newValue)
newFields = append(newFields, field)
}
if setField {
curAuth.Fields = newFields
}
} else {
log.Printf("[INFO] AUTH IS NOT ENCRYPTED - attempting encrypting!")
err = shuffle.SetWorkflowAppAuthDatastore(ctx, curAuth, curAuth.Id)
if err != nil {
log.Printf("[WARNING] Failed running encryption during execution: %s", err)
}
}
newParams := []shuffle.WorkflowAppActionParameter{}
if strings.ToLower(curAuth.Type) == "oauth2" {
log.Printf("[DEBUG] Should replace auth parameters (Oauth2)")
for _, param := range curAuth.Fields {
if param.Key == "expiration" {
continue
}
newParams = append(newParams, shuffle.WorkflowAppActionParameter{
Name: param.Key,
Value: param.Value,
})
}
for _, param := range action.Parameters {
//log.Printf("Param: %#v", param)
if param.Configuration {
continue
}
newParams = append(newParams, param)
}
} else {
// Rebuild params with the right data. This is to prevent issues on the frontend
for _, param := range action.Parameters {
for _, authparam := range curAuth.Fields {
if param.Name == authparam.Key {
param.Value = authparam.Value
//log.Printf("Name: %s - value: %s", param.Name, param.Value)
//log.Printf("Name: %s - value: %s\n", param.Name, param.Value)
break
}
}
newParams = append(newParams, param)
}
}
action.Parameters = newParams
}
action.LargeImage = ""
if len(action.Label) == 0 {
action.Label = action.ID
}
//log.Printf("LABEL: %s", action.Label)
newActions = append(newActions, action)
// If the node is NOT found, it's supposed to be set to SKIPPED,
// as it's not a childnode of the startnode
// This is a configuration item for the workflow itself.
if len(workflowExecution.Results) > 0 {
defaultResults = []shuffle.ActionResult{}
for _, result := range workflowExecution.Results {
if result.Status == "WAITING" {
result.Status = "FINISHED"
result.Result = "Continuing"
}
defaultResults = append(defaultResults, result)
}
} else if len(workflowExecution.Results) == 0 && !workflowExecution.Workflow.Configuration.StartFromTop {
found := false
for _, nodeId := range childNodes {
if nodeId == action.ID {
//log.Printf("Found %s", action.ID)
found = true
}
}
if !found {
if action.ID == workflowExecution.Start {
continue
}
//log.Printf("[WARNING] Set %s to SKIPPED as it's NOT a childnode of the startnode.", action.ID)
curaction := shuffle.Action{
AppName: action.AppName,
AppVersion: action.AppVersion,
Label: action.Label,
Name: action.Name,
ID: action.ID,
}
//action
//curaction.Parameters = []
defaultResults = append(defaultResults, shuffle.ActionResult{
Action: curaction,
ExecutionId: workflowExecution.ExecutionId,
Authorization: workflowExecution.Authorization,
Result: "Skipped because it's not under the startnode",
StartedAt: 0,
CompletedAt: 0,
Status: "SKIPPED",
})
}
}
}
removeTriggers := []string{}
for triggerIndex, trigger := range workflowExecution.Workflow.Triggers {
//log.Printf("[INFO] ID: %s vs %s", trigger.ID, workflowExecution.Start)
if trigger.ID == workflowExecution.Start {
if trigger.AppName == "User Input" {
startFound = true
break
}
}
if trigger.AppName == "User Input" || trigger.AppName == "Shuffle Workflow" {
found := false
for _, node := range childNodes {
if node == trigger.ID {
found = true
break
}
}
if !found {
//log.Printf("SHOULD SET TRIGGER %s TO BE SKIPPED", trigger.ID)
curaction := shuffle.Action{
AppName: "shuffle-subflow",
AppVersion: trigger.AppVersion,
Label: trigger.Label,
Name: trigger.Name,
ID: trigger.ID,
}
defaultResults = append(defaultResults, shuffle.ActionResult{
Action: curaction,
ExecutionId: workflowExecution.ExecutionId,
Authorization: workflowExecution.Authorization,
Result: "Skipped because it's not under the startnode",
StartedAt: 0,
CompletedAt: 0,
Status: "SKIPPED",
})
} else {
// Replaces trigger with the subflow
//if trigger.AppName == "Shuffle Workflow" {
// replaceActions := false
// workflowAction := ""
// for _, param := range trigger.Parameters {
// if param.Name == "argument" && !strings.Contains(param.Value, ".#") {
// replaceActions = true
// }
// if param.Name == "startnode" {
// workflowAction = param.Value
// }
// }
// if replaceActions {
// replacementNodes, newBranches, lastnode := shuffle.GetReplacementNodes(ctx, workflowExecution, trigger, trigger.Label)
// log.Printf("REPLACEMENTS: %d, %d", len(replacementNodes), len(newBranches))
// if len(replacementNodes) > 0 {
// for _, action := range replacementNodes {
// found := false
// for subActionIndex, subaction := range newActions {
// if subaction.ID == action.ID {
// found = true
// //newActions[subActionIndex].Name = action.Name
// newActions[subActionIndex].Label = action.Label
// break
// }
// }
// if !found {
// action.SubAction = true
// newActions = append(newActions, action)
// }
// // Check if it's already set to have a value
// for resultIndex, result := range defaultResults {
// if result.Action.ID == action.ID {
// defaultResults = append(defaultResults[:resultIndex], defaultResults[resultIndex+1:]...)
// break
// }
// }
// }
// for _, branch := range newBranches {
// workflowExecution.Workflow.Branches = append(workflowExecution.Workflow.Branches, branch)
// }
// // Append branches:
// // parent -> new inner node (FIRST one)
// for branchIndex, branch := range workflowExecution.Workflow.Branches {
// if branch.DestinationID == trigger.ID {
// log.Printf("REPLACE DESTINATION WITH %s!!", workflowAction)
// workflowExecution.Workflow.Branches[branchIndex].DestinationID = workflowAction
// }
// if branch.SourceID == trigger.ID {
// log.Printf("REPLACE SOURCE WITH LASTNODE %s!!", lastnode)
// workflowExecution.Workflow.Branches[branchIndex].SourceID = lastnode
// }
// }
// // Remove the trigger
// removeTriggers = append(removeTriggers, workflowExecution.Workflow.Triggers[triggerIndex].ID)
// }
// log.Printf("NEW ACTION LENGTH %d, RESULT: %d, Triggers: %d, BRANCHES: %d", len(newActions), len(defaultResults), len(workflowExecution.Workflow.Triggers), len(workflowExecution.Workflow.Branches))
// }
//}
_ = triggerIndex
}
}
}
//newTriggers := []shuffle.Trigger{}
//for _, trigger := range workflowExecution.Workflow.Triggers {
// found := false
// for _, triggerId := range removeTriggers {
// if trigger.ID == triggerId {
// found = true
// break
// }
// }
// if found {
// log.Printf("[WARNING] Removed trigger %s during execution", trigger.ID)
// continue
// }
// newTriggers = append(newTriggers, trigger)
//}
//workflowExecution.Workflow.Triggers = newTriggers
_ = removeTriggers
if !startFound {
if len(workflowExecution.Start) == 0 && len(workflowExecution.Workflow.Start) > 0 {
workflowExecution.Start = workflow.Start
} else if len(workflowExecution.Workflow.Actions) > 0 {
workflowExecution.Start = workflowExecution.Workflow.Actions[0].ID
} else {
log.Printf("[ERROR] Startnode %s doesn't exist!!", workflowExecution.Start)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Workflow action %s doesn't exist in workflow", workflowExecution.Start), errors.New(fmt.Sprintf(`Workflow start node "%s" doesn't exist. Exiting!`, workflowExecution.Start))
}
}
//log.Printf("EXECUTION START: %s", workflowExecution.Start)
// Verification for execution environments
workflowExecution.Results = defaultResults
workflowExecution.Workflow.Actions = newActions
onpremExecution := true
_ = onpremExecution
environments := []string{}
if len(workflowExecution.ExecutionOrg) == 0 && len(workflow.ExecutingOrg.Id) > 0 {
workflowExecution.ExecutionOrg = workflow.ExecutingOrg.Id
}
var allEnvs []shuffle.Environment
if len(workflowExecution.ExecutionOrg) > 0 {
//log.Printf("[INFO] Executing ORG: %s", workflowExecution.ExecutionOrg)
allEnvironments, err := shuffle.GetEnvironments(ctx, workflowExecution.ExecutionOrg)
if err != nil {
log.Printf("[WARNING] Failed finding environments: %s", err)
return shuffle.WorkflowExecution{}, fmt.Sprintf("Workflow environments not found for this org"), errors.New(fmt.Sprintf("Workflow environments not found for this org"))
}
for _, curenv := range allEnvironments {
if curenv.Archived {
continue
}
allEnvs = append(allEnvs, curenv)
}
} else {
log.Printf("[ERROR] No org identified for execution of %s. Returning", workflowExecution.Workflow.ID)
return shuffle.WorkflowExecution{}, "No org identified for execution", errors.New("No org identified for execution")
}
if len(allEnvs) == 0 {
log.Printf("[ERROR] No active environments found for org: %s", workflowExecution.ExecutionOrg)
return shuffle.WorkflowExecution{}, "No active environments found", errors.New(fmt.Sprintf("No active env found for org %s", workflowExecution.ExecutionOrg))
}
// Check if the actions are children of the startnode?
imageNames := []string{}
cloudExec := false
_ = cloudExec
for _, action := range workflowExecution.Workflow.Actions {
// Verify if the action environment exists and append
found := false
for _, env := range allEnvs {
if env.Name == action.Environment {
found = true
if env.Type == "cloud" {
cloudExec = true
} else if env.Type == "onprem" {
onpremExecution = true
} else {
log.Printf("[ERROR] No handler for environment type %s", env.Type)
return shuffle.WorkflowExecution{}, "No active environments found", errors.New(fmt.Sprintf("No handler for environment type %s", env.Type))
}
break
}
}
if !found {
log.Printf("[ERROR] Couldn't find environment %s. Maybe it's inactive?", action.Environment)
return shuffle.WorkflowExecution{}, "Couldn't find the environment", errors.New(fmt.Sprintf("Couldn't find env %s in org %s", action.Environment, workflowExecution.ExecutionOrg))
}
found = false
for _, env := range environments {
if env == action.Environment {
found = true
break
}
}
// Check if the app exists?
newName := action.AppName
newName = strings.ReplaceAll(newName, " ", "-")
imageNames = append(imageNames, fmt.Sprintf("%s:%s_%s", baseDockerName, newName, action.AppVersion))
if !found {
environments = append(environments, action.Environment)
}
}
err = imageCheckBuilder(imageNames)
if err != nil {
log.Printf("[ERROR] Failed building the required images from %#v: %s", imageNames, err)
return shuffle.WorkflowExecution{}, "Failed building missing Docker images", err
}
@@ -2867,7 +3541,7 @@ func executeSingleAction(resp http.ResponseWriter, request *http.Request) {
}
resp.WriteHeader(200)
resp.Write(returnBytes)
resp.Write([]byte(returnBytes))
}
// Onlyname is used to
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
+4 -4
View File
@@ -1,7 +1,7 @@
version: '3'
services:
frontend:
image: ghcr.io/shuffle/shuffle-frontend:latest
image: ghcr.io/shuffle/shuffle-frontend:nightly
container_name: shuffle-frontend
hostname: shuffle-frontend
ports:
@@ -15,7 +15,7 @@ services:
depends_on:
- backend
backend:
image: ghcr.io/shuffle/shuffle-backend:latest
image: ghcr.io/shuffle/shuffle-backend:nightly
container_name: shuffle-backend
hostname: ${BACKEND_HOSTNAME}
# Here for debugging:
@@ -34,7 +34,7 @@ services:
- SHUFFLE_FILE_LOCATION=/shuffle-files
restart: unless-stopped
orborus:
image: ghcr.io/shuffle/shuffle-orborus:latest
image: ghcr.io/shuffle/shuffle-orborus:nightly
container_name: shuffle-orborus
hostname: shuffle-orborus
networks:
@@ -62,7 +62,7 @@ services:
container_name: shuffle-opensearch
environment:
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms1024m -Xmx1024m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- "OPENSEARCH_JAVA_OPTS=-Xms2048m -Xmx2048m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- cluster.initial_master_nodes=shuffle-opensearch
- cluster.routing.allocation.disk.threshold_enabled=false
- cluster.name=shuffle-cluster
Regular → Executable
+1 -1
View File
@@ -1,5 +1,5 @@
# Build environment
FROM node:14 as builder
FROM node:18 as builder
RUN mkdir /usr/src/app
Regular → Executable
View File
Regular → Executable
+1 -1
View File
@@ -1,3 +1,3 @@
npm run build
yarn run build
rm -rf ../backend/go-app/build
cp -r build/ ../backend/go-app/build
Regular → Executable
View File
Regular → Executable
View File
Regular → Executable
View File
View File
View File
Regular → Executable
+27 -19
View File
@@ -1,31 +1,25 @@
{
"name": "shuffler",
"homepage": "https://shuffler.io",
"version": "1.1.4",
"version": "1.3.0",
"private": true,
"dependencies": {
"@babel/core": "^7.15.8",
"@codemirror/commands": "^6.2.2",
"@emotion/is-prop-valid": "^1.1.1",
"@emotion/react": "^11.7.0",
"@emotion/styled": "^11.6.0",
"@lezer/highlight": "^1.1.3",
"@material-ui/core": "^4.5.2",
"@material-ui/icons": "^4.5.1",
"@material-ui/lab": "^4.0.0-alpha.58",
"@material-ui/styles": "^4.5.2",
"@material-ui/utils": "^4.11.2",
"@metamask/detect-provider": "^1.2.0",
"@mui/icons-material": "^5.2.1",
"@mui/icons-material": "^5.14.0",
"@mui/material": "^5.2.3",
"@mui/styles": "^5.14.0",
"@mui/x-data-grid": "^5.17.11",
"@uiw/codemirror-themes": "^4.19.9",
"@uiw/react-codemirror": "^3.2.1",
"@use-it/interval": "^1.0.0",
"algoliasearch": "^4.13.1",
"babel-eslint": "^10.1.0",
"class-transformer": "^0.4.0",
"create-react-app": "^4.0.3",
"create-react-app": "^5.0.1",
"cytoscape": "^3.15.1",
"cytoscape-clipboard": "^2.2.1",
"cytoscape-cxtmenu": "^3.1.1",
@@ -40,14 +34,23 @@
"github-markdown-css": "^3.0.1",
"import": "0.0.6",
"interweave": "^11.2.0",
"jss": "^10.10.0",
"jss-camel-case": "^6.1.0",
"jss-default-unit": "^8.0.2",
"jss-global": "^3.0.0",
"jss-nested": "^6.0.1",
"jss-props-sort": "^6.0.0",
"jss-vendor-prefixer": "^8.0.1",
"material-icons": "^0.7.7",
"material-icons-react": "^1.0.4",
"material-ui-chip-input": "^2.0.0-beta.2",
"material-ui-nested-menu-item": "^1.0.2",
"md5-file": "^4.0.0",
"mdbreact": "^4.21.1",
"mime": "^3.0.0",
"moment": "^2.29.1",
"react": "^16.14.0",
"mui-nested-menu": "^3.2.1",
"process": "^0.11.10",
"react": "^18.2.0",
"react-alert": "^5.5.0",
"react-alert-template-basic": "^1.0.0",
"react-alice-carousel": "^2.6.4",
@@ -57,7 +60,7 @@
"react-cookie": "^4.0.1",
"react-cytoscapejs": "^1.2.0",
"react-device-detect": "^1.9.10",
"react-dom": "^16.14.0",
"react-dom": "^18.2.0",
"react-draggable": "^3.3.2",
"react-driftjs": "^1.2.2",
"react-dropzone": "^10.1.10",
@@ -66,12 +69,12 @@
"react-instantsearch-dom": "^6.28.0",
"react-json-pretty": "^2.2.0",
"react-json-view": "^1.19.1",
"react-markdown": "^4.2.2",
"react-markdown": "^8.0.7",
"react-markdown-github": "^3.3.1",
"react-powerhooks": "0.0.7",
"react-router": "6.2.1",
"react-router-dom": "6.2.1",
"react-scripts": "^4.0.1",
"react-scripts": "^5.0.1",
"react-shepherd": "^3.3.6",
"reactstrap": "^7.1.0",
"reaviz": "^12.1.0",
@@ -79,14 +82,12 @@
"shellwords": "^0.1.1",
"simplebar": "^4.2.3",
"styled-components": "^4.4.0",
"webpack": "^4.44.2",
"websocket": "^1.0.30",
"yaml": "^1.7.2",
"yamljs": "^0.3.0",
"zone.js": "~0.11.4"
},
"scripts": {
"start": "PORT=3000 react-scripts start",
"start": "HTTPS=false&&PORT=3000 react-scripts --openssl-legacy-provider start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject",
@@ -108,7 +109,14 @@
"not op_mini all"
],
"devDependencies": {
"@babel/core": "^7.15.8",
"@babel/plugin-proposal-private-property-in-object": "^7.21.11",
"babel-eslint": "^10.1.0",
"prettier": "2.4.1",
"promise-window": "^1.2.1"
"promise-window": "^1.2.1",
"react-16": "npm:react@16.13.1",
"react-dom-16": "npm:react-dom@16.13.1",
"react-error-overlay": "6.0.9",
"webpack": "^4.46.0"
}
}
Regular → Executable
View File

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

Before

Width:  |  Height:  |  Size: 217 KiB

After

Width:  |  Height:  |  Size: 217 KiB

Regular → Executable
View File
Regular → Executable
View File
+6 -6
View File
@@ -10,9 +10,9 @@ docker tag ghcr.io/frikky/shuffle-frontend:nightly ghcr.io/shuffle/shuffle-front
echo "Starting server"
# Rerun build locally for it to update :)
#docker run -it \
# -p 3001:80 \
# -p 3002:443 \
# -v $(pwd)/build:/usr/share/nginx/html:ro \
# --rm \
# nginx
docker run -it \
-p 3001:80 \
-p 3002:443 \
-v $(pwd)/build:/usr/share/nginx/html:ro \
--rm \
ghcr.io/frikky/shuffle-frontend:nightly
Regular → Executable
+16 -69
View File
@@ -1,14 +1,12 @@
import React, { useState, useEffect } from "react";
//import { Route, Routes } from "react-router";
import { Route, Routes, BrowserRouter } from "react-router-dom";
import { Link, Route, Routes, BrowserRouter, useNavigate } from "react-router-dom";
import { CookiesProvider } from "react-cookie";
import { removeCookies, useCookies } from "react-cookie";
import Workflows from "./views/Workflows";
import GettingStarted from "./views/GettingStarted";
import EditWebhook from "./views/EditWebhook";
import AngularWorkflow from "./views/AngularWorkflow";
import AngularWorkflow from "./views/AngularWorkflow.jsx";
import Header from "./components/Header.jsx";
import theme from "./theme";
@@ -20,22 +18,20 @@ import Dashboard from "./views/Dashboard.jsx";
import DashboardView from "./views/DashboardViews.jsx";
import AdminSetup from "./views/AdminSetup";
import Admin from "./views/Admin";
import Docs from "./views/Docs";
import Introduction from "./views/Introduction";
import Docs from "./views/Docs.jsx";
//import Introduction from "./views/Introduction";
import SetAuthentication from "./views/SetAuthentication";
import SetAuthenticationSSO from "./views/SetAuthenticationSSO";
import Search from "./views/Search.jsx";
import RunWorkflow from "./views/RunWorkflow.jsx";
import LandingPageNew from "./views/LandingpageNew";
import LoginPage from "./views/LoginPage";
import SettingsPage from "./views/SettingsPage";
import KeepAlive from "./views/KeepAlive.jsx";
import MyView from "./views/MyView";
import { createMuiTheme, MuiThemeProvider } from "@material-ui/core/styles";
import { ThemeProvider } from "@mui/material/styles";
import UpdateAuthentication from "./views/UpdateAuthentication.jsx";
import FrameworkWrapper from "./views/FrameworkWrapper.jsx";
import ScrollToTop from "./components/ScrollToTop";
import AlertTemplate from "./components/AlertTemplate";
@@ -44,7 +40,6 @@ import { isMobile } from "react-device-detect";
import detectEthereumProvider from "@metamask/detect-provider";
import Drift from "react-driftjs";
import DashboardPage from "./views/TempDashboard.jsx";
// Production - backend proxy forwarding in nginx
var globalUrl = window.location.origin;
@@ -284,18 +279,6 @@ const App = (message, props) => {
}
const includedData =
window.location.pathname === "/home" ||
window.location.pathname === "/features" ? (
<div>
<Routes>
<Route
exact
path="/home"
render={(props) => <LandingPageNew isLoaded={isLoaded} {...props} />}
/>
</Routes>
</div>
) : (
<div
style={{
backgroundColor: "#1F2023",
@@ -473,17 +456,18 @@ const App = (message, props) => {
/>
}
/>
<Route exact path="/apps/authentication" element={<UpdateAuthentication serverside={false} userdata={userdata} isLoggedIn={isLoggedIn} setIsLoggedIn={setIsLoggedIn} register={true} isLoaded={isLoaded} globalUrl={globalUrl} setCookie={setCookie} cookies={cookies} {...props} />} />
<Route
exact
path="/apps"
element={
<Apps
isLoaded={isLoaded}
isLoggedIn={isLoggedIn}
globalUrl={globalUrl}
userdata={userdata}
{...props}
/>
<Apps
isLoaded={isLoaded}
isLoggedIn={isLoggedIn}
globalUrl={globalUrl}
userdata={userdata}
{...props}
/>
}
/>
<Route
@@ -585,28 +569,6 @@ const App = (message, props) => {
/>
}
/>
<Route
exact
path="/introduction"
element={
<Introduction
isLoaded={isLoaded}
globalUrl={globalUrl}
{...props}
/>
}
/>
<Route
exact
path="/introduction/:key"
element={
<Introduction
isLoaded={isLoaded}
globalUrl={globalUrl}
{...props}
/>
}
/>
<Route
exact
path="/set_authentication"
@@ -656,17 +618,6 @@ const App = (message, props) => {
{...props}
/>
}
/>
<Route
exact
path="/testdashboard"
element={
<DashboardPage
isLoaded={isLoaded}
globalUrl={globalUrl}
{...props}
/>
}
/>
<Route
exact
@@ -714,13 +665,9 @@ const App = (message, props) => {
/>
</Routes>
</div>
);
// <div style={{backgroundColor: "rgba(21, 32, 43, 1)", color: "#fffff", minHeight: "100vh"}}>
// backgroundColor: "#213243",
// This is a mess hahahah
return (
<MuiThemeProvider theme={theme}>
<ThemeProvider theme={theme}>
<CookiesProvider>
<BrowserRouter>
<Provider template={AlertTemplate} {...options}>
@@ -728,7 +675,7 @@ const App = (message, props) => {
</Provider>
</BrowserRouter>
</CookiesProvider>
</MuiThemeProvider>
</ThemeProvider>
);
};
Regular → Executable
View File
View File
View File
View File
View File
Regular → Executable
View File

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 14 KiB

Regular → Executable
View File

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

View File

Before

Width:  |  Height:  |  Size: 449 KiB

After

Width:  |  Height:  |  Size: 449 KiB

View File

Before

Width:  |  Height:  |  Size: 431 KiB

After

Width:  |  Height:  |  Size: 431 KiB

View File

Before

Width:  |  Height:  |  Size: 6.2 KiB

After

Width:  |  Height:  |  Size: 6.2 KiB

Some files were not shown because too many files have changed in this diff Show More