marker/shuffle-cracked
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

allow to enable debug mode for shuffle components

Browse Source 
Signed-off-by: Pascal Sthamer <pascal+github@sthamer.xyz>
This commit is contained in:
Pascal Sthamer
2026-03-19 09:18:22 +01:00
parent ee0efb2622
commit 368b5f6393
8 changed files with 176 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
functions/kubernetes/charts/shuffle/README.md
+122 -9
View File
@@ -215,7 +215,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
## Parameters
##### Global parameters
###### Global parameters
| Name | Description | Value |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
@@ -225,7 +225,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `global.compatibility.openshift.adaptSecurityContext` | Adapt the securityContext sections of the deployment to make them compatible with Openshift restricted-v2 SCC: remove runAsUser, runAsGroup and fsGroup and let the platform use their allowed default IDs. Possible values: auto (apply if the detected running cluster is Openshift), force (perform the adaptation always), disabled (do not perform adaptation) | `auto` |
| `global.compatibility.omitEmptySeLinuxOptions` | If set to true, removes the seLinuxOptions from the securityContexts when it is set to an empty object | `false` |
##### Common parameters
###### Common parameters
| Name | Description | Value |
| ------------------------ | --------------------------------------------------------------------------------------- | --------------- |
@@ -241,7 +241,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `diagnosticMode.command` | Command to override all containers in the chart release | `["sleep"]` |
| `diagnosticMode.args` | Args to override all containers in the chart release | `["infinity"]` |
##### Shared Shuffle Parameters
###### Shared Shuffle Parameters
| Name | Description | Value |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------- |
@@ -251,7 +251,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `shuffle.appBaseImageName` | The base image used for shuffle apps. The final image for an app is <appRegistr>/<appBaseImageName>/<appName>:<appVersion> | `frikky` |
| `shuffle.timezone` | The timezone used by Shuffle | `Europe/Berlin` |
##### backend Parameters
###### backend Parameters
| Name | Description | Value |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
@@ -356,6 +356,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `backend.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
| `backend.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
| `backend.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy (ignored if allowExternalEgress=true) | `[]` |
| `backend.debug` | Enable debug mode for backend, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments. | `false` |
| `backend.cleanupSchedule` | The interval in seconds at which the cleanup job runs | `300` |
| `backend.openSearch.url` | The URL at which OpenSearch is available | `http://{{ .Release.Name }}-opensearch:9200` |
| `backend.openSearch.username` | The username that is used for authenticating with OpenSearch | `admin` |
@@ -366,7 +367,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `backend.apps.downloadBranch` | The branch from which apps should be downloaded on startup. | `master` |
| `backend.apps.forceUpdate` | Force an update of apps on startup. | `false` |
##### frontend Parameters
###### frontend Parameters
| Name | Description | Value |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
@@ -472,7 +473,7 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `frontend.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
| `frontend.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy (ignored if allowExternalEgress=true) | `[]` |
##### orborus Parameters
###### orborus Parameters
| Name | Description | Value |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
@@ -575,10 +576,11 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `orborus.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
| `orborus.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
| `orborus.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy (ignored if allowExternalEgress=true) | `[]` |
| `orborus.debug` | Enable debug mode for orborus, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments. | `false` |
| `orborus.executionConcurrency` | The maximum amount of concurrent workflow executions per worker | `25` |
| `orborus.manageWorkerDeployments` | Whether workers are deployed and managed by orborus. When disabled, every worker is expected to be already deployed (see worker.enableHelmDeployment). | `true` |
##### worker Parameters
###### worker Parameters
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
@@ -684,9 +686,10 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `worker.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
| `worker.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
| `worker.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy (ignored if allowExternalEgress=true) | `[]` |
| `worker.debug` | Enable debug mode for worker, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments. | `false` |
| `worker.manageAppDeployments` | Whether apps are deployed and managed by worker. When disabled, every used app is expected to to be already deployed (see apps.enabled). | `true` |
##### app Parameters
###### app Parameters
| Name | Description | Value |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
@@ -789,12 +792,122 @@ The password should be provided with the `SHUFFLE_OPENSEARCH_PASSWORD` env varia
| `app.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
| `app.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
| `app.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy (ignored if allowExternalEgress=true) | `[]` |
| `app.debug` | Enable debug mode for app, which enables extra logging | `false` |
| `app.mountTmpVolume` | Whether a writable /tmp emptyDir volume should be mounted to the app. | `true` |
| `app.exposedContainerPort` | The port that shuffle app containers will listen on for new requests. | `80` |
| `app.sdkTimeout` | The timeout in seconds for app actions. | `300` |
| `app.disableLogs` | Do not capture app logs. By default, app logs are captured, so that they are visible in the frontend. | `false` |
##### Parameters to deploy apps using helm
###### Parameters to deploy apps using helm
| Name | Description | Value |
| ----------------------------- | -------------------------------------------------- | ------- |
| `apps.enabled` | Whether apps should be deployed using helm. | `false` |
| `apps.shuffleTools.enabled` | Whether the shuffle-tools app is enabled | `true` |
| `apps.shuffleTools.version` | The version of the shuffle-tools app to deploy. | `1.2.0` |
| `apps.shuffleSubflow.enabled` | Whether the shuffle-subflow app is enabled | `true` |
| `apps.shuffleSubflow.version` | The version of the shuffle-subflow app to deploy. | `1.1.0` |
| `apps.http.enabled` | Whether the http app is enabled | `true` |
| `apps.http.version` | The version of the http app to deploy. | `1.4.0` |
| `apps.MY_APP.app` | The name of the app (required, e.g. shuffle-tools) | |
| `apps.MY_APP.version` | The version of the app (required, e.g. 1.2.0) | |
###### Traffic Exposure Parameters
| Name | Description | Value |
| -------------------------- | ----------------------------------------------------------------------------------------------------- | --------------- |
| `ingress.enabled` | Enable ingress record generation for frontend and backend | `false` |
| `ingress.pathType` | Ingress path type for the frontend path | `Prefix` |
| `ingress.backendPathType` | Ingress path type for the backend path | `Prefix` |
| `ingress.apiVersion` | Force Ingress API version (automatically detected if not set) | `""` |
| `ingress.hostname` | Default host for the ingress record | `shuffle.local` |
| `ingress.ingressClassName` | IngressClass that will be be used to implement the Ingress (Kubernetes 1.18+) | `nginx` |
| `ingress.path` | Ingress path for Shuffle frontend | `"/"` |
| `ingress.backendPath` | Ingress path for Shuffle backend | `"/api/"` |
| `ingress.annotations` | Additional annotations for the Ingress resource. | `{}` |
| `ingress.tls` | Enable TLS configuration for the host defined at `ingress.hostname` parameter | `false` |
| `ingress.selfSigned` | Create a TLS secret for this ingress record using self-signed certificates generated by Helm | `false` |
| `ingress.extraHosts` | An array with additional hostname(s) to be covered with the ingress record | `[]` |
| `ingress.extraPaths` | An array with additional arbitrary paths that may need to be added to the ingress under the main host | `[]` |
| `ingress.extraTls` | TLS configuration for additional hostname(s) to be covered with this ingress record | `[]` |
| `ingress.secrets` | Custom TLS certificates as secrets | `[]` |
| `ingress.extraRules` | Additional rules to be covered with this ingress record | `[]` |
###### Istio Parameters
| Name | Description | Value |
| --------------------------------------- | ------------------------------------------------------------------------------- | ------------------------ |
| `istio.enabled` | Enable creation of an Istio Gateway and VirtualService for frontend and backend | `false` |
| `istio.apiVersion` | The istio apiVersion to use for Gateway and VirtualService resources | `networking.istio.io/v1` |
| `istio.hosts` | One or more hosts exposed by Istio | `[]` |
| `istio.gateway.annotations` | Additional annotations for the Gateway resource | `{}` |
| `istio.gateway.selector` | The selector matches the ingress gateway pod labels | `{ istio: ingress }` |
| `istio.gateway.http.enabled` | Enable HTTP server port 80 | `true` |
| `istio.gateway.http.httpsRedirect` | If set to true, a 301 redirect is send for all HTTP connections | `false` |
| `istio.gateway.https.enabled` | Enable HTTPS server on port 443 | `false` |
| `istio.gateway.https.tlsCredentialName` | The name of the secret that holds the TLS certs including the CA certificates. | `""` |
| `istio.gateway.https.tlsCipherSuites` | If specified, only support the specified cipher list. | `[]` |
| `istio.gateway.extraServers` | Additional servers for the Gateway resource | `[]` |
| `istio.virtualService.annotations` | Additional annotations for the VirtualService resource. | `{}` |
| `istio.virtualService.backendHeaders` | Header manipulation rules for backend traffic | `{}` |
| `istio.virtualService.frontendHeaders` | Header manipulation rules for frontend traffic | `{}` |
###### Persistence Parameters
| Name | Description | Value |
| ------------------------------------- | ------------------------------------------------- | ------------------- |
| `persistence.enabled` | Enable persistence using Persistent Volume Claims | `true` |
| `persistence.apps.existingClaim` | Name of an existing PVC to use | `""` |
| `persistence.apps.storageClass` | PVC Storage Class for shuffle-apps volume | `""` |
| `persistence.apps.subPath` | The sub path used in the volume | `""` |
| `persistence.apps.accessModes` | The access mode of the volume | `["ReadWriteOnce"]` |
| `persistence.apps.size` | The size of the volume | `5Gi` |
| `persistence.apps.annotations` | Annotations for the PVC | `{}` |
| `persistence.apps.selector` | Selector to match an existing Persistent Volume | `{}` |
| `persistence.appBuilder.storageClass` | PVC Storage Class for backend-apps-claim volume | `""` |
| `persistence.appBuilder.accessModes` | The access mode of the volume | `["ReadWriteOnce"]` |
| `persistence.appBuilder.size` | The size of the volume | `5Gi` |
| `persistence.appBuilder.annotations` | Annotations for the PVC | `{}` |
| `persistence.appBuilder.selector` | Selector to match an existing Persistent Volume | `{}` |
| `persistence.files.existingClaim` | Name of an existing PVC to use | `""` |
| `persistence.files.storageClass` | PVC Storage Class for shuffle-files volume | `""` |
| `persistence.files.subPath` | The sub path used in the volume | `""` |
| `persistence.files.accessModes` | The access mode of the volume | `["ReadWriteOnce"]` |
| `persistence.files.size` | The size of the volume | `5Gi` |
| `persistence.files.annotations` | Annotations for the PVC | `{}` |
| `persistence.files.selector` | Selector to match an existing Persistent Volume | `{}` |
###### Init Container Parameters
| Name | Description | Value |
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `volumePermissions.enabled` | Enable init container that changes the owner/group of the PV mount point to `runAsUser:fsGroup` | `false` |
| `volumePermissions.image.registry` | OS Shell + Utility image registry | `docker.io` |
| `volumePermissions.image.repository` | OS Shell + Utility image repository | `bitnamilegacy/os-shell` |
| `volumePermissions.image.tag` | OS Shell + Utility image tag (immutable tags are recommended) | `12-debian-12-r30` |
| `volumePermissions.image.pullPolicy` | OS Shell + Utility image pull policy | `IfNotPresent` |
| `volumePermissions.image.pullSecrets` | OS Shell + Utility image pull secrets | `[]` |
| `volumePermissions.resourcesPreset` | Set init container resources according to one common preset (allowed values: none, nano, small, medium, large, xlarge, 2xlarge). This is ignored if volumePermissions.resources is set (volumePermissions.resources is recommended for production). | `nano` |
| `volumePermissions.resources` | Set init container requests and limits for different resources like CPU or memory (essential for production workloads) | `{}` |
| `volumePermissions.containerSecurityContext.enabled` | Enabled init container' Security Context | `true` |
| `volumePermissions.containerSecurityContext.seLinuxOptions` | Set SELinux options in init container | `{}` |
| `volumePermissions.containerSecurityContext.runAsUser` | Set init container's Security Context runAsUser | `0` |
###### OpenSearch Parameters
| Name | Description | Value |
| -------------------- | ----------------------------------------------------- | ------ |
| `opensearch.enabled` | Switch to enable or disable the opensearch helm chart | `true` |
###### Vault Parameters
| Name | Description | Value |
| --------------- | -------------------------------------------------------------------------- | ----- |
| `vault.role` | Specify the Vault role, which should be used to get the secret from Vault. | `""` |
| `vault.secrets` | A list of VaultSecrets to create | `[]` |
###### Other Parameters
| Name | Description | Value |
| ----------------------------- | -------------------------------------------------- | ------- |
functions/kubernetes/charts/shuffle/templates/backend/_helpers.tpl
+3
View File
@@ -113,4 +113,7 @@ SHUFFLE_OPENSEARCH_INDEX_PREFIX: "{{ .Values.backend.openSearch.indexPrefix }}"
SHUFFLE_RERUN_SCHEDULE: "{{ .Values.backend.cleanupSchedule }}"
TZ: "{{ .Values.shuffle.timezone }}"
REGISTRY_URL: "{{ .Values.shuffle.appRegistry }}" # Used by app builder
{{- if .Values.backend.debug }}
DEBUG: "true"
{{- end }}
{{- end -}}
functions/kubernetes/charts/shuffle/templates/orborus/_helpers.tpl
+3
View File
@@ -89,6 +89,9 @@ TZ: "{{ .Values.shuffle.timezone }}"
BASE_URL: {{ include "shuffle.backend.baseUrl" . | quote }}
KUBERNETES_NAMESPACE: "{{ .Release.Namespace }}"
SHUFFLE_ORBORUS_EXECUTION_CONCURRENCY: {{ .Values.orborus.executionConcurrency | quote }}
{{- if .Values.orborus.debug }}
DEBUG: "true"
{{- end }}
{{- if .Values.orborus.manageWorkerDeployments }}
# Shuffle worker configuration
functions/kubernetes/charts/shuffle/templates/shuffle-app/_helpers.tpl
+8 -3
View File
@@ -121,9 +121,14 @@ Usage:
{{/*
Return the environment variables of shuffle apps in the format
KEY: VALUE
Usage:
{{- include "shuffle.appInstance.env" (dict "app" $appValues "context" $) -}}
*/}}
{{- define "shuffle.appInstance.env" -}}
SHUFFLE_APP_SDK_TIMEOUT: {{ .Values.app.sdkTimeout | quote }}
SHUFFLE_APP_EXPOSED_PORT: {{ .Values.app.exposedContainerPort | quote }}
SHUFFLE_LOGS_DISABLED: {{ .Values.app.disableLogs | quote }}
SHUFFLE_APP_SDK_TIMEOUT: {{ .app.sdkTimeout | quote }}
SHUFFLE_APP_EXPOSED_PORT: {{ .app.exposedContainerPort | quote }}
SHUFFLE_LOGS_DISABLED: {{ .app.disableLogs | quote }}
{{- if .app.debug }}
DEBUG: "true"
{{- end }}
{{- end -}}
functions/kubernetes/charts/shuffle/templates/shuffle-app/shuffle-apps.yaml
+1 -1
View File
@@ -168,7 +168,7 @@ spec:
value: {{ include "shuffle.backend.baseUrl" $ | quote }}
- name: SHUFFLE_SWARM_CONFIG
value: run # Shuffle Worker requires this to be set even when using K8s instead of swarm
{{- $env := include "shuffle.appInstance.env" $ | fromYaml }}
{{- $env := include "shuffle.appInstance.env" (dict "app" $appValues "context" $) | fromYaml }}
{{- range $key, $val := $env }}
- name: {{ $key | quote }}
value: {{ $val | quote }}
functions/kubernetes/charts/shuffle/templates/shuffle-worker/_helpers.tpl
+4 -1
View File
@@ -139,6 +139,9 @@ SHUFFLE_SWARM_CONFIG: "run" # Shuffle Worker requires this to be set even when u
BASE_URL: {{ include "shuffle.backend.baseUrl" . | quote }}
SHUFFLE_APP_EXPOSED_PORT: {{ .Values.app.exposedContainerPort | quote }}
WORKER_HOSTNAME: {{ include "shuffle.worker.hostname" . }}
{{- if .Values.worker.debug }}
DEBUG: "true"
{{- end }}
{{- if .Values.worker.manageAppDeployments }}
# Shuffle app images
@@ -177,6 +180,6 @@ SHUFFLE_APP_EPHEMERAL_STORAGE_LIMIT: {{ (index $appResources.limits "ephemeral-s
{{- end }}
# Include shuffle app environment variables. Worker passes them down to apps, when creating their deployment.
{{ include "shuffle.appInstance.env" . }}
{{ include "shuffle.appInstance.env" (dict "app" .Values.app "context" $) -}}
{{- end }}
{{- end -}}
functions/kubernetes/charts/shuffle/values.schema.json
+20
View File
@@ -771,6 +771,11 @@
}
}
},
"debug": {
"type": "boolean",
"description": "Enable debug mode for backend, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.",
"default": false
},
"cleanupSchedule": {
"type": "number",
"description": "The interval in seconds at which the cleanup job runs",
@@ -2059,6 +2064,11 @@
}
}
},
"debug": {
"type": "boolean",
"description": "Enable debug mode for orborus, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.",
"default": false
},
"executionConcurrency": {
"type": "number",
"description": "The maximum amount of concurrent workflow executions per worker",
@@ -2703,6 +2713,11 @@
}
}
},
"debug": {
"type": "boolean",
"description": "Enable debug mode for worker, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.",
"default": false
},
"manageAppDeployments": {
"type": "boolean",
"description": "Whether apps are deployed and managed by worker. When disabled, every used app is expected to to be already deployed (see apps.enabled).",
@@ -3322,6 +3337,11 @@
}
}
},
"debug": {
"type": "boolean",
"description": "Enable debug mode for app, which enables extra logging",
"default": false
},
"mountTmpVolume": {
"type": "boolean",
"description": "Whether a writable /tmp emptyDir volume should be mounted to the app.",
functions/kubernetes/charts/shuffle/values.yaml
+15
View File
@@ -516,6 +516,10 @@ backend:
##
extraEgress: []
## @param backend.debug Enable debug mode for backend, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.
##
debug: false
## @param backend.cleanupSchedule The interval in seconds at which the cleanup job runs
##
cleanupSchedule: 300
@@ -1329,6 +1333,10 @@ orborus:
##
extraEgress: []
## @param orborus.debug Enable debug mode for orborus, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.
##
debug: false
## @param orborus.executionConcurrency The maximum amount of concurrent workflow executions per worker
##
executionConcurrency: 25
@@ -1739,6 +1747,10 @@ worker:
##
extraEgress: []
## @param worker.debug Enable debug mode for worker, which enables extra logging. orborus.debug, worker.debug, and app.debug should be set to identical values, if shuffle manages these deployments.
##
debug: false
## @param worker.manageAppDeployments Whether apps are deployed and managed by worker. When disabled, every used app is expected to to be already deployed (see apps.enabled).
## This effectively removes required RBAC permissions from the shuffle-worker service account to create deployments and services.
## The worker might still attempt to create kubernetes objects, resulting in an error. There is currently no way to tell the worker, that it should not manage k8s resources.
@@ -2124,6 +2136,9 @@ app:
##
extraEgress: []
## @param app.debug Enable debug mode for app, which enables extra logging
##
debug: false
## @param app.mountTmpVolume Whether a writable /tmp emptyDir volume should be mounted to the app.
##
mountTmpVolume: true