DSCI Report: dsci@2183
07:06:58 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: main [task run: task.bash - create directory scm] [task stdout] 07:06:59 :: directory path: scm 07:06:59 :: directory owner: <ubuntu> 07:06:59 :: directory group: <ubuntu> 07:06:59 :: directory access rights: drwxr-xr-x scm: http://localhost:8080/test1.git sha: 007aefe message: keep question by dsci@sparrowhub.io - manual run container runtime: podman raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 ======= [task run: task.bash - bash: git config --global advice.detachedHead false] [task stdout] [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 07:06:59 :: /home/ubuntu/.dsci/.sparky/.cache/007aefe.1789628812/.sparrowdo/scm [task stderr] 07:07:00 :: Cloning into '.'... [task run: task.bash - bash: git checkout 007aefe] [task stdout] 07:07:00 :: /home/ubuntu/.dsci/.sparky/.cache/007aefe.1789628812/.sparrowdo/scm [task stderr] 07:07:00 :: HEAD is now at 007aefe keep question pipeline mode: localhost run dsci agent container from image: melezhik/dsci-agent-rakupp ai_agent_message set: 637265617465206473636920706970656c696e6520666f7220432b2b2070726f6a656374207573696e67206d616b65206275696c646572 >>> POST http://127.0.0.1:8181/stash >>> POST http://127.0.0.1:8181/queue >>> wait for jobs, timeout: 600 sec >>> GET http://127.0.0.1:8181/stash/job.run/jnlyhcegstbzvpduwxom.3983205 job test ... [OK] ... >>> GET job report, status=200 07:07:08 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: job-run raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 run job: test === >>> GET http://127.0.0.1:8181/stash/job.run/jnlyhcegstbzvpduwxom.3983205 [task run: task.bash - create directory scm] [task stdout] 07:07:09 :: directory path: scm 07:07:09 :: directory owner: <ubuntu> 07:07:09 :: directory group: <ubuntu> 07:07:09 :: directory access rights: drwxr-xr-x [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 07:07:09 :: /home/ubuntu/.dsci/.sparky/.cache/jnlyhcegstbzvpduwxom.3983205/.sparrowdo/scm [task stderr] 07:07:09 :: Cloning into '.'... [task run: task.bash - bash: git checkout 007aefe] [task stdout] 07:07:10 :: /home/ubuntu/.dsci/.sparky/.cache/jnlyhcegstbzvpduwxom.3983205/.sparrowdo/scm [task stderr] 07:07:10 :: HEAD is now at 007aefe keep question [task run: task.bash - bash: prepare dsci scripts] [task stdout] >>> GET http://127.0.0.1:8181/stash/job.run/jnlyhcegstbzvpduwxom.3983205 [task run: task.bash - .] [task stdout] 07:07:10 :: total 8 07:07:10 :: -rw-r--r-- 1 ubuntu ubuntu 109 Sep 17 07:07 jobs.yaml 07:07:10 :: -rw-r--r-- 1 ubuntu ubuntu 301 Sep 17 07:07 task.bash 07:07:10 :: 637265617465206473636920706970656c696e6520666f7220432b2b2070726f6a656374207573696e67206d616b65206275696c646572 07:07:18 :: === 07:07:18 :: create dsci pipeline for C++ project using make builder 07:07:18 :: === 07:07:18 :: Below is a complete **DSCI** pipeline that builds, tests and (optionally) deploys a generic C++ project that uses a **Makefile**. 07:07:18 :: All heavy‑lifting is done in normal Bash scripts – the only DSCI‑specific parts are the tiny `jobs.yaml` file and the helper calls (`run_task`, `config`, `update_state`, …). 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 1️⃣ Repository layout 07:07:18 :: 07:07:18 :: ``` 07:07:18 :: . 07:07:18 :: ├── .dsci/ 07:07:18 :: │ ├── jobs.yaml # pipeline definition (list of jobs) 07:07:18 :: │ ├── build_job/ # ── Job 1 – compile the project 07:07:18 :: │ │ ├── job.bash 07:07:18 :: │ │ ├── config.yaml # default job parameters 07:07:18 :: │ │ └── tasks/ 07:07:18 :: │ │ └── build/ 07:07:18 :: │ │ └── task.bash 07:07:18 :: │ ├── test_job/ # ── Job 2 – run unit‑tests 07:07:18 :: │ │ ├── job.bash 07:07:18 :: │ │ └── tasks/ 07:07:18 :: │ │ └── test/ 07:07:18 :: │ │ └── task.bash 07:07:18 :: │ └── deploy_job/ # ── Job 3 – optional deployment 07:07:18 :: │ ├── job.bash 07:07:18 :: │ └── tasks/ 07:07:18 :: │ └── deploy/ 07:07:18 :: │ └── task.bash 07:07:18 :: ├── Makefile # (your regular C++ makefile – not shown) 07:07:18 :: └── src/ … # C++ source tree 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: All files that contain DSCI logic are shown below inside **markdown code blocks** labelled with their filenames. 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 2️⃣ `jobs.yaml` – only the job list 07:07:18 :: 07:07:18 :: ```yaml 07:07:18 :: # .dsci/jobs.yaml 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # The pipeline consists of three sequential jobs: 07:07:18 :: # 1️⃣ build_job – compiles the code with `make` 07:07:18 :: # 2️⃣ test_job – runs the test suite 07:07:18 :: # 3️⃣ deploy_job – copies the binary somewhere if tests passed 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: jobs: 07:07:18 :: - id: build_job 07:07:18 :: path: build_job/ 07:07:18 :: # optional override of the default build type (Debug|Release) 07:07:18 :: # params: 07:07:18 :: # BUILD_TYPE: Debug 07:07:18 :: 07:07:18 :: - id: test_job 07:07:18 :: path: test_job/ 07:07:18 :: 07:07:18 :: - id: deploy_job 07:07:18 :: path: deploy_job/ 07:07:18 :: # you could add a `skip: $(config SKIP_DEPLOY) == "true"` flag here 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: *Only the list of jobs (and optional per‑job `params`) is declared – all real logic lives in the script files.* 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 3️⃣ Job 1 – **build_job** 07:07:18 :: 07:07:18 :: ### 3.1 Default parameters (`config.yaml`) 07:07:18 :: 07:07:18 :: ```yaml 07:07:18 :: # .dsci/build_job/config.yaml 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Default values for this job. They can be overridden in `jobs.yaml` 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: BUILD_TYPE: Release # can be “Debug” or “Release” 07:07:18 :: BUILD_TARGET: all # the target you want `make` to run 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ### 3.2 Job entry point (`job.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/build_job/job.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Job “build_job” – simply runs the single “build” task. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: # Run the only task of this job 07:07:18 :: run_task "build" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ### 3.3 Build task (`task.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/build_job/tasks/build/task.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Task “build” – compiles the C++ sources with `make`. 07:07:18 :: # It reads the job parameters via the `config` helper and stores the 07:07:18 :: # produced binary in the shared `~/artifacts` directory. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 1️⃣ Read job parameters (they come from config.yaml or from the pipeline) 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: BUILD_TYPE=$(config BUILD_TYPE) # e.g. Release or Debug 07:07:18 :: BUILD_TARGET=$(config BUILD_TARGET) # default is “all” 07:07:18 :: 07:07:18 :: echo "=== Build job: BUILD_TYPE=${BUILD_TYPE}, BUILD_TARGET=${BUILD_TARGET} ===" 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 2️⃣ Run make. Pass the build type as a variable – adapt to your Makefile. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: make ${BUILD_TARGET} BUILD_TYPE=${BUILD_TYPE} 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 3️⃣ Export the binary as an artifact for downstream jobs. 07:07:18 :: # (Assume the binary is ./bin/app) 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: ARTIFACT_DIR="${HOME}/artifacts" 07:07:18 :: mkdir -p "${ARTIFACT_DIR}" 07:07:18 :: cp ./bin/app "${ARTIFACT_DIR}/app_${BUILD_TYPE}" 07:07:18 :: echo "✅ Built binary copied to ${ARTIFACT_DIR}/app_${BUILD_TYPE}" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 4️⃣ Job 2 – **test_job** 07:07:18 :: 07:07:18 :: ### 4.1 Job entry point (`job.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/test_job/job.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Job “test_job” – executes the unit‑test suite. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: run_task "test" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ### 4.2 Test task (`task.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/test_job/tasks/test/task.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Task “test” – runs `make test` (or any command you prefer) and 07:07:18 :: # records the result in the job state so that later jobs can read it. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: echo "=== Running test suite ===" 07:07:18 :: 07:07:18 :: # Run the tests – adapt the target name to your Makefile. 07:07:18 :: # The command must return a non‑zero exit code on failure, 07:07:18 :: # otherwise the whole job would abort (set -e). 07:07:18 :: if make test; then 07:07:18 :: TEST_STATUS="passed" 07:07:18 :: echo "✅ Tests passed" 07:07:18 :: else 07:07:18 :: TEST_STATUS="failed" 07:07:18 :: echo "❌ Tests failed" 07:07:18 :: fi 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 5️⃣ Export the test status as state (shared with later jobs) 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # IMPORTANT: update_state must receive ONE JSON object. 07:07:18 :: update_state "{ \"test_status\": \"${TEST_STATUS}\" }" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: *The downstream **deploy_job** can read `test_status` from the pipeline state.* 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 5️⃣ Job 3 – **deploy_job** 07:07:18 :: 07:07:18 :: > **Optional** – you can skip this job (or add a `skip:` clause in `jobs.yaml`) if you only want to build & test. 07:07:18 :: 07:07:18 :: ### 5️⃣1 Job entry point (`job.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/deploy_job/job.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Job “deploy_job” – copies the built binary to a deployment folder 07:07:18 :: # only when the previous test job succeeded. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: run_task "deploy" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ### 5️⃣2 Deploy task (`task.bash`) 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/deploy_job/tasks/deploy/task.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # Task “deploy” – reads the test result from the pipeline state, 07:07:18 :: # selects the appropriate artifact, and copies it to a target location. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: set -euo pipefail 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 1️⃣ Read the whole pipeline state (JSON) and pull the test status. 07:07:18 :: # The state is automatically placed under the reserved key "_dsci_". 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: PIPELINE_STATE=$(config) # returns the full JSON 07:07:18 :: TEST_STATUS=$(echo "$PIPELINE_STATE" | jq -r '._dsci_.test_job.test_status') 07:07:18 :: 07:07:18 :: echo "=== Deploy job – test status reported as: ${TEST_STATUS} ===" 07:07:18 :: 07:07:18 :: if [[ "${TEST_STATUS}" != "passed" ]]; then 07:07:18 :: echo "⚠️ Tests did not pass – aborting deployment." 07:07:18 :: exit 0 # not an error, just skip the deployment step 07:07:18 :: fi 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 2️⃣ Locate the built artifact. We stored it as: 07:07:18 :: # ~/artifacts/app_<BUILD_TYPE> 07:07:18 :: # The build type is the same parameter used in the build job. 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: BUILD_TYPE=$(config BUILD_TYPE) # will fall back to default if not overridden 07:07:18 :: ARTIFACT="${HOME}/artifacts/app_${BUILD_TYPE}" 07:07:18 :: 07:07:18 :: if [[ ! -f "${ARTIFACT}" ]]; then 07:07:18 :: echo "❌ Expected artifact ${ARTIFACT} not found" 07:07:18 :: exit 1 07:07:18 :: fi 07:07:18 :: 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: # 3️⃣ Deploy – here we simply copy it to /opt/deploy (adjust to your env) 07:07:18 :: # ---------------------------------------------------------------------- 07:07:18 :: DEPLOY_DIR="/opt/deploy" 07:07:18 :: mkdir -p "${DEPLOY_DIR}" 07:07:18 :: cp "${ARTIFACT}" "${DEPLOY_DIR}/app" 07:07:18 :: 07:07:18 :: echo "🚀 Deployed ${ARTIFACT} → ${DEPLOY_DIR}/app" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: *If you need a more sophisticated deployment (e.g. Docker push, SSH upload, etc.) replace the copy command with the appropriate logic – the same DSCI helpers (`config`, `update_state`, …) remain available.* 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 6️⃣ How the pieces talk together 07:07:18 :: 07:07:18 :: | Component | How it accesses data | 07:07:18 :: |-----------|---------------------| 07:07:18 :: | **build task** | `$(config BUILD_TYPE)` – reads job defaults or pipeline overrides. | 07:07:18 :: | **test task** | `update_state "{ \"test_status\": \"${TEST_STATUS}\" }"` – publishes a JSON object that DSCI stores under `._dsci_.test_job`. | 07:07:18 :: | **deploy task** | `config` → `jq` extracts `._dsci_.test_job.test_status`; also reads `$(config BUILD_TYPE)` to locate the right artifact. | 07:07:18 :: | **Artifacts** | Anything written under `~/artifacts/` in one job is automatically visible to the next jobs. | 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 7️⃣ Quick checklist for a real project 07:07:18 :: 07:07:18 :: 1. **Makefile** must expose at least two targets used above: 07:07:18 :: *`make <BUILD_TARGET> BUILD_TYPE=…`* (default target `all`) and *`make test`*. 07:07:18 :: 2. **Permissions** – ensure every `*.bash` file is executable (`chmod +x`). 07:07:18 :: 3. **Optional overrides** – to build a Debug version only for a specific CI run, add to `jobs.yaml`: 07:07:18 :: 07:07:18 :: ```yaml 07:07:18 :: - id: build_job 07:07:18 :: path: build_job/ 07:07:18 :: params: 07:07:18 :: BUILD_TYPE: Debug 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: 4. **Environment** – DSCI runs each job in a fresh container (default `alpine:latest`). 07:07:18 :: If you need extra packages (e.g. `g++`, `make`), add a small wrapper script or a Dockerfile and set `image: your‑image` in the job definition (the documentation shows how – omitted here for brevity). 07:07:18 :: 07:07:18 :: --- 07:07:18 :: 07:07:18 :: ## 8️⃣ All files together (for copy‑paste) 07:07:18 :: 07:07:18 :: ```yaml 07:07:18 :: # .dsci/jobs.yaml 07:07:18 :: jobs: 07:07:18 :: - id: build_job 07:07:18 :: path: build_job/ 07:07:18 :: - id: test_job 07:07:18 :: path: test_job/ 07:07:18 :: - id: deploy_job 07:07:18 :: path: deploy_job/ 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ```yaml 07:07:18 :: # .dsci/build_job/config.yaml 07:07:18 :: BUILD_TYPE: Release 07:07:18 :: BUILD_TARGET: all 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/build_job/job.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: set -euo pipefail 07:07:18 :: run_task "build" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/build_job/tasks/build/task.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: set -euo pipefail 07:07:18 :: BUILD_TYPE=$(config BUILD_TYPE) 07:07:18 :: BUILD_TARGET=$(config BUILD_TARGET) 07:07:18 :: 07:07:18 :: echo "=== Build job: BUILD_TYPE=${BUILD_TYPE}, BUILD_TARGET=${BUILD_TARGET} ===" 07:07:18 :: make ${BUILD_TARGET} BUILD_TYPE=${BUILD_TYPE} 07:07:18 :: 07:07:18 :: ARTIFACT_DIR="${HOME}/artifacts" 07:07:18 :: mkdir -p "${ARTIFACT_DIR}" 07:07:18 :: cp ./bin/app "${ARTIFACT_DIR}/app_${BUILD_TYPE}" 07:07:18 :: echo "✅ Built binary copied to ${ARTIFACT_DIR}/app_${BUILD_TYPE}" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/test_job/job.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: set -euo pipefail 07:07:18 :: run_task "test" 07:07:18 :: ``` 07:07:18 :: 07:07:18 :: ```bash 07:07:18 :: # .dsci/test_job/tasks/test/task.bash 07:07:18 :: #!/usr/bin/env bash 07:07:18 :: set -euo pipefail 07:07:18 :: echo "=== Running test [task stderr] 07:07:18 :: 07:07:18 :: real 0m8.073s 07:07:18 :: user 0m0.740s 07:07:18 :: sys 0m0.103s saving job artifacts ... >> PUT http://127.0.0.1:8181/file/project/dsci/job/007aefe.1789628812/filename/answer.md >> PUT http://127.0.0.1:8181/file/project/dsci/job/007aefe.1789628812/filename/question.txt >>> POST http://127.0.0.1:8181/stash --- ---