DSCI Report: dsci@2167
14:57:03 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: main [task run: task.bash - create directory scm] [task stdout] 14:57:03 :: directory path: scm 14:57:03 :: directory owner: <ubuntu> 14:57:03 :: directory group: <ubuntu> 14:57:03 :: directory access rights: drwxr-xr-x scm: http://localhost:8080/test1.git sha: 1c9fb55 message: feat: add or update file via dsci web by dsci@sparrowhub.io - manual run container runtime: podman raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 ======= [task run: task.bash - bash: git config --global advice.detachedHead false] [task stdout] [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 14:57:04 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789570619/.sparrowdo/scm [task stderr] 14:57:04 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 14:57:05 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789570619/.sparrowdo/scm [task stderr] 14:57:05 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web pipeline mode: localhost [task run: task.bash - bash: podman stop -t 1 dsci-agent || :] [task stdout] 14:57:07 :: dsci-agent [task stderr] 14:57:07 :: time="2026-09-16T14:57:06Z" level=warning msg="StopSignal SIGTERM failed to stop container dsci-agent in 1 seconds, resorting to SIGKILL" [task run: task.bash - bash: podman rm dsci-agent || :] [task stdout] [task stderr] 14:57:07 :: Error: no container with ID or name "dsci-agent" found: no such container run dsci agent container from image: melezhik/dsci-agent-rakupp podman run --network host -dit --rm --name dsci-agent --entrypoint /bin/sh melezhik/dsci-agent-rakupp [task run: task.bash - bash: podman run --network host -dit --rm --name dsci-a ...] [task stdout] 14:57:08 :: e6a9c956e557e751978639f4d8120f3857aaeaf7f0e113cdacd30b7503820c82 [task stderr] 14:57:08 :: time="2026-09-16T14:57:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" ai_agent_message set: 4275696c64204453434920706970656c696e6520666f72207479706963616c20476f6c616e672070726f6a656374207769746820756e69742074657374732c20636f646520636f766572616765206c657373207468656e205820616e64207075626c697368696e6720646f636b657220696d61676520746f206120726567697374727920 >>> POST http://127.0.0.1:8181/stash >>> POST http://127.0.0.1:8181/queue >>> wait for jobs, timeout: 600 sec >>> GET http://127.0.0.1:8181/stash/job.run/zjlxqgftaeocpbrmidsy.3907828 job test ... [OK] ... >>> GET job report, status=200 14:57:13 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: job-run raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 run job: test === >>> GET http://127.0.0.1:8181/stash/job.run/zjlxqgftaeocpbrmidsy.3907828 [task run: task.bash - create directory scm] [task stdout] 14:57:13 :: directory path: scm 14:57:13 :: directory owner: <ubuntu> 14:57:13 :: directory group: <ubuntu> 14:57:13 :: directory access rights: drwxr-xr-x [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 14:57:14 :: /home/ubuntu/.dsci/.sparky/.cache/zjlxqgftaeocpbrmidsy.3907828/.sparrowdo/scm [task stderr] 14:57:14 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 14:57:14 :: /home/ubuntu/.dsci/.sparky/.cache/zjlxqgftaeocpbrmidsy.3907828/.sparrowdo/scm [task stderr] 14:57:14 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web [task run: task.bash - bash: prepare dsci scripts] [task stdout] >>> GET http://127.0.0.1:8181/stash/job.run/zjlxqgftaeocpbrmidsy.3907828 [task run: task.bash - .] [task stdout] 14:57:14 :: total 8 14:57:14 :: -rw-r--r-- 1 ubuntu ubuntu 109 Sep 16 14:57 jobs.yaml 14:57:14 :: -rw-r--r-- 1 ubuntu ubuntu 301 Sep 16 14:57 task.bash 14:57:14 :: 4275696c64204453434920706970656c696e6520666f72207479706963616c20476f6c616e672070726f6a656374207769746820756e69742074657374732c20636f646520636f766572616765206c657373207468656e205820616e64207075626c697368696e6720646f636b657220696d61676520746f206120726567697374727920 14:57:23 :: === 14:57:23 :: Build DSCI pipeline for typical Golang project with unit tests, code coverage less then X and publishing docker image to a registry 14:57:23 :: === 14:57:23 :: **File structure** 14:57:23 :: 14:57:23 :: ``` 14:57:23 :: . 14:57:23 :: ├── .dsci 14:57:23 :: │ └── jobs.yaml 14:57:23 :: ├── build 14:57:23 :: │ ├── config.yaml 14:57:23 :: │ ├── job.bash 14:57:23 :: │ └── tasks 14:57:23 :: │ ├── compile 14:57:23 :: │ │ └── task.bash 14:57:23 :: │ └── store_binary 14:57:23 :: │ └── task.bash 14:57:23 :: ├── test 14:57:23 :: │ ├── config.yaml 14:57:23 :: │ ├── job.bash 14:57:23 :: │ └── tasks 14:57:23 :: │ ├── run_tests 14:57:23 :: │ │ └── task.bash 14:57:23 :: │ └── check_coverage 14:57:23 :: │ └── task.py 14:57:23 :: ├── publish 14:57:23 :: │ ├── config.yaml 14:57:23 :: │ ├── job.bash 14:57:23 :: │ └── tasks 14:57:23 :: │ ├── docker_build 14:57:23 :: │ │ └── task.bash 14:57:23 :: │ └── docker_push 14:57:23 :: │ └── task.bash 14:57:23 :: └── Dockerfile # (assumed to exist for the Go binary) 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: --- 14:57:23 :: 14:57:23 :: ## 1️⃣ Pipeline definition – **`.dsci/jobs.yaml`** 14:57:23 :: 14:57:23 :: Only the list of jobs (and optional **input parameters**) is kept in the top‑level YAML. 14:57:23 :: 14:57:23 :: ```yaml 14:57:23 :: # .dsci/jobs.yaml 14:57:23 :: jobs: 14:57:23 :: - 14:57:23 :: id: build 14:57:23 :: path: build/ 14:57:23 :: params: 14:57:23 :: binary_name: myapp # name of the compiled binary 14:57:23 :: - 14:57:23 :: id: test 14:57:23 :: path: test/ 14:57:23 :: params: 14:57:23 :: coverage_threshold: 80 # required coverage % (fail if lower) 14:57:23 :: - 14:57:23 :: id: publish 14:57:23 :: path: publish/ 14:57:23 :: params: 14:57:23 :: docker_registry: registry.example.com 14:57:23 :: image_name: myorg/myapp 14:57:23 :: image_tag: latest 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: *No job dependencies, matrices or third‑party actions – just a plain list of jobs.* 14:57:23 :: 14:57:23 :: --- 14:57:23 :: 14:57:23 :: ## 2️⃣ **Build** job – `build/` 14:57:23 :: 14:57:23 :: ### 2.1 Default parameters – `build/config.yaml` 14:57:23 :: 14:57:23 :: ```yaml 14:57:23 :: # build/config.yaml 14:57:23 :: binary_name: myapp # can be overridden by jobs.yaml params 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 2.2 Job entry point – `build/job.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # build/job.bash 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # Build job – compiles the Go binary and stores it as an artifact. 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: 14:57:23 :: # run the compile task 14:57:23 :: run_task compile 14:57:23 :: 14:57:23 :: # store the resulting binary in the shared artifacts directory 14:57:23 :: run_task store_binary 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 2.3 Tasks 14:57:23 :: 14:57:23 :: #### `build/tasks/compile/task.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # build/tasks/compile/task.bash 14:57:23 :: # Compile the Go binary 14:57:23 :: 14:57:23 :: set -euo pipefail 14:57:23 :: 14:57:23 :: # read the binary name (job parameter) 14:57:23 :: binary_name=$(config binary_name) 14:57:23 :: 14:57:23 :: # build the binary (assumes Go modules are present at repo root) 14:57:23 :: echo "🔨 Building Go binary '${binary_name}'..." 14:57:23 :: go build -o "${binary_name}" ./... 14:57:23 :: 14:57:23 :: # export the binary path as a state so downstream jobs can read it 14:57:23 :: update_state "{ \"binary_path\": \"${PWD}/${binary_name}\" }" 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: #### `build/tasks/store_binary/task.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # build/tasks/store_binary/task.bash 14:57:23 :: # Copy the binary to the DSCI artifacts directory (~/artifacts) 14:57:23 :: 14:57:23 :: set -euo pipefail 14:57:23 :: 14:57:23 :: # get the path from the state set by the previous task 14:57:23 :: binary_path=$(get_state | jq -r '._dsci_.build.binary_path') 14:57:23 :: 14:57:23 :: echo "📦 Storing binary at ~/artifacts/${binary_path##*/}" 14:57:23 :: mkdir -p ~/artifacts 14:57:23 :: cp "${binary_path}" ~/artifacts/ 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: --- 14:57:23 :: 14:57:23 :: ## 3️⃣ **Test** job – `test/` 14:57:23 :: 14:57:23 :: ### 3.1 Default parameters – `test/config.yaml` 14:57:23 :: 14:57:23 :: ```yaml 14:57:23 :: # test/config.yaml 14:57:23 :: coverage_threshold: 80 # % – can be overridden in jobs.yaml 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 3.2 Job entry point – `test/job.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # test/job.bash 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # Test job – runs `go test` with coverage, checks the threshold. 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: 14:57:23 :: # execute the unit‑test task 14:57:23 :: run_task run_tests 14:57:23 :: 14:57:23 :: # evaluate the coverage percentage 14:57:23 :: run_task check_coverage 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 3.3 Tasks 14:57:23 :: 14:57:23 :: #### `test/tasks/run_tests/task.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # test/tasks/run_tests/task.bash 14:57:23 :: # Run Go tests and generate a coverage profile 14:57:23 :: 14:57:23 :: set -euo pipefail 14:57:23 :: 14:57:23 :: COVERAGE_FILE="coverage.out" 14:57:23 :: 14:57:23 :: echo "🧪 Running unit tests with coverage..." 14:57:23 :: go test ./... -coverprofile="${COVERAGE_FILE}" -covermode=atomic 14:57:23 :: 14:57:23 :: # store the coverage file as an artifact (shared with the next task) 14:57:23 :: mkdir -p ~/artifacts 14:57:23 :: cp "${COVERAGE_FILE}" ~/artifacts/ 14:57:23 :: 14:57:23 :: # also expose the path via state (optional) 14:57:23 :: update_state "{ \"coverage_file\": \"${PWD}/${COVERAGE_FILE}\" }" 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: #### `test/tasks/check_coverage/task.py` 14:57:23 :: 14:57:23 :: ```python 14:57:23 :: #!/usr/bin/env python3 14:57:23 :: # test/tasks/check_coverage/task.py 14:57:23 :: # Parse the coverage report, compare with the required threshold, 14:57:23 :: # and fail the job if the threshold is not met. 14:57:23 :: 14:57:23 :: import subprocess 14:57:23 :: import sys 14:57:23 :: import json 14:57:23 :: from pathlib import Path 14:57:23 :: 14:57:23 :: # ---------------------------------------------------------------------- 14:57:23 :: # Helper to read DSCI configuration / state 14:57:23 :: # ---------------------------------------------------------------------- 14:57:23 :: def config(): 14:57:23 :: """Return the merged job configuration (params + defaults).""" 14:57:23 :: # DSCI injects a `config` function at runtime; we just call it. 14:57:23 :: # The implementation is provided by the DSCI runtime. 14:57:23 :: ... 14:57:23 :: 14:57:23 :: def get_state(): 14:57:23 :: """Return the current job state as a Python dict.""" 14:57:23 :: ... 14:57:23 :: 14:57:23 :: # ---------------------------------------------------------------------- 14:57:23 :: # Main logic 14:57:23 :: # ---------------------------------------------------------------------- 14:57:23 :: def main(): 14:57:23 :: cfg = config() 14:57:23 :: threshold = float(cfg.get("coverage_threshold", 0)) 14:57:23 :: 14:57:23 :: # locate the coverage file (saved as an artifact by previous task) 14:57:23 :: coverage_path = Path.home() / "artifacts" / "coverage.out" 14:57:23 :: if not coverage_path.is_file(): 14:57:23 :: print(f"❌ Coverage file not found at {coverage_path}", file=sys.stderr) 14:57:23 :: sys.exit(1) 14:57:23 :: 14:57:23 :: # Use `go tool cover -func` to get a line like: 14:57:23 :: # total: (statements) 85.3% 14:57:23 :: result = subprocess.run( 14:57:23 :: ["go", "tool", "cover", "-func", str(coverage_path)], 14:57:23 :: capture_output=True, 14:57:23 :: text=True, 14:57:23 :: check=True, 14:57:23 :: ) 14:57:23 :: 14:57:23 :: # Extract the total coverage percentage 14:57:23 :: total_line = next( 14:57:23 :: (ln for ln in result.stdout.splitlines() if ln.startswith("total:")), None 14:57:23 :: ) 14:57:23 :: if not total_line: 14:57:23 :: print("❌ Could not find total coverage line.", file=sys.stderr) 14:57:23 :: sys.exit(1) 14:57:23 :: 14:57:23 :: coverage_percent = float(total_line.split()[-1].replace("%", "")) 14:57:23 :: print(f"📈 Total coverage: {coverage_percent:.2f}% (threshold: {threshold}%)") 14:57:23 :: 14:57:23 :: # Save the percentage in the state for downstream jobs (optional) 14:57:23 :: update_state(json.dumps({"coverage_percent": coverage_percent})) 14:57:23 :: 14:57:23 :: if coverage_percent < threshold: 14:57:23 :: print( 14:57:23 :: f"❌ Coverage {coverage_percent:.2f}% is below the required {threshold}%!", 14:57:23 :: file=sys.stderr, 14:57:23 :: ) 14:57:23 :: sys.exit(1) 14:57:23 :: 14:57:23 :: print("✅ Coverage threshold satisfied.") 14:57:23 :: 14:57:23 :: 14:57:23 :: if __name__ == "__main__": 14:57:23 :: main() 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: --- 14:57:23 :: 14:57:23 :: ## 4️⃣ **Publish** job – `publish/` 14:57:23 :: 14:57:23 :: ### 4.1 Default parameters – `publish/config.yaml` 14:57:23 :: 14:57:23 :: ```yaml 14:57:23 :: # publish/config.yaml 14:57:23 :: docker_registry: registry.example.com 14:57:23 :: image_name: myorg/myapp 14:57:23 :: image_tag: latest 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 4.2 Job entry point – `publish/job.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # publish/job.bash 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # Publish job – builds a Docker image from the compiled binary and 14:57:23 :: # pushes it to a registry. 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: 14:57:23 :: # Build the Docker image 14:57:23 :: run_task docker_build 14:57:23 :: 14:57:23 :: # Push the Docker image 14:57:23 :: run_task docker_push 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: ### 4.3 Tasks 14:57:23 :: 14:57:23 :: #### `publish/tasks/docker_build/task.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # publish/tasks/docker_build/task.bash 14:57:23 :: # Build a Docker image that contains the compiled Go binary. 14:57:23 :: 14:57:23 :: set -euo pipefail 14:57:23 :: 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # 1. Resolve configuration 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: binary_name=$(config binary_name) # comes from build job defaults 14:57:23 :: registry=$(config docker_registry) 14:57:23 :: image_name=$(config image_name) 14:57:23 :: image_tag=$(config image_tag) 14:57:23 :: 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # 2. Locate the binary saved as an artifact by the build job 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: binary_path=$(get_state | jq -r '._dsci_.build.binary_path // ""') 14:57:23 :: if [[ -z "$binary_path" ]]; then 14:57:23 :: # fallback – the binary should also be present in ~/artifacts 14:57:23 :: binary_path="${HOME}/artifacts/${binary_name}" 14:57:23 :: fi 14:57:23 :: 14:57:23 :: if [[ ! -f "$binary_path" ]]; then 14:57:23 :: echo "❌ Compiled binary not found at $binary_path" 14:57:23 :: exit 1 14:57:23 :: fi 14:57:23 :: 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: # 3. Build the Docker image 14:57:23 :: # ------------------------------------------------------------------ 14:57:23 :: full_image="${registry}/${image_name}:${image_tag}" 14:57:23 :: echo "🐳 Building Docker image ${full_image}..." 14:57:23 :: 14:57:23 :: # We copy the binary into the current workdir because Dockerfile 14:57:23 :: # expects a file called ${binary_name} 14:57:23 :: cp "${binary_path}" "./${binary_name}" 14:57:23 :: 14:57:23 :: docker build \ 14:57:23 :: --pull \ 14:57:23 :: -t "${full_image}" \ 14:57:23 :: --build-arg BINARY_NAME="${binary_name}" \ 14:57:23 :: . 14:57:23 :: 14:57:23 :: # expose image name via state (optional, useful for downstream jobs) 14:57:23 :: update_state "{ \"docker_image\": \"${full_image}\" }" 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: > **Dockerfile example** (must exist at repository root) 14:57:23 :: 14:57:23 :: ```Dockerfile 14:57:23 :: # Dockerfile (used by the publish job) 14:57:23 :: FROM alpine:3.20 AS runtime 14:57:23 :: ARG BINARY_NAME 14:57:23 :: COPY ${BINARY_NAME} /usr/local/bin/${BINARY_NAME} 14:57:23 :: RUN chmod +x /usr/local/bin/${BINARY_NAME} 14:57:23 :: ENTRYPOINT ["/usr/local/bin/${BINARY_NAME}"] 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: #### `publish/tasks/docker_push/task.bash` 14:57:23 :: 14:57:23 :: ```bash 14:57:23 :: #!/usr/bin/env bash 14:57:23 :: # publish/tasks/docker_push/task.bash 14:57:23 :: # Push the Docker image built in the previous task. 14:57:23 :: 14:57:23 :: set -euo pipefail 14:57:23 :: 14:57:23 :: registry=$(config docker_registry) 14:57:23 :: image_name=$(config image_name) 14:57:23 :: image_tag=$(config image_tag) 14:57:23 :: 14:57:23 :: full_image="${registry}/${image_name}:${image_tag}" 14:57:23 :: 14:57:23 :: echo "📤 Pushing Docker image ${full_image} to registry ${registry}..." 14:57:23 :: 14:57:23 :: # Login – assumes the runner provides DOCKER_USERNAME / DOCKER_PASSWORD env vars 14:57:23 :: if [[ -n "${DOCKER_USERNAME:-}" && -n "${DOCKER_PASSWORD:-}" ]]; then 14:57:23 :: echo "${DOCKER_PASSWORD}" | docker login "${registry}" -u "${DOCKER_USERNAME}" --password-stdin 14:57:23 :: fi 14:57:23 :: 14:57:23 :: docker push "${full_image}" 14:57:23 :: echo "✅ Image pushed successfully." 14:57:23 :: ``` 14:57:23 :: 14:57:23 :: --- 14:57:23 :: 14:57:23 :: ## 5️⃣ How the pieces talk together 14:57:23 :: 14:57:23 :: | Job / Task | What it does | How data is passed | 14:57:23 :: |------------|--------------|-------------------| 14:57:23 :: | **build → compile** | `go build -o <binary>` | `update_state` → `binary_path` | 14:57:23 :: | **build → store_binary** | Copies binary to `~/artifacts/` | Binary is now a file that any later job can read | 14:57:23 :: | **test → run_tests** | `go test -coverprofile=coverage.out` | Writes `coverage.out` to `~/artifacts/` | 14:57:23 :: | **test → check_coverage** | Python parses `coverage.out`, compares with `coverage_threshold` (from `config()`), `update_state` → `coverage_percent` | 14:57:23 :: | **publish → docker_build** | Reads binary from `~/artifacts/` (or from state), builds Docker image, `update_state` → `docker_image` | 14:57:23 :: | **publish → docker_push** | [task stderr] 14:57:23 :: 14:57:23 :: real 0m8.245s 14:57:23 :: user 0m0.691s 14:57:23 :: sys 0m0.104s saving job artifacts ... >> PUT http://127.0.0.1:8181/file/project/dsci/job/1c9fb55.1789570619/filename/answer.md >>> POST http://127.0.0.1:8181/stash --- ---