DSCI Report: dsci@2165
14:54:23 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: main [task run: task.bash - create directory scm] [task stdout] 14:54:23 :: directory path: scm 14:54:23 :: directory owner: <ubuntu> 14:54:23 :: directory group: <ubuntu> 14:54:23 :: directory access rights: drwxr-xr-x scm: http://localhost:8080/test1.git sha: 1c9fb55 message: feat: add or update file via dsci web by dsci@sparrowhub.io - manual run container runtime: podman raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 ======= [task run: task.bash - bash: git config --global advice.detachedHead false] [task stdout] [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 14:54:24 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789570454/.sparrowdo/scm [task stderr] 14:54:24 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 14:54:24 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789570454/.sparrowdo/scm [task stderr] 14:54:24 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web pipeline mode: localhost [task run: task.bash - bash: podman stop -t 1 dsci-agent || :] [task stdout] 14:54:26 :: dsci-agent [task stderr] 14:54:26 :: time="2026-09-16T14:54:26Z" level=warning msg="StopSignal SIGTERM failed to stop container dsci-agent in 1 seconds, resorting to SIGKILL" [task run: task.bash - bash: podman rm dsci-agent || :] [task stdout] [task stderr] 14:54:26 :: Error: no container with ID or name "dsci-agent" found: no such container run dsci agent container from image: melezhik/dsci-agent-rakupp podman run --network host -dit --rm --name dsci-agent --entrypoint /bin/sh melezhik/dsci-agent-rakupp [task run: task.bash - bash: podman run --network host -dit --rm --name dsci-a ...] [task stdout] 14:54:27 :: 1dc081de85eb923646bfece4d6fb38a5e6299fda2468375d9cb7c4d4db720836 [task stderr] 14:54:27 :: time="2026-09-16T14:54:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" ai_agent_message set: 4275696c64204453434920706970656c696e6520666f72207479706963616c20507974686f6e2070726f6a65637420776974682070797465737420756e697420746573747320616e6420636f646520636f766572616765207769746820636f766572616765207468726573686f6c64206c657373207468656e20580d0a >>> POST http://127.0.0.1:8181/stash >>> POST http://127.0.0.1:8181/queue >>> wait for jobs, timeout: 600 sec >>> GET http://127.0.0.1:8181/stash/job.run/yaoceibrjvkqpduhmlns.3907140 job test ... [OK] ... >>> GET job report, status=200 14:54:33 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: job-run raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 run job: test === >>> GET http://127.0.0.1:8181/stash/job.run/yaoceibrjvkqpduhmlns.3907140 [task run: task.bash - create directory scm] [task stdout] 14:54:33 :: directory path: scm 14:54:33 :: directory owner: <ubuntu> 14:54:33 :: directory group: <ubuntu> 14:54:33 :: directory access rights: drwxr-xr-x [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 14:54:34 :: /home/ubuntu/.dsci/.sparky/.cache/yaoceibrjvkqpduhmlns.3907140/.sparrowdo/scm [task stderr] 14:54:34 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 14:54:34 :: /home/ubuntu/.dsci/.sparky/.cache/yaoceibrjvkqpduhmlns.3907140/.sparrowdo/scm [task stderr] 14:54:34 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web [task run: task.bash - bash: prepare dsci scripts] [task stdout] >>> GET http://127.0.0.1:8181/stash/job.run/yaoceibrjvkqpduhmlns.3907140 [task run: task.bash - .] [task stdout] 14:54:34 :: total 8 14:54:34 :: -rw-r--r-- 1 ubuntu ubuntu 109 Sep 16 14:54 jobs.yaml 14:54:34 :: -rw-r--r-- 1 ubuntu ubuntu 301 Sep 16 14:54 task.bash 14:54:34 :: 4275696c64204453434920706970656c696e6520666f72207479706963616c20507974686f6e2070726f6a65637420776974682070797465737420756e697420746573747320616e6420636f646520636f766572616765207769746820636f766572616765207468726573686f6c64206c657373207468656e20580d0a 14:54:40 :: === 14:54:40 :: Build DSCI pipeline for typical Python project with pytest unit tests and code coverage with coverage threshold less then X 14:54:40 :: 14:54:40 :: === 14:54:40 :: Below is a **complete, ready‑to‑use DSCI pipeline** that builds a typical Python project, runs its unit‑tests with **pytest**, measures code‑coverage with **coverage.py**, and fails the pipeline when the overall coverage is **below a configurable threshold**. 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 1️⃣ Repository layout 14:54:40 :: 14:54:40 :: ```text 14:54:40 :: . 14:54:40 :: ├── .dsci/ 14:54:40 :: │ ├── jobs.yaml # pipeline definition (only lists jobs) 14:54:40 :: │ └── job_test/ # single job that does everything 14:54:40 :: │ ├── config.yaml # default job parameters (e.g. coverage threshold) 14:54:40 :: │ ├── job.bash # entry‑point for the job – runs the tasks 14:54:40 :: │ └── tasks/ # individual steps 14:54:40 :: │ ├── install/ 14:54:40 :: │ │ └── task.bash 14:54:40 :: │ ├── test/ 14:54:40 :: │ │ └── task.bash 14:54:40 :: │ └── coverage/ 14:54:40 :: │ └── task.bash 14:54:40 :: ├── requirements.txt # Python dependencies (pytest, coverage, …) 14:54:40 :: └── src/ # your application code (example placeholder) 14:54:40 :: └── __init__.py 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: All **pipeline logic** lives inside the `job_test` folder; the top‑level `jobs.yaml` only tells DSCI *which* job to execute. 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 2️⃣ `jobs.yaml` – pipeline definition 14:54:40 :: 14:54:40 :: ```yaml 14:54:40 :: # .dsci/jobs.yaml 14:54:40 :: # Only the list of jobs – no matrix, no extra YAML tricks 14:54:40 :: jobs: 14:54:40 :: - id: test 14:54:40 :: path: job_test/ 14:54:40 :: # optional: override the default coverage threshold here 14:54:40 :: # params: 14:54:40 :: # coverage_threshold: 85 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 3️⃣ Job‑wide defaults (`config.yaml`) 14:54:40 :: 14:54:40 :: ```yaml 14:54:40 :: # .dsci/job_test/config.yaml 14:54:40 :: # Default values that can be overridden from the pipeline (jobs.yaml) or CLI 14:54:40 :: coverage_threshold: 80 # fail if total coverage < 80 % 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 4️⃣ Job entry point (`job.bash`) 14:54:40 :: 14:54:40 :: ```bash 14:54:40 :: # .dsci/job_test/job.bash 14:54:40 :: #!/usr/bin/env bash 14:54:40 :: set -euo pipefail 14:54:40 :: 14:54:40 :: # Run the three tasks in the required order 14:54:40 :: run_task "install" 14:54:40 :: run_task "test" 14:54:40 :: run_task "coverage" 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 5️⃣ Tasks 14:54:40 :: 14:54:40 :: ### 5.1 Install dependencies 14:54:40 :: 14:54:40 :: ```bash 14:54:40 :: # .dsci/job_test/tasks/install/task.bash 14:54:40 :: #!/usr/bin/env bash 14:54:40 :: set -euo pipefail 14:54:40 :: 14:54:40 :: echo "=== 📦 Installing Python dependencies ===" 14:54:40 :: python -m pip install --upgrade pip 14:54:40 :: pip install -r requirements.txt 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ### 5.2 Run unit‑tests with **pytest** (generates a coverage data file) 14:54:40 :: 14:54:40 :: ```bash 14:54:40 :: # .dsci/job_test/tasks/test/task.bash 14:54:40 :: #!/usr/bin/env bash 14:54:40 :: set -euo pipefail 14:54:40 :: 14:54:40 :: echo "=== 🧪 Running pytest with coverage ===" 14:54:40 :: # –‑cov=src measures coverage for the src/ package 14:54:40 :: coverage run -m pytest 14:54:40 :: # Save the raw .coverage file as an artifact so the next task can reuse it 14:54:40 :: mkdir -p ~/artifacts 14:54:40 :: cp .coverage ~/artifacts/ 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ### 5.3 Check coverage against the threshold 14:54:40 :: 14:54:40 :: ```bash 14:54:40 :: # .dsci/job_test/tasks/coverage/task.bash 14:54:40 :: #!/usr/bin/env bash 14:54:40 :: set -euo pipefail 14:54:40 :: 14:54:40 :: # ------------------------------------------------------------------ 14:54:40 :: # Retrieve the threshold (default 80) – can be overridden in jobs.yaml 14:54:40 :: # ------------------------------------------------------------------ 14:54:40 :: threshold=$(config coverage_threshold) 14:54:40 :: 14:54:40 :: echo "=== 📈 Evaluating coverage (threshold = ${threshold}%) ===" 14:54:40 :: 14:54:40 :: # Use the .coverage file produced by the previous task 14:54:40 :: if [[ ! -f ~/artifacts/.coverage ]]; then 14:54:40 :: echo "❌ No coverage data found!" 14:54:40 :: exit 1 14:54:40 :: fi 14:54:40 :: cp ~/artifacts/.coverage .coverage 14:54:40 :: 14:54:40 :: # Produce a human‑readable report and also store it as an artifact 14:54:40 :: coverage report -m | tee ~/artifacts/coverage.txt 14:54:40 :: 14:54:40 :: # Extract the total % from the report (format: TOTAL xx.xx%) 14:54:40 :: total=$(coverage report -m | awk '/TOTAL/ {gsub("%","",$4); print $4}') 14:54:40 :: 14:54:40 :: # Guard against missing data 14:54:40 :: if [[ -z "$total" ]]; then 14:54:40 :: echo "❌ Could not determine total coverage" 14:54:40 :: exit 1 14:54:40 :: fi 14:54:40 :: 14:54:40 :: # Compare using bc (handles floating‑point numbers) 14:54:40 :: if (( $(echo "$total < $threshold" | bc -l) )); then 14:54:40 :: echo "❌ Coverage $total% is below the required $threshold%" 14:54:40 :: exit 1 14:54:40 :: else 14:54:40 :: echo "✅ Coverage $total% meets the required $threshold%" 14:54:40 :: fi 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 6️⃣ How the pieces fit together 14:54:40 :: 14:54:40 :: | Component | What it does | How it talks to the rest | 14:54:40 :: |-----------|--------------|--------------------------| 14:54:40 :: | **`jobs.yaml`** | Lists the single job `test` | DSCI reads it and executes the job at `job_test/` | 14:54:40 :: | **`config.yaml`** | Supplies default parameters (e.g. `coverage_threshold`) | Accessible via `$(config name)` in Bash tasks | 14:54:40 :: | **`job.bash`** | Orchestrates the three tasks using `run_task` | Calls each task in order | 14:54:40 :: | **`install/task.bash`** | Installs `requirements.txt` | No special communication needed | 14:54:40 :: | **`test/task.bash`** | Executes `pytest` under `coverage` and stores raw data in `~/artifacts/` | Shares the `.coverage` file via the artifacts folder | 14:54:40 :: | **`coverage/task.bash`** | Reads the threshold, generates a readable report, checks the total % and fails the job if below threshold | Uses `$(config coverage_threshold)` and the artifact created by the previous task | 14:54:40 :: 14:54:40 :: All **state sharing** is performed through the `~/artifacts/` directory, which DSCI automatically mounts across tasks of the same job. 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ## 7️⃣ Extending / Overriding the threshold 14:54:40 :: 14:54:40 :: If you need a stricter (or looser) threshold for a particular pipeline run, simply add a `params` block to the job entry in `jobs.yaml`: 14:54:40 :: 14:54:40 :: ```yaml 14:54:40 :: jobs: 14:54:40 :: - id: test 14:54:40 :: path: job_test/ 14:54:40 :: params: 14:54:40 :: coverage_threshold: 90 # require 90 % coverage for this run 14:54:40 :: ``` 14:54:40 :: 14:54:40 :: The `$(config coverage_threshold)` call inside `coverage/task.bash` will pick up the overridden value automatically. 14:54:40 :: 14:54:40 :: --- 14:54:40 :: 14:54:40 :: ### 🎉 Done! 14:54:40 :: 14:54:40 :: You now have a **minimal yet fully functional DSCI pipeline** that: 14:54:40 :: 14:54:40 :: 1. Installs the project’s Python dependencies. 14:54:40 :: 2. Runs all `pytest` unit‑tests while collecting coverage data. 14:54:40 :: 3. Enforces a configurable coverage threshold, failing the pipeline if the threshold is not met. 14:54:40 :: 14:54:40 :: All logic lives in clean Bash scripts (the simplest language for these steps), while configuration values are handled via the built‑in `config()` SDK function, exactly as required by DSCI. [task stderr] 14:54:40 :: 14:54:40 :: real 0m5.533s 14:54:40 :: user 0m0.647s 14:54:40 :: sys 0m0.131s saving job artifacts ... >> PUT http://127.0.0.1:8181/file/project/dsci/job/1c9fb55.1789570454/filename/answer.md >>> POST http://127.0.0.1:8181/stash --- ---