DSCI

Below is a **complete, ready‑to‑use DSCI pipeline** that builds a typical Python project, runs its unit‑tests with **pytest**, measures code‑coverage with **coverage.py**, and fails the pipeline when the overall coverage is **below a configurable threshold**. --- ## 1️⃣ Repository layout ```text . ├── .dsci/ │ ├── jobs.yaml # pipeline definition (only lists jobs) │ └── job_test/ # single job that does everything │ ├── config.yaml # default job parameters (e.g. coverage threshold) │ ├── job.bash # entry‑point for the job – runs the tasks │ └── tasks/ # individual steps │ ├── install/ │ │ └── task.bash │ ├── test/ │ │ └── task.bash │ └── coverage/ │ └── task.bash ├── requirements.txt # Python dependencies (pytest, coverage, …) └── src/ # your application code (example placeholder) └── __init__.py ``` All **pipeline logic** lives inside the `job_test` folder; the top‑level `jobs.yaml` only tells DSCI *which* job to execute. --- ## 2️⃣ `jobs.yaml` – pipeline definition ```yaml # .dsci/jobs.yaml # Only the list of jobs – no matrix, no extra YAML tricks jobs: - id: test path: job_test/ # optional: override the default coverage threshold here # params: # coverage_threshold: 85 ``` --- ## 3️⃣ Job‑wide defaults (`config.yaml`) ```yaml # .dsci/job_test/config.yaml # Default values that can be overridden from the pipeline (jobs.yaml) or CLI coverage_threshold: 80 # fail if total coverage < 80 % ``` --- ## 4️⃣ Job entry point (`job.bash`) ```bash # .dsci/job_test/job.bash #!/usr/bin/env bash set -euo pipefail # Run the three tasks in the required order run_task "install" run_task "test" run_task "coverage" ``` --- ## 5️⃣ Tasks ### 5.1 Install dependencies ```bash # .dsci/job_test/tasks/install/task.bash #!/usr/bin/env bash set -euo pipefail echo "=== 📦 Installing Python dependencies ===" python -m pip install --upgrade pip pip install -r requirements.txt ``` --- ### 5.2 Run unit‑tests with **pytest** (generates a coverage data file) ```bash # .dsci/job_test/tasks/test/task.bash #!/usr/bin/env bash set -euo pipefail echo "=== 🧪 Running pytest with coverage ===" # –‑cov=src measures coverage for the src/ package coverage run -m pytest # Save the raw .coverage file as an artifact so the next task can reuse it mkdir -p ~/artifacts cp .coverage ~/artifacts/ ``` --- ### 5.3 Check coverage against the threshold ```bash # .dsci/job_test/tasks/coverage/task.bash #!/usr/bin/env bash set -euo pipefail # ------------------------------------------------------------------ # Retrieve the threshold (default 80) – can be overridden in jobs.yaml # ------------------------------------------------------------------ threshold=$(config coverage_threshold) echo "=== 📈 Evaluating coverage (threshold = ${threshold}%) ===" # Use the .coverage file produced by the previous task if [[ ! -f ~/artifacts/.coverage ]]; then echo "❌ No coverage data found!" exit 1 fi cp ~/artifacts/.coverage .coverage # Produce a human‑readable report and also store it as an artifact coverage report -m | tee ~/artifacts/coverage.txt # Extract the total % from the report (format: TOTAL xx.xx%) total=$(coverage report -m | awk '/TOTAL/ {gsub("%","",$4); print $4}') # Guard against missing data if [[ -z "$total" ]]; then echo "❌ Could not determine total coverage" exit 1 fi # Compare using bc (handles floating‑point numbers) if (( $(echo "$total < $threshold" | bc -l) )); then echo "❌ Coverage $total% is below the required $threshold%" exit 1 else echo "✅ Coverage $total% meets the required $threshold%" fi ``` --- ## 6️⃣ How the pieces fit together | Component | What it does | How it talks to the rest | |-----------|--------------|--------------------------| | **`jobs.yaml`** | Lists the single job `test` | DSCI reads it and executes the job at `job_test/` | | **`config.yaml`** | Supplies default parameters (e.g. `coverage_threshold`) | Accessible via `$(config name)` in Bash tasks | | **`job.bash`** | Orchestrates the three tasks using `run_task` | Calls each task in order | | **`install/task.bash`** | Installs `requirements.txt` | No special communication needed | | **`test/task.bash`** | Executes `pytest` under `coverage` and stores raw data in `~/artifacts/` | Shares the `.coverage` file via the artifacts folder | | **`coverage/task.bash`** | Reads the threshold, generates a readable report, checks the total % and fails the job if below threshold | Uses `$(config coverage_threshold)` and the artifact created by the previous task | All **state sharing** is performed through the `~/artifacts/` directory, which DSCI automatically mounts across tasks of the same job. --- ## 7️⃣ Extending / Overriding the threshold If you need a stricter (or looser) threshold for a particular pipeline run, simply add a `params` block to the job entry in `jobs.yaml`: ```yaml jobs: - id: test path: job_test/ params: coverage_threshold: 90 # require 90 % coverage for this run ``` The `$(config coverage_threshold)` call inside `coverage/task.bash` will pick up the overridden value automatically. --- ### 🎉 Done! You now have a **minimal yet fully functional DSCI pipeline** that: 1. Installs the project’s Python dependencies. 2. Runs all `pytest` unit‑tests while collecting coverage data. 3. Enforces a configurable coverage threshold, failing the pipeline if the threshold is not met. All logic lives in clean Bash scripts (the simplest language for these steps), while configuration values are handled via the built‑in `config()` SDK function, exactly as required by DSCI.