DSCI

**File‑tree** ``` .dsci/ │ ├─ jobs.yaml # pipeline definition │ ├─ job_test/ │ ├─ job.bash # test job entry point │ └─ tasks/ │ ├─ go_test/ │ │ └─ task.bash # run `go test` → coverage file │ └─ check_coverage/ │ └─ task.py # verify coverage ≥ minimal │ ├─ job_build/ │ ├─ job.bash # build job entry point │ └─ tasks/ │ └─ docker_build/ │ └─ task.bash # `docker build` → image tag │ └─ job_push/ ├─ job.bash # push job entry point └─ tasks/ └─ docker_push/ └─ task.bash # `docker push` ``` --- ## 1️⃣ `jobs.yaml` ```yaml # .dsci/jobs.yaml jobs: - id: test path: job_test/ params: minimal: 80 # minimal coverage % (pipeline parameter) version: "1.0.0" # image tag version (pipeline parameter) - id: build path: job_build/ params: version: "1.0.0" - id: push path: job_push/ params: version: "1.0.0" ``` *`minimal`* and *`version`* are injected into the jobs through the `params:` block. Inside jobs/tasks they are accessed with `config()` (Python) or `$(config name)` (Bash). --- ## 2️⃣ Test job – *job_test/job.bash* ```bash #!/bin/bash # .dsci/job_test/job.bash # run unit‑tests → coverage file run_task go_test # verify coverage against the minimal threshold run_task check_coverage ``` ### 2.1 `go_test` task – *job_test/tasks/go_test/task.bash* ```bash #!/bin/bash # .dsci/job_test/tasks/go_test/task.bash set -euo pipefail echo "=== Running Go tests with coverage ===" go test ./... -coverprofile=coverage.out # expose the file to the next job via the artifacts directory mkdir -p ~/artifacts cp coverage.out ~/artifacts/ ``` ### 2.2 `check_coverage` task – *job_test/tasks/check_coverage/task.py* ```python #!/usr/bin/python3 # .dsci/job_test/tasks/check_coverage/task.py import os, re, sys, subprocess from pathlib import Path # ----- read pipeline parameters ----- cfg = config() # DSCI SDK call minimal = float(cfg.get("minimal", 80)) # default 80 % if not supplied # ----- read the coverage profile ----- cov_path = Path.home() / "artifacts" / "coverage.out" if not cov_path.is_file(): print(f"❌ Coverage file not found at {cov_path}") sys.exit(1) # ----- compute total coverage ----- proc = subprocess.run( ["go", "tool", "cover", "-func", str(cov_path)], capture_output=True, text=True, check=True, ) total_line = next((l for l in proc.stdout.splitlines() if l.startswith("total:")), "") match = re.search(r"(\d+\.\d+)%", total_line) if not match: print("❌ Could not parse total coverage") sys.exit(1) coverage = float(match.group(1)) print(f"📊 Total coverage: {coverage}% (minimum required: {minimal}%)") # ----- fail if below threshold ----- if coverage < minimal: print("🚨 Coverage is below the required minimum → failing pipeline") sys.exit(1) # (optional) export coverage for downstream jobs update_state({"coverage": coverage}) ``` --- ## 3️⃣ Build job – *job_build/job.bash* ```bash #!/bin/bash # .dsci/job_build/job.bash run_task docker_build ``` ### 3.1 `docker_build` task – *job_build/tasks/docker_build/task.bash* ```bash #!/bin/bash # .dsci/job_build/tasks/docker_build/task.bash set -euo pipefail # image tag comes from the pipeline parameter `version` VERSION=$(config version) IMAGE="myregistry.example.com/my-golang-app:${VERSION}" echo "=== Building Docker image ${IMAGE} ===" docker build -t "${IMAGE}" . # make the image name available to the next job mkdir -p ~/artifacts echo "${IMAGE}" > ~/artifacts/image_name.txt # also expose it via DSCI state (useful for Python downstream) update_state "image_name" "${IMAGE}" ``` --- ## 4️⃣ Push job – *job_push/job.bash* ```bash #!/bin/bash # .dsci/job_push/job.bash run_task docker_push ``` ### 4.1 `docker_push` task – *job_push/tasks/docker_push/task.bash* ```bash #!/bin/bash # .dsci/job_push/tasks/docker_push/task.bash set -euo pipefail # Prefer the artifact written by the build job; fall back to state if missing if [ -f ~/artifacts/image_name.txt ]; then IMAGE_NAME=$(cat ~/artifacts/image_name.txt) else # state is a dict; we fetch the value that build job stored IMAGE_NAME=$(config image_name) fi echo "=== Pushing Docker image ${IMAGE_NAME} ===" docker push "${IMAGE_NAME}" ``` --- ### How the pieces work together | Job | Main steps | Artifacts / State passed forward | |-----|------------|----------------------------------| | **test** | `go test` → `coverage.out` → Python checks coverage | `~/artifacts/coverage.out` (and optional `coverage` state) | | **build** | Docker image built & tagged with `$(config version)` | `~/artifacts/image_name.txt` + state key `image_name` | | **push** | Reads image name & pushes to registry | – | The pipeline will **fail** automatically if the coverage verification task exits with a non‑zero status (i.e. coverage < `minimal`). All other logic is pure Bash/Python, keeping the `jobs.yaml` tiny and declarative, exactly as DSCI expects.