DSCI

**File structure** ``` . ├── jobs.yaml # pipeline definition ├── build_job/ # first job – build / test / lint │ ├── config.yaml # default parameters for this job │ ├── job.py # job script (orchestrates the tasks) │ └── tasks/ │ ├── install_deps/ │ │ └── task.bash │ ├── lint/ │ │ └── task.bash │ ├── test/ │ │ └── task.py │ ├── version/ │ │ └── task.py │ └── build/ │ └── task.bash └── deploy_job/ # second job – consumes the artifact ├── config.yaml ├── job.bash └── tasks/ └── show_artifact/ └── task.bash ``` --- ## `jobs.yaml` – pipeline definition Only the **jobs** list (no matrix, no extra YAML tricks) is required. ```yaml # jobs.yaml jobs: - id: build path: build_job/ # optional – override any defaults from build_job/config.yaml # params: # python_version: "3.11" - id: deploy path: deploy_job/ ``` --- ## `build_job/config.yaml` – default parameters for the **build** job ```yaml # build_job/config.yaml python_version: "3.11" poetry_version: "1.7.1" ``` --- ## `build_job/job.py` – orchestrates all tasks for the **build** job ```python # build_job/job.py #!/usr/bin/env python3 # The SDK will inject `run_task` into the namespace. # No explicit imports are needed. # 1️⃣ Install Poetry and all project dependencies run_task('install_deps') # 2️⃣ Lint the source code run_task('lint') # 3️⃣ Run the test‑suite (pytest) and export the test result run_task('test') # 4️⃣ Extract the package version from pyproject.toml and share it via state run_task('version') # 5️⃣ Build a wheel / sdist and place it into the shared artifacts directory run_task('build') ``` --- ### Task **install_deps** – install Poetry and the project’s dependencies ```bash # build_job/tasks/install_deps/task.bash #!/usr/bin/env bash set -euo pipefail # Read parameters (if they were overridden in the pipeline) – not strictly needed here PY_VER=$(config python_version) POETRY_VER=$(config poetry_version) echo "=== Installing Poetry ${POETRY_VER} for Python ${PY_VER} ===" # Install a specific Poetry version (uses the official installer) curl -sSL https://install.python-poetry.org | python3 - --version "${POETRY_VER}" # Make Poetry available in the current shell export PATH="${HOME}/.local/bin:${PATH}" # Verify installation poetry --version echo "=== Installing project dependencies with Poetry ===" poetry install --no-interaction --no-ansi ``` --- ### Task **lint** – run `flake8` on the source tree ```bash # build_job/tasks/lint/task.bash #!/usr/bin/env bash set -euo pipefail # Poetry already created a virtualenv; use it export PATH="${HOME}/.local/bin:${PATH}" poetry run flake8 . echo "✅ Linting passed" ``` --- ### Task **test** – execute the test suite with `pytest` and export the result ```python # build_job/tasks/test/task.py #!/usr/bin/env python3 import subprocess import sys # Run pytest inside Poetry's virtual environment result = subprocess.run( ["poetry", "run", "pytest", "--quiet"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, ) print(result.stdout) # Export test outcome to the job state (so later tasks / jobs can read it) test_passed = result.returncode == 0 update_state({"tests_passed": test_passed}) # Fail the job if any test failed if not test_passed: sys.exit(1) ``` --- ### Task **version** – read the package version from `pyproject.toml` and share it ```python # build_job/tasks/version/task.py #!/usr/bin/env python3 import tomllib from pathlib import Path # Load pyproject.toml (PEP 518) pyproject_path = Path.cwd() / "pyproject.toml" with pyproject_path.open("rb") as f: data = tomllib.load(f) # Poetry stores the version under [tool.poetry] version = data.get("tool", {}).get("poetry", {}).get("version", "0.0.0") print(f"Package version: {version}") # Export the version – other jobs can read it via `config()` update_state({"package_version": version}) ``` --- ### Task **build** – create distribution files and put them into the shared artifacts directory ```bash # build_job/tasks/build/task.bash #!/usr/bin/env bash set -euo pipefail # Build wheel + sdist with Poetry poetry build --quiet # All artifacts end up in ./dist/ ARTIFACT_DIR="${HOME}/artifacts" mkdir -p "${ARTIFACT_DIR}" # Copy everything from dist/ to the shared artifacts directory cp -a dist/* "${ARTIFACT_DIR}/" echo "✅ Build artifacts copied to ${ARTIFACT_DIR}" ``` --- ## `deploy_job/config.yaml` – default parameters for the **deploy** job ```yaml # deploy_job/config.yaml # (no mandatory parameters – kept for symmetry) ``` --- ## `deploy_job/job.bash` – a tiny job that consumes the artifact created by the build job ```bash # deploy_job/job.bash #!/usr/bin/env bash set -euo pipefail # ------------------------------------------------------------------------- # 1️⃣ Retrieve the version that the *build* job stored in its state # ------------------------------------------------------------------------- VERSION=$(config _dsci_.build.package_version) echo "🚀 Deploying version ${VERSION}" # ------------------------------------------------------------------------- # 2️⃣ List the artifact(s) that were placed into ~/artifacts by the build job # ------------------------------------------------------------------------- ARTIFACT_DIR="${HOME}/artifacts" echo "📦 Available artifact files:" ls -1 "${ARTIFACT_DIR}" # ------------------------------------------------------------------------- # 3️⃣ (Placeholder) – here you could upload the wheel to PyPI, a private # repository, or any other deployment target. # ------------------------------------------------------------------------- echo "✅ Deploy step finished (placeholder – replace with real upload)." ``` --- ## How the pipeline works (high‑level flow) 1. **Job *build*** (`build_job/job.py`) runs five tasks in sequence: * `install_deps` – ensures Poetry is present and installs the project’s dependencies. * `lint` – runs `flake8` (fails the job on lint errors). * `test` – runs `pytest`; the Boolean result (`tests_passed`) is stored in the job state. * `version` – extracts the package version from `pyproject.toml` and puts it into the state (`package_version`). * `build` – builds wheels / sdist with Poetry and copies the resulting files into `~/artifacts/`. 2. **Job *deploy*** (`deploy_job/job.bash`) reads the state exported by the previous job (`_dsci_.build.package_version`) and lists the artifact files that live in the shared `~/artifacts/` directory. In a real world scenario you would replace the placeholder echo with an actual upload command (e.g. `twine upload`). 3. **State & artifacts** * `update_state` is used **once per task** (the pitfall‑avoidance rule) to guarantee that earlier state entries are not overwritten. * The `~/artifacts/` directory automatically persists files between jobs, acting as a simple cross‑job data buffer. --- ### Summary of SDK functions used | Location | Language | Function | Purpose | |----------|----------|----------|---------| | `install_deps` (bash) | Bash | `$(config …)` | read job parameters (`python_version`, `poetry_version`) | | `test` (python) | Python | `update_state({...})` | export `tests_passed` | | `version` (python) | Python | `update_state({...})` | export `package_version` | | `deploy_job/job.bash` | Bash | `config …` | read upstream state (`_dsci_.build.package_version`) | | All tasks | — | `~/artifacts/` | shared directory for files between jobs | --- **You now have a complete DSCI pipeline that builds a Python project with a modern dependency manager (Poetry), runs linting & tests, captures version & test status via state, and passes the built wheel through the shared `artifacts` directory to a downstream deployment job.**