DSCI Report: dsci@2283
18:47:58 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: main [task run: task.bash - create directory scm] [task stdout] 18:47:59 :: directory path: scm 18:47:59 :: directory owner: <ubuntu> 18:47:59 :: directory group: <ubuntu> 18:47:59 :: directory access rights: drwxr-xr-x scm: http://localhost:8080/pipeline-generator.git sha: 1f7f688 message: fix get_state by dsci@sparrowhub.io - manual run container runtime: podman raku version Raku++ 4.0.1 (2026-09-17) x86_64-linux sparrow version 0.0.95 runner version: 0.1.1 ======= [task run: task.bash - bash: git config --global advice.detachedHead false] [task stdout] [task run: task.bash - bash: git clone http://localhost:8080/pipeline-generator ...] [task stdout] 18:48:00 :: /home/ubuntu/.dsci/.sparky/.cache/1f7f688.1790275674/.sparrowdo/scm [task stderr] 18:48:00 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1f7f688] [task stdout] 18:48:01 :: /home/ubuntu/.dsci/.sparky/.cache/1f7f688.1790275674/.sparrowdo/scm [task stderr] 18:48:01 :: HEAD is now at 1f7f688 fix get_state pipeline mode: localhost run dsci agent container from image: melezhik/dsci-agent-rakupp ai_agent_message set: 437265617465204453434920706970656c696e6520746f206275696c6420707974686f6e2070726f6a656374207573696e67206d6f6465726e20707974686f6e20646570656e64656e63696573206d616e6167657220 >>> POST http://127.0.0.1:8181/stash >>> POST http://127.0.0.1:8181/queue >>> wait for jobs, timeout: 600 sec >>> GET http://127.0.0.1:8181/stash/job.run/wkdypcqmnbjgersvfhoi.592888 job test ... [OK] ... >>> GET job report, status=200 18:48:08 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: job-run raku version Raku++ 4.0.1 (2026-09-17) x86_64-linux sparrow version 0.0.95 runner version: 0.1.1 run job: test === >>> GET http://127.0.0.1:8181/stash/job.run/wkdypcqmnbjgersvfhoi.592888 [task run: task.bash - create directory scm] [task stdout] 18:48:09 :: directory path: scm 18:48:09 :: directory owner: <ubuntu> 18:48:09 :: directory group: <ubuntu> 18:48:09 :: directory access rights: drwxr-xr-x [task run: task.bash - bash: git clone http://localhost:8080/pipeline-generator ...] [task stdout] 18:48:10 :: /home/ubuntu/.dsci/.sparky/.cache/wkdypcqmnbjgersvfhoi.592888/.sparrowdo/scm [task stderr] 18:48:10 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1f7f688] [task stdout] 18:48:10 :: /home/ubuntu/.dsci/.sparky/.cache/wkdypcqmnbjgersvfhoi.592888/.sparrowdo/scm [task stderr] 18:48:10 :: HEAD is now at 1f7f688 fix get_state [task run: task.bash - bash: prepare dsci scripts] [task stdout] >>> GET http://127.0.0.1:8181/stash/job.run/wkdypcqmnbjgersvfhoi.592888 [task run: task.bash - .] [task stdout] 18:48:11 :: total 8 18:48:11 :: -rw-r--r-- 1 ubuntu ubuntu 109 Sep 24 18:48 jobs.yaml 18:48:11 :: -rw-r--r-- 1 ubuntu ubuntu 323 Sep 24 18:48 task.bash 18:48:11 :: 437265617465204453434920706970656c696e6520746f206275696c6420707974686f6e2070726f6a656374207573696e67206d6f6465726e20707974686f6e20646570656e64656e63696573206d616e6167657220 18:48:18 :: === 18:48:18 :: Create DSCI pipeline to build python project using modern python dependencies manager 18:48:18 :: === 18:48:18 :: **File structure** 18:48:18 :: 18:48:18 :: ``` 18:48:18 :: . 18:48:18 :: ├── jobs.yaml # pipeline definition 18:48:18 :: ├── build_job/ # first job – build / test / lint 18:48:18 :: │ ├── config.yaml # default parameters for this job 18:48:18 :: │ ├── job.py # job script (orchestrates the tasks) 18:48:18 :: │ └── tasks/ 18:48:18 :: │ ├── install_deps/ 18:48:18 :: │ │ └── task.bash 18:48:18 :: │ ├── lint/ 18:48:18 :: │ │ └── task.bash 18:48:18 :: │ ├── test/ 18:48:18 :: │ │ └── task.py 18:48:18 :: │ ├── version/ 18:48:18 :: │ │ └── task.py 18:48:18 :: │ └── build/ 18:48:18 :: │ └── task.bash 18:48:18 :: └── deploy_job/ # second job – consumes the artifact 18:48:18 :: ├── config.yaml 18:48:18 :: ├── job.bash 18:48:18 :: └── tasks/ 18:48:18 :: └── show_artifact/ 18:48:18 :: └── task.bash 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## `jobs.yaml` – pipeline definition 18:48:18 :: 18:48:18 :: Only the **jobs** list (no matrix, no extra YAML tricks) is required. 18:48:18 :: 18:48:18 :: ```yaml 18:48:18 :: # jobs.yaml 18:48:18 :: jobs: 18:48:18 :: - id: build 18:48:18 :: path: build_job/ 18:48:18 :: # optional – override any defaults from build_job/config.yaml 18:48:18 :: # params: 18:48:18 :: # python_version: "3.11" 18:48:18 :: - id: deploy 18:48:18 :: path: deploy_job/ 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## `build_job/config.yaml` – default parameters for the **build** job 18:48:18 :: 18:48:18 :: ```yaml 18:48:18 :: # build_job/config.yaml 18:48:18 :: python_version: "3.11" 18:48:18 :: poetry_version: "1.7.1" 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## `build_job/job.py` – orchestrates all tasks for the **build** job 18:48:18 :: 18:48:18 :: ```python 18:48:18 :: # build_job/job.py 18:48:18 :: #!/usr/bin/env python3 18:48:18 :: 18:48:18 :: # The SDK will inject `run_task` into the namespace. 18:48:18 :: # No explicit imports are needed. 18:48:18 :: 18:48:18 :: # 1️⃣ Install Poetry and all project dependencies 18:48:18 :: run_task('install_deps') 18:48:18 :: 18:48:18 :: # 2️⃣ Lint the source code 18:48:18 :: run_task('lint') 18:48:18 :: 18:48:18 :: # 3️⃣ Run the test‑suite (pytest) and export the test result 18:48:18 :: run_task('test') 18:48:18 :: 18:48:18 :: # 4️⃣ Extract the package version from pyproject.toml and share it via state 18:48:18 :: run_task('version') 18:48:18 :: 18:48:18 :: # 5️⃣ Build a wheel / sdist and place it into the shared artifacts directory 18:48:18 :: run_task('build') 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Task **install_deps** – install Poetry and the project’s dependencies 18:48:18 :: 18:48:18 :: ```bash 18:48:18 :: # build_job/tasks/install_deps/task.bash 18:48:18 :: #!/usr/bin/env bash 18:48:18 :: set -euo pipefail 18:48:18 :: 18:48:18 :: # Read parameters (if they were overridden in the pipeline) – not strictly needed here 18:48:18 :: PY_VER=$(config python_version) 18:48:18 :: POETRY_VER=$(config poetry_version) 18:48:18 :: 18:48:18 :: echo "=== Installing Poetry ${POETRY_VER} for Python ${PY_VER} ===" 18:48:18 :: 18:48:18 :: # Install a specific Poetry version (uses the official installer) 18:48:18 :: curl -sSL https://install.python-poetry.org | python3 - --version "${POETRY_VER}" 18:48:18 :: 18:48:18 :: # Make Poetry available in the current shell 18:48:18 :: export PATH="${HOME}/.local/bin:${PATH}" 18:48:18 :: 18:48:18 :: # Verify installation 18:48:18 :: poetry --version 18:48:18 :: 18:48:18 :: echo "=== Installing project dependencies with Poetry ===" 18:48:18 :: poetry install --no-interaction --no-ansi 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Task **lint** – run `flake8` on the source tree 18:48:18 :: 18:48:18 :: ```bash 18:48:18 :: # build_job/tasks/lint/task.bash 18:48:18 :: #!/usr/bin/env bash 18:48:18 :: set -euo pipefail 18:48:18 :: 18:48:18 :: # Poetry already created a virtualenv; use it 18:48:18 :: export PATH="${HOME}/.local/bin:${PATH}" 18:48:18 :: poetry run flake8 . 18:48:18 :: echo "✅ Linting passed" 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Task **test** – execute the test suite with `pytest` and export the result 18:48:18 :: 18:48:18 :: ```python 18:48:18 :: # build_job/tasks/test/task.py 18:48:18 :: #!/usr/bin/env python3 18:48:18 :: import subprocess 18:48:18 :: import sys 18:48:18 :: 18:48:18 :: # Run pytest inside Poetry's virtual environment 18:48:18 :: result = subprocess.run( 18:48:18 :: ["poetry", "run", "pytest", "--quiet"], 18:48:18 :: stdout=subprocess.PIPE, 18:48:18 :: stderr=subprocess.STDOUT, 18:48:18 :: text=True, 18:48:18 :: ) 18:48:18 :: 18:48:18 :: print(result.stdout) 18:48:18 :: 18:48:18 :: # Export test outcome to the job state (so later tasks / jobs can read it) 18:48:18 :: test_passed = result.returncode == 0 18:48:18 :: update_state({"tests_passed": test_passed}) 18:48:18 :: 18:48:18 :: # Fail the job if any test failed 18:48:18 :: if not test_passed: 18:48:18 :: sys.exit(1) 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Task **version** – read the package version from `pyproject.toml` and share it 18:48:18 :: 18:48:18 :: ```python 18:48:18 :: # build_job/tasks/version/task.py 18:48:18 :: #!/usr/bin/env python3 18:48:18 :: import tomllib 18:48:18 :: from pathlib import Path 18:48:18 :: 18:48:18 :: # Load pyproject.toml (PEP 518) 18:48:18 :: pyproject_path = Path.cwd() / "pyproject.toml" 18:48:18 :: with pyproject_path.open("rb") as f: 18:48:18 :: data = tomllib.load(f) 18:48:18 :: 18:48:18 :: # Poetry stores the version under [tool.poetry] 18:48:18 :: version = data.get("tool", {}).get("poetry", {}).get("version", "0.0.0") 18:48:18 :: print(f"Package version: {version}") 18:48:18 :: 18:48:18 :: # Export the version – other jobs can read it via `config()` 18:48:18 :: update_state({"package_version": version}) 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Task **build** – create distribution files and put them into the shared artifacts directory 18:48:18 :: 18:48:18 :: ```bash 18:48:18 :: # build_job/tasks/build/task.bash 18:48:18 :: #!/usr/bin/env bash 18:48:18 :: set -euo pipefail 18:48:18 :: 18:48:18 :: # Build wheel + sdist with Poetry 18:48:18 :: poetry build --quiet 18:48:18 :: 18:48:18 :: # All artifacts end up in ./dist/ 18:48:18 :: ARTIFACT_DIR="${HOME}/artifacts" 18:48:18 :: mkdir -p "${ARTIFACT_DIR}" 18:48:18 :: 18:48:18 :: # Copy everything from dist/ to the shared artifacts directory 18:48:18 :: cp -a dist/* "${ARTIFACT_DIR}/" 18:48:18 :: echo "✅ Build artifacts copied to ${ARTIFACT_DIR}" 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## `deploy_job/config.yaml` – default parameters for the **deploy** job 18:48:18 :: 18:48:18 :: ```yaml 18:48:18 :: # deploy_job/config.yaml 18:48:18 :: # (no mandatory parameters – kept for symmetry) 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## `deploy_job/job.bash` – a tiny job that consumes the artifact created by the build job 18:48:18 :: 18:48:18 :: ```bash 18:48:18 :: # deploy_job/job.bash 18:48:18 :: #!/usr/bin/env bash 18:48:18 :: set -euo pipefail 18:48:18 :: 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: # 1️⃣ Retrieve the version that the *build* job stored in its state 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: VERSION=$(config _dsci_.build.package_version) 18:48:18 :: echo "🚀 Deploying version ${VERSION}" 18:48:18 :: 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: # 2️⃣ List the artifact(s) that were placed into ~/artifacts by the build job 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: ARTIFACT_DIR="${HOME}/artifacts" 18:48:18 :: echo "📦 Available artifact files:" 18:48:18 :: ls -1 "${ARTIFACT_DIR}" 18:48:18 :: 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: # 3️⃣ (Placeholder) – here you could upload the wheel to PyPI, a private 18:48:18 :: # repository, or any other deployment target. 18:48:18 :: # ------------------------------------------------------------------------- 18:48:18 :: echo "✅ Deploy step finished (placeholder – replace with real upload)." 18:48:18 :: ``` 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ## How the pipeline works (high‑level flow) 18:48:18 :: 18:48:18 :: 1. **Job *build*** (`build_job/job.py`) runs five tasks in sequence: 18:48:18 :: * `install_deps` – ensures Poetry is present and installs the project’s 18:48:18 :: dependencies. 18:48:18 :: * `lint` – runs `flake8` (fails the job on lint errors). 18:48:18 :: * `test` – runs `pytest`; the Boolean result (`tests_passed`) is stored in 18:48:18 :: the job state. 18:48:18 :: * `version` – extracts the package version from `pyproject.toml` and puts it 18:48:18 :: into the state (`package_version`). 18:48:18 :: * `build` – builds wheels / sdist with Poetry and copies the resulting files 18:48:18 :: into `~/artifacts/`. 18:48:18 :: 18:48:18 :: 2. **Job *deploy*** (`deploy_job/job.bash`) reads the state exported by the 18:48:18 :: previous job (`_dsci_.build.package_version`) and lists the artifact files 18:48:18 :: that live in the shared `~/artifacts/` directory. In a real world scenario 18:48:18 :: you would replace the placeholder echo with an actual upload command (e.g. 18:48:18 :: `twine upload`). 18:48:18 :: 18:48:18 :: 3. **State & artifacts** 18:48:18 :: * `update_state` is used **once per task** (the pitfall‑avoidance rule) to 18:48:18 :: guarantee that earlier state entries are not overwritten. 18:48:18 :: * The `~/artifacts/` directory automatically persists files between jobs, 18:48:18 :: acting as a simple cross‑job data buffer. 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: ### Summary of SDK functions used 18:48:18 :: 18:48:18 :: | Location | Language | Function | Purpose | 18:48:18 :: |----------|----------|----------|---------| 18:48:18 :: | `install_deps` (bash) | Bash | `$(config …)` | read job parameters (`python_version`, `poetry_version`) | 18:48:18 :: | `test` (python) | Python | `update_state({...})` | export `tests_passed` | 18:48:18 :: | `version` (python) | Python | `update_state({...})` | export `package_version` | 18:48:18 :: | `deploy_job/job.bash` | Bash | `config …` | read upstream state (`_dsci_.build.package_version`) | 18:48:18 :: | All tasks | — | `~/artifacts/` | shared directory for files between jobs | 18:48:18 :: 18:48:18 :: --- 18:48:18 :: 18:48:18 :: **You now have a complete DSCI pipeline that builds a Python project with a 18:48:18 :: modern dependency manager (Poetry), runs linting & tests, captures version & 18:48:18 :: test status via state, and passes the built wheel through the shared 18:48:18 :: `artifacts` directory to a downstream deployment job.** [task stderr] 18:48:18 :: 18:48:18 :: real 0m7.181s 18:48:18 :: user 0m0.621s 18:48:18 :: sys 0m0.104s saving job artifacts ... >> PUT http://127.0.0.1:8181/file/project/dsci/job/1f7f688.1790275674/filename/02_answer.md >> PUT http://127.0.0.1:8181/file/project/dsci/job/1f7f688.1790275674/filename/01_question.txt >>> POST http://127.0.0.1:8181/stash --- ---