DSCI Report: dsci@2171
17:42:14 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: main [task run: task.bash - create directory scm] [task stdout] 17:42:14 :: directory path: scm 17:42:14 :: directory owner: <ubuntu> 17:42:14 :: directory group: <ubuntu> 17:42:14 :: directory access rights: drwxr-xr-x scm: http://localhost:8080/test1.git sha: 1c9fb55 message: feat: add or update file via dsci web by dsci@sparrowhub.io - manual run container runtime: podman raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 ======= [task run: task.bash - bash: git config --global advice.detachedHead false] [task stdout] [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 17:42:15 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789580532/.sparrowdo/scm [task stderr] 17:42:15 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 17:42:15 :: /home/ubuntu/.dsci/.sparky/.cache/1c9fb55.1789580532/.sparrowdo/scm [task stderr] 17:42:15 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web pipeline mode: localhost [task run: task.bash - bash: podman stop -t 1 dsci-agent || :] [task stdout] 17:42:17 :: dsci-agent [task stderr] 17:42:17 :: time="2026-09-16T17:42:17Z" level=warning msg="StopSignal SIGTERM failed to stop container dsci-agent in 1 seconds, resorting to SIGKILL" [task run: task.bash - bash: podman rm dsci-agent || :] [task stdout] [task stderr] 17:42:18 :: Error: no container with ID or name "dsci-agent" found: no such container run dsci agent container from image: melezhik/dsci-agent-rakupp podman run --network host -dit --rm --name dsci-agent --entrypoint /bin/sh melezhik/dsci-agent-rakupp [task run: task.bash - bash: podman run --network host -dit --rm --name dsci-a ...] [task stdout] 17:42:18 :: 2a4a650ad54964e7ebf989376c339c23aa85f51da4077aca9cf49d58ff270dae [task stderr] 17:42:18 :: time="2026-09-16T17:42:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" ai_agent_message set: 437265617465206473636920706970656c696e6520666f7220507974686f6e2070726f6a656374207468617420686176652073656c656e69756d20746573747320 >>> POST http://127.0.0.1:8181/stash >>> POST http://127.0.0.1:8181/queue >>> wait for jobs, timeout: 600 sec >>> GET http://127.0.0.1:8181/stash/job.run/ztxakdepylfqogmunrji.3921624 job test ... [OK] ... >>> GET job report, status=200 17:42:24 :: [repository] - index updated from http://sparrowhub.io/repo/api/v1/index run stage: job-run raku version Raku++ (rakupp) 3.25.0 — a Raku interpreter and compiler in C++ Implements Raku 6.d, and 6.e under `use v6.e.PREVIEW`. Build v3.25.0 (2026-09-03), x86_64-linux, clang 18.1.3 Home https://raku.online — docs, a tour of the language, and a browser playground sparrow version 0.0.94 runner version: 0.1.1 run job: test === >>> GET http://127.0.0.1:8181/stash/job.run/ztxakdepylfqogmunrji.3921624 [task run: task.bash - create directory scm] [task stdout] 17:42:25 :: directory path: scm 17:42:25 :: directory owner: <ubuntu> 17:42:25 :: directory group: <ubuntu> 17:42:25 :: directory access rights: drwxr-xr-x [task run: task.bash - bash: git clone http://localhost:8080/test1.git .] [task stdout] 17:42:25 :: /home/ubuntu/.dsci/.sparky/.cache/ztxakdepylfqogmunrji.3921624/.sparrowdo/scm [task stderr] 17:42:25 :: Cloning into '.'... [task run: task.bash - bash: git checkout 1c9fb55] [task stdout] 17:42:25 :: /home/ubuntu/.dsci/.sparky/.cache/ztxakdepylfqogmunrji.3921624/.sparrowdo/scm [task stderr] 17:42:25 :: HEAD is now at 1c9fb55 feat: add or update file via dsci web [task run: task.bash - bash: prepare dsci scripts] [task stdout] >>> GET http://127.0.0.1:8181/stash/job.run/ztxakdepylfqogmunrji.3921624 [task run: task.bash - .] [task stdout] 17:42:26 :: total 8 17:42:26 :: -rw-r--r-- 1 ubuntu ubuntu 109 Sep 16 17:42 jobs.yaml 17:42:26 :: -rw-r--r-- 1 ubuntu ubuntu 301 Sep 16 17:42 task.bash 17:42:26 :: 437265617465206473636920706970656c696e6520666f7220507974686f6e2070726f6a656374207468617420686176652073656c656e69756d20746573747320 17:42:33 :: === 17:42:33 :: Create dsci pipeline for Python project that have selenium tests 17:42:33 :: === 17:42:33 :: Below is a complete, ready‑to‑use DSCI pipeline that 17:42:33 :: 17:42:33 :: * installs the Python requirements of the project, 17:42:33 :: * runs Selenium tests written in **Python**, 17:42:33 :: * stores the test report and any screenshots as artifacts, and 17:42:33 :: * publishes a small state (`tests_passed`) that a downstream job could read. 17:42:33 :: 17:42:33 :: The pipeline is expressed only with a tiny `jobs.yaml`. 17:42:33 :: All real logic lives in normal Bash or Python files, exactly as DSCI expects. 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 1️⃣ Repository layout 17:42:33 :: 17:42:33 :: ``` 17:42:33 :: . 17:42:33 :: ├── requirements.txt # your normal Python deps (incl. selenium) 17:42:33 :: ├── .dsci/ 17:42:33 :: │ ├── jobs.yaml # pipeline definition 17:42:33 :: │ ├── install_job/ 17:42:33 :: │ │ ├── job.bash # entry‑point for the “install” job 17:42:33 :: │ │ ├── config.yaml # default job parameters (can be overridden) 17:42:33 :: │ │ └── tasks/ 17:42:33 :: │ │ └── deps/ 17:42:33 :: │ │ └── task.bash # install Python deps 17:42:33 :: │ └── test_job/ 17:42:33 :: │ ├── job.py # entry‑point for the “test” job 17:42:33 :: │ ├── config.yaml # default parameters for the test run 17:42:33 :: │ └── tasks/ 17:42:33 :: │ ├── env_setup/ 17:42:33 :: │ │ └── task.bash # optional: create virtual‑env, clean artifacts 17:42:33 :: │ ├── run_selenium/ 17:42:33 :: │ │ └── task.py # **your Selenium test runner** 17:42:33 :: │ └── collect/ 17:42:33 :: │ └── task.bash # move reports/screenshots to ~/artifacts 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: > **Note** – All files below are shown with the exact **filename** you should create. 17:42:33 :: > The `config.yaml` files are optional; they simply provide default values that can be 17:42:33 :: > overridden from `jobs.yaml` via the `params:` block. 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 2️⃣ Pipeline definition – `jobs.yaml` 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: # .dsci/jobs.yaml 17:42:33 :: # ------------------------------------------------- 17:42:33 :: # List of jobs that DSCI will execute **sequentially** 17:42:33 :: # ------------------------------------------------- 17:42:33 :: jobs: 17:42:33 :: - id: install 17:42:33 :: path: install_job/ 17:42:33 :: # you can override defaults here, e.g.: 17:42:33 :: # params: 17:42:33 :: # requirements_file: custom-requirements.txt 17:42:33 :: 17:42:33 :: - id: test 17:42:33 :: path: test_job/ 17:42:33 :: params: 17:42:33 :: # Browser can be overridden when triggering the pipeline 17:42:33 :: selenium_browser: chrome 17:42:33 :: base_url: "https://example.com" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: *Only* the job IDs, their source paths and optional parameters appear here – no script logic. 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 3️⃣ **Install** job (pure Bash) 17:42:33 :: 17:42:33 :: ### 3.1 `install_job/config.yaml` (default parameters) 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: # .dsci/install_job/config.yaml 17:42:33 :: requirements_file: requirements.txt 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ### 3.2 `install_job/job.bash` – entry point 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: # .dsci/install_job/job.bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: # Run the single task that installs the Python deps 17:42:33 :: run_task "deps" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ### 3.3 `install_job/tasks/deps/task.bash` – actually install the deps 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: # .dsci/install_job/tasks/deps/task.bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: # Read the (possibly overridden) parameter 17:42:33 :: REQ_FILE=$(config requirements_file) 17:42:33 :: 17:42:33 :: echo "📦 Installing Python requirements from ${REQ_FILE}" 17:42:33 :: python3 -m pip install --upgrade pip 17:42:33 :: python3 -m pip install -r "${REQ_FILE}" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 4️⃣ **Test** job (Python + Bash helpers) 17:42:33 :: 17:42:33 :: ### 4.1 `test_job/config.yaml` (default test parameters) 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: # .dsci/test_job/config.yaml 17:42:33 :: selenium_browser: chrome # can be chrome, firefox, edge … 17:42:33 :: base_url: "http://localhost:8000" 17:42:33 :: tests_path: "tests/" # folder that contains your pytest‑style Selenium tests 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ### 4.2 `test_job/job.py` – orchestrates the tasks 17:42:33 :: 17:42:33 :: ```python 17:42:33 :: # .dsci/test_job/job.py 17:42:33 :: #!/usr/bin/env python3 17:42:33 :: # ------------------------------------------------- 17:42:33 :: # Job entry point – runs the three tasks in order 17:42:33 :: # ------------------------------------------------- 17:42:33 :: run_task("env_setup") # optional clean‑up / virtual‑env handling 17:42:33 :: run_task("run_selenium") # actual Selenium test execution 17:42:33 :: run_task("collect") # move reports / screenshots to ~/artifacts 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ### 4.3 `test_job/tasks/env_setup/task.bash` 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: # .dsci/test_job/tasks/env_setup/task.bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: # (Optional) make sure the artifacts directory is clean before a new run 17:42:33 :: echo "🧹 Cleaning previous artifacts" 17:42:33 :: rm -rf ~/artifacts/* 17:42:33 :: mkdir -p ~/artifacts 17:42:33 :: 17:42:33 :: # You could also create a virtual‑env here if you prefer isolation 17:42:33 :: # python3 -m venv .venv 17:42:33 :: # source .venv/bin/activate 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ### 4.4 `test_job/tasks/run_selenium/task.py` 17:42:33 :: 17:42:33 :: ```python 17:42:33 :: # .dsci/test_job/tasks/run_selenium/task.py 17:42:33 :: #!/usr/bin/env python3 17:42:33 :: import sys 17:42:33 :: import subprocess 17:42:33 :: from pathlib import Path 17:42:33 :: 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: # Helper to read job parameters (defaults or pipeline overrides) 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: cfg = config() 17:42:33 :: browser = cfg.get("selenium_browser", "chrome") 17:42:33 :: base_url = cfg.get("base_url", "http://localhost") 17:42:33 :: tests_path = cfg.get("tests_path", "tests/") 17:42:33 :: 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: # Run the Selenium test suite (we assume you are using pytest + selenium) 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: # The command below can be adapted to your own test runner. 17:42:33 :: # `pytest` will generate a JUnit XML report and, if you use 17:42:33 :: # `pytest-selenium` or custom code, screenshots will be stored under 17:42:33 :: # the `screenshots/` folder. 17:42:33 :: cmd = [ 17:42:33 :: sys.executable, "-m", "pytest", 17:42:33 :: tests_path, 17:42:33 :: f"--base-url={base_url}", 17:42:33 :: f"-n", "auto", # run tests in parallel if you have pytest-xdist 17:42:33 :: "--junitxml=report.xml", # JUnit report that we will archive later 17:42:33 :: ] 17:42:33 :: 17:42:33 :: print(f"🚀 Running Selenium tests on {browser} against {base_url}") 17:42:33 :: # Export the chosen browser to the environment – many Selenium helper libs read it 17:42:33 :: import os 17:42:33 :: os.environ["SELENIUM_BROWSER"] = browser 17:42:33 :: 17:42:33 :: # Execute the test command 17:42:33 :: result = subprocess.run(cmd, capture_output=True, text=True) 17:42:33 :: 17:42:33 :: # Show stdout / stderr for visibility in CI logs 17:42:33 :: print(result.stdout) 17:42:33 :: print(result.stderr, file=sys.stderr) 17:42:33 :: 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: # Publish a tiny state so downstream jobs can know if the suite passed 17:42:33 :: # ---------------------------------------------------------------------- 17:42:33 :: tests_passed = result.returncode == 0 17:42:33 :: update_state({"tests_passed": tests_passed}) 17:42:33 :: 17:42:33 :: # Exit with the same code so the job fails if any test failed 17:42:33 :: sys.exit(result.returncode) 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: > **Why `update_state`?** 17:42:33 :: > The next job (e.g. a “publish” job) could read `config()['_dsci_']['test']['tests_passed']` 17:42:33 :: > to decide whether to push results, send a Slack notification, etc. 17:42:33 :: 17:42:33 :: ### 4.5 `test_job/tasks/collect/task.bash` 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: # .dsci/test_job/tasks/collect/task.bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: # The Selenium task produced: 17:42:33 :: # - report.xml (JUnit report) 17:42:33 :: # - screenshots/ (optional folder with PNGs) 17:42:33 :: 17:42:33 :: ARTIFACTS_DIR=~/artifacts 17:42:33 :: 17:42:33 :: echo "📦 Collecting test artifacts" 17:42:33 :: 17:42:33 :: # Move the JUnit report 17:42:33 :: if [[ -f report.xml ]]; then 17:42:33 :: mv report.xml "${ARTIFACTS_DIR}/" 17:42:33 :: echo "✅ report.xml stored" 17:42:33 :: fi 17:42:33 :: 17:42:33 :: # Move any screenshots if they exist 17:42:33 :: if [[ -d screenshots ]]; then 17:42:33 :: mv screenshots "${ARTIFACTS_DIR}/" 17:42:33 :: echo "✅ screenshots stored" 17:42:33 :: fi 17:42:33 :: 17:42:33 :: # You could also zip them if you like 17:42:33 :: # zip -r "${ARTIFACTS_DIR}/selenium_artifacts.zip" report.xml screenshots/ 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 5️⃣ (Optional) Example of a downstream job that **reads the state** 17:42:33 :: 17:42:33 :: If you later add a job called `publish` you could access the state produced by the 17:42:33 :: `test` job like this: 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: # .dsci/jobs.yaml (append at the end) 17:42:33 :: - id: publish 17:42:33 :: path: publish_job/ 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: ```python 17:42:33 :: # .dsci/publish_job/job.py 17:42:33 :: #!/usr/bin/env python3 17:42:33 :: cfg = config() 17:42:33 :: tests_passed = cfg["_dsci_"]["test"]["tests_passed"] 17:42:33 :: if tests_passed: 17:42:33 :: print("✅ All Selenium tests passed – publishing results …") 17:42:33 :: # … e.g. upload artifacts, notify Slack, etc. 17:42:33 :: else: 17:42:33 :: print("❌ Tests failed – aborting publish step.") 17:42:33 :: sys.exit(1) 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 6️⃣ How the pieces work together 17:42:33 :: 17:42:33 :: | Step | What happens | Where | 17:42:33 :: |------|--------------|-------| 17:42:33 :: | **1️⃣ Install job** | `install/job.bash` → `install/tasks/deps/task.bash` runs `pip install -r …` | Bash | 17:42:33 :: | **2️⃣ Test job – env** | `test/job.py` → `env_setup/task.bash` cleans `~/artifacts` (optional) | Bash | 17:42:33 :: | **3️⃣ Test job – run** | `run_selenium/task.py` reads parameters via `config()`, launches Selenium tests, writes `report.xml` + screenshots, calls `update_state({"tests_passed": …})` | Python | 17:42:33 :: | **4️⃣ Test job – collect** | `collect/task.bash` moves the generated files into `~/artifacts` so any later job can read them | Bash | 17:42:33 :: | **5️⃣ (Optional) Publish job** | Reads the state (`config()['_dsci_']['test']['tests_passed']`) and decides what to do | Python | 17:42:33 :: 17:42:33 :: All **job‑level parameters** (e.g. `selenium_browser`, `base_url`) are declared in 17:42:33 :: `test_job/config.yaml` and can be overridden directly in `jobs.yaml` under the 17:42:33 :: `params:` block. Inside any task you retrieve them with `config()` (Python) or 17:42:33 :: `$(config name)` (Bash). 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ## 7️⃣ TL;DR – Copy‑paste ready files 17:42:33 :: 17:42:33 :: Below are the exact files you need to create. 17:42:33 :: Copy each block into a file with the *same* path and name shown in the header. 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `.dsci/jobs.yaml` 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: jobs: 17:42:33 :: - id: install 17:42:33 :: path: install_job/ 17:42:33 :: 17:42:33 :: - id: test 17:42:33 :: path: test_job/ 17:42:33 :: params: 17:42:33 :: selenium_browser: chrome 17:42:33 :: base_url: "https://example.com" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `requirements.txt` *(example – add your own deps)* 17:42:33 :: 17:42:33 :: ```text 17:42:33 :: selenium 17:42:33 :: pytest 17:42:33 :: pytest-xdist 17:42:33 :: pytest-selenium 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `install_job/config.yaml` 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: requirements_file: requirements.txt 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `install_job/job.bash` 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: run_task "deps" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `install_job/tasks/deps/task.bash` 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: REQ_FILE=$(config requirements_file) 17:42:33 :: 17:42:33 :: echo "📦 Installing Python requirements from ${REQ_FILE}" 17:42:33 :: python3 -m pip install --upgrade pip 17:42:33 :: python3 -m pip install -r "${REQ_FILE}" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `test_job/config.yaml` 17:42:33 :: 17:42:33 :: ```yaml 17:42:33 :: selenium_browser: chrome 17:42:33 :: base_url: "http://localhost:8000" 17:42:33 :: tests_path: "tests/" 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `test_job/job.py` 17:42:33 :: 17:42:33 :: ```python 17:42:33 :: #!/usr/bin/env python3 17:42:33 :: run_task("env_setup") 17:42:33 :: run_task("run_selenium") 17:42:33 :: run_task("collect") 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `test_job/tasks/env_setup/task.bash` 17:42:33 :: 17:42:33 :: ```bash 17:42:33 :: #!/usr/bin/env bash 17:42:33 :: set -euo pipefail 17:42:33 :: 17:42:33 :: echo "🧹 Cleaning previous artifacts" 17:42:33 :: rm -rf ~/artifacts/* 17:42:33 :: mkdir -p ~/artifacts 17:42:33 :: ``` 17:42:33 :: 17:42:33 :: --- 17:42:33 :: 17:42:33 :: ### `test_job/tasks/run_selenium/task.py` 17:42:33 :: 17:42:33 :: ```python 17:42:33 :: #!/usr/bin/env python3 17:42:33 :: import sys 17:42:33 :: import subprocess 17:42:33 :: from pathlib import Path 17:42:33 :: 17:42:33 :: cfg = config() 17:42:33 :: browser = cfg.get("selenium_browser", "chrome") 17:42:33 :: base_url = cfg.get("base_url", "http://localhost") 17:42:33 :: tests_path = cfg.get("tests_path", "tests/") 17:42:33 :: 17:42:33 :: # expose the chosen browser to any Selenium helper that reads env [task stderr] 17:42:33 :: 17:42:33 :: real 0m7.825s 17:42:33 :: user 0m0.677s 17:42:33 :: sys 0m0.113s saving job artifacts ... >> PUT http://127.0.0.1:8181/file/project/dsci/job/1c9fb55.1789580532/filename/answer.md >>> POST http://127.0.0.1:8181/stash --- ---