DSCI

Below is a complete, ready‑to‑use DSCI pipeline that * installs the Python requirements of the project, * runs Selenium tests written in **Python**, * stores the test report and any screenshots as artifacts, and * publishes a small state (`tests_passed`) that a downstream job could read. The pipeline is expressed only with a tiny `jobs.yaml`. All real logic lives in normal Bash or Python files, exactly as DSCI expects. --- ## 1️⃣ Repository layout ``` . ├── requirements.txt # your normal Python deps (incl. selenium) ├── .dsci/ │ ├── jobs.yaml # pipeline definition │ ├── install_job/ │ │ ├── job.bash # entry‑point for the “install” job │ │ ├── config.yaml # default job parameters (can be overridden) │ │ └── tasks/ │ │ └── deps/ │ │ └── task.bash # install Python deps │ └── test_job/ │ ├── job.py # entry‑point for the “test” job │ ├── config.yaml # default parameters for the test run │ └── tasks/ │ ├── env_setup/ │ │ └── task.bash # optional: create virtual‑env, clean artifacts │ ├── run_selenium/ │ │ └── task.py # **your Selenium test runner** │ └── collect/ │ └── task.bash # move reports/screenshots to ~/artifacts ``` > **Note** – All files below are shown with the exact **filename** you should create. > The `config.yaml` files are optional; they simply provide default values that can be > overridden from `jobs.yaml` via the `params:` block. --- ## 2️⃣ Pipeline definition – `jobs.yaml` ```yaml # .dsci/jobs.yaml # ------------------------------------------------- # List of jobs that DSCI will execute **sequentially** # ------------------------------------------------- jobs: - id: install path: install_job/ # you can override defaults here, e.g.: # params: # requirements_file: custom-requirements.txt - id: test path: test_job/ params: # Browser can be overridden when triggering the pipeline selenium_browser: chrome base_url: "https://example.com" ``` *Only* the job IDs, their source paths and optional parameters appear here – no script logic. --- ## 3️⃣ **Install** job (pure Bash) ### 3.1 `install_job/config.yaml` (default parameters) ```yaml # .dsci/install_job/config.yaml requirements_file: requirements.txt ``` ### 3.2 `install_job/job.bash` – entry point ```bash # .dsci/install_job/job.bash #!/usr/bin/env bash set -euo pipefail # Run the single task that installs the Python deps run_task "deps" ``` ### 3.3 `install_job/tasks/deps/task.bash` – actually install the deps ```bash # .dsci/install_job/tasks/deps/task.bash #!/usr/bin/env bash set -euo pipefail # Read the (possibly overridden) parameter REQ_FILE=$(config requirements_file) echo "📦 Installing Python requirements from ${REQ_FILE}" python3 -m pip install --upgrade pip python3 -m pip install -r "${REQ_FILE}" ``` --- ## 4️⃣ **Test** job (Python + Bash helpers) ### 4.1 `test_job/config.yaml` (default test parameters) ```yaml # .dsci/test_job/config.yaml selenium_browser: chrome # can be chrome, firefox, edge … base_url: "http://localhost:8000" tests_path: "tests/" # folder that contains your pytest‑style Selenium tests ``` ### 4.2 `test_job/job.py` – orchestrates the tasks ```python # .dsci/test_job/job.py #!/usr/bin/env python3 # ------------------------------------------------- # Job entry point – runs the three tasks in order # ------------------------------------------------- run_task("env_setup") # optional clean‑up / virtual‑env handling run_task("run_selenium") # actual Selenium test execution run_task("collect") # move reports / screenshots to ~/artifacts ``` ### 4.3 `test_job/tasks/env_setup/task.bash` ```bash # .dsci/test_job/tasks/env_setup/task.bash #!/usr/bin/env bash set -euo pipefail # (Optional) make sure the artifacts directory is clean before a new run echo "🧹 Cleaning previous artifacts" rm -rf ~/artifacts/* mkdir -p ~/artifacts # You could also create a virtual‑env here if you prefer isolation # python3 -m venv .venv # source .venv/bin/activate ``` ### 4.4 `test_job/tasks/run_selenium/task.py` ```python # .dsci/test_job/tasks/run_selenium/task.py #!/usr/bin/env python3 import sys import subprocess from pathlib import Path # ---------------------------------------------------------------------- # Helper to read job parameters (defaults or pipeline overrides) # ---------------------------------------------------------------------- cfg = config() browser = cfg.get("selenium_browser", "chrome") base_url = cfg.get("base_url", "http://localhost") tests_path = cfg.get("tests_path", "tests/") # ---------------------------------------------------------------------- # Run the Selenium test suite (we assume you are using pytest + selenium) # ---------------------------------------------------------------------- # The command below can be adapted to your own test runner. # `pytest` will generate a JUnit XML report and, if you use # `pytest-selenium` or custom code, screenshots will be stored under # the `screenshots/` folder. cmd = [ sys.executable, "-m", "pytest", tests_path, f"--base-url={base_url}", f"-n", "auto", # run tests in parallel if you have pytest-xdist "--junitxml=report.xml", # JUnit report that we will archive later ] print(f"🚀 Running Selenium tests on {browser} against {base_url}") # Export the chosen browser to the environment – many Selenium helper libs read it import os os.environ["SELENIUM_BROWSER"] = browser # Execute the test command result = subprocess.run(cmd, capture_output=True, text=True) # Show stdout / stderr for visibility in CI logs print(result.stdout) print(result.stderr, file=sys.stderr) # ---------------------------------------------------------------------- # Publish a tiny state so downstream jobs can know if the suite passed # ---------------------------------------------------------------------- tests_passed = result.returncode == 0 update_state({"tests_passed": tests_passed}) # Exit with the same code so the job fails if any test failed sys.exit(result.returncode) ``` > **Why `update_state`?** > The next job (e.g. a “publish” job) could read `config()['_dsci_']['test']['tests_passed']` > to decide whether to push results, send a Slack notification, etc. ### 4.5 `test_job/tasks/collect/task.bash` ```bash # .dsci/test_job/tasks/collect/task.bash #!/usr/bin/env bash set -euo pipefail # The Selenium task produced: # - report.xml (JUnit report) # - screenshots/ (optional folder with PNGs) ARTIFACTS_DIR=~/artifacts echo "📦 Collecting test artifacts" # Move the JUnit report if [[ -f report.xml ]]; then mv report.xml "${ARTIFACTS_DIR}/" echo "✅ report.xml stored" fi # Move any screenshots if they exist if [[ -d screenshots ]]; then mv screenshots "${ARTIFACTS_DIR}/" echo "✅ screenshots stored" fi # You could also zip them if you like # zip -r "${ARTIFACTS_DIR}/selenium_artifacts.zip" report.xml screenshots/ ``` --- ## 5️⃣ (Optional) Example of a downstream job that **reads the state** If you later add a job called `publish` you could access the state produced by the `test` job like this: ```yaml # .dsci/jobs.yaml (append at the end) - id: publish path: publish_job/ ``` ```python # .dsci/publish_job/job.py #!/usr/bin/env python3 cfg = config() tests_passed = cfg["_dsci_"]["test"]["tests_passed"] if tests_passed: print("✅ All Selenium tests passed – publishing results …") # … e.g. upload artifacts, notify Slack, etc. else: print("❌ Tests failed – aborting publish step.") sys.exit(1) ``` --- ## 6️⃣ How the pieces work together | Step | What happens | Where | |------|--------------|-------| | **1️⃣ Install job** | `install/job.bash` → `install/tasks/deps/task.bash` runs `pip install -r …` | Bash | | **2️⃣ Test job – env** | `test/job.py` → `env_setup/task.bash` cleans `~/artifacts` (optional) | Bash | | **3️⃣ Test job – run** | `run_selenium/task.py` reads parameters via `config()`, launches Selenium tests, writes `report.xml` + screenshots, calls `update_state({"tests_passed": …})` | Python | | **4️⃣ Test job – collect** | `collect/task.bash` moves the generated files into `~/artifacts` so any later job can read them | Bash | | **5️⃣ (Optional) Publish job** | Reads the state (`config()['_dsci_']['test']['tests_passed']`) and decides what to do | Python | All **job‑level parameters** (e.g. `selenium_browser`, `base_url`) are declared in `test_job/config.yaml` and can be overridden directly in `jobs.yaml` under the `params:` block. Inside any task you retrieve them with `config()` (Python) or `$(config name)` (Bash). --- ## 7️⃣ TL;DR – Copy‑paste ready files Below are the exact files you need to create. Copy each block into a file with the *same* path and name shown in the header. --- ### `.dsci/jobs.yaml` ```yaml jobs: - id: install path: install_job/ - id: test path: test_job/ params: selenium_browser: chrome base_url: "https://example.com" ``` --- ### `requirements.txt` *(example – add your own deps)* ```text selenium pytest pytest-xdist pytest-selenium ``` --- ### `install_job/config.yaml` ```yaml requirements_file: requirements.txt ``` --- ### `install_job/job.bash` ```bash #!/usr/bin/env bash set -euo pipefail run_task "deps" ``` --- ### `install_job/tasks/deps/task.bash` ```bash #!/usr/bin/env bash set -euo pipefail REQ_FILE=$(config requirements_file) echo "📦 Installing Python requirements from ${REQ_FILE}" python3 -m pip install --upgrade pip python3 -m pip install -r "${REQ_FILE}" ``` --- ### `test_job/config.yaml` ```yaml selenium_browser: chrome base_url: "http://localhost:8000" tests_path: "tests/" ``` --- ### `test_job/job.py` ```python #!/usr/bin/env python3 run_task("env_setup") run_task("run_selenium") run_task("collect") ``` --- ### `test_job/tasks/env_setup/task.bash` ```bash #!/usr/bin/env bash set -euo pipefail echo "🧹 Cleaning previous artifacts" rm -rf ~/artifacts/* mkdir -p ~/artifacts ``` --- ### `test_job/tasks/run_selenium/task.py` ```python #!/usr/bin/env python3 import sys import subprocess from pathlib import Path cfg = config() browser = cfg.get("selenium_browser", "chrome") base_url = cfg.get("base_url", "http://localhost") tests_path = cfg.get("tests_path", "tests/") # expose the chosen browser to any Selenium helper that reads env