DSCI

Below is a **complete, ready‑to‑use DSCI pipeline** that * receives the remote host information as job parameters (or falls back to defaults in `config.yaml`) * copies the Chef cookbook to the remote host * runs `chef-client` on the remote host pointing at the copied cookbook * stores the Chef run output as an artifact and publishes a small state (the remote run‑log path) that later jobs could consume. --- ## 1️⃣ Directory layout ``` . ├── .dsci │ └── jobs.yaml # pipeline definition ├── chef_deploy/ # <-- job directory (id = chef_deploy) │ ├── config.yaml # default job parameters │ ├── job.bash # job entry point │ └── tasks │ ├── copy_cookbook │ │ └── task.bash │ └── run_chef │ └── task.bash └── README.md # (optional) documentation for the repo ``` --- ## 2️⃣ Pipeline file – **`.dsci/jobs.yaml`** Only the job list and (optionally) input parameters are declared – **no logic here**. ```yaml # .dsci/jobs.yaml jobs: - id: chef_deploy path: chef_deploy/ params: # These values can be overridden when the pipeline is triggered. remote_host: 10.0.0.42 # IP or hostname of the target remote_user: ubuntu # SSH user on the target ssh_key_path: /home/ci/.ssh/id_rsa # Private key visible to the runner cookbook_src: ./cookbooks/my_app # Path (relative to repo root) to the cookbook remote_cookbook_dir: /tmp/chef_cook # Where to copy the cookbook on the remote host chef_version: latest # Chef client version to use on the remote ``` *If you want to use the defaults defined in `chef_deploy/config.yaml`, simply omit the `params:` block or any of its keys – the `config()` SDK will merge them automatically.* --- ## 3️⃣ Job default parameters – **`chef_deploy/config.yaml`** ```yaml # chef_deploy/config.yaml remote_host: 127.0.0.1 remote_user: root ssh_key_path: /root/.ssh/id_rsa cookbook_src: ./cookbooks/default remote_cookbook_dir: /opt/chef_cookbooks chef_version: latest ``` --- ## 4️⃣ Job script – **`chef_deploy/job.bash`** The job orchestrates the two tasks (`copy_cookbook` → `run_chef`). All parameter values are read with the `$(config …)` helper; the job also makes the Chef run‑log path available to downstream jobs via `update_state`. ```bash #!/usr/bin/env bash # chef_deploy/job.bash # ------------------------------------------------------------------ # Helper: read a parameter (fallback to default from config.yaml) # ------------------------------------------------------------------ remote_host=$(config remote_host) remote_user=$(config remote_user) ssh_key_path=$(config ssh_key_path) cookbook_src=$(config cookbook_src) remote_cookbook_dir=$(config remote_cookbook_dir) chef_version=$(config chef_version) # ------------------------------------------------------------------ # 1️⃣ Copy the cookbook to the remote host # ------------------------------------------------------------------ run_task copy_cookbook \ remote_host "$remote_host" \ remote_user "$remote_user" \ ssh_key_path "$ssh_key_path" \ cookbook_src "$cookbook_src" \ remote_cookbook_dir "$remote_cookbook_dir" # ------------------------------------------------------------------ # 2️⃣ Execute chef-client on the remote host # ------------------------------------------------------------------ run_task run_chef \ remote_host "$remote_host" \ remote_user "$remote_user" \ ssh_key_path "$ssh_key_path" \ remote_cookbook_dir "$remote_cookbook_dir" \ chef_version "$chef_version" # ------------------------------------------------------------------ # Export the location of the Chef run‑log so other jobs can read it. # ------------------------------------------------------------------ run_log_path="${HOME}/artifacts/chef_run_$(date +%s).log" update_state "{ \"chef_run_log\": \"${run_log_path}\" }" ``` > **Note** – `run_task` is the SDK‑provided helper that runs a task folder under the current job’s `tasks/` directory. > All *task variables* are passed as `NAME VALUE` pairs; they become accessible inside the task via `$(task_var NAME)` (Bash) or `task_var("NAME")` (Python). --- ## 5️⃣ Task 1 – copy the cookbook – **`chef_deploy/tasks/copy_cookbook/task.bash`** ```bash #!/usr/bin/env bash # chef_deploy/tasks/copy_cookbook/task.bash # ------------------------------------------------------------------ # Read task‑specific variables (they were passed by the job) # ------------------------------------------------------------------ remote_host=$(task_var remote_host) remote_user=$(task_var remote_user) ssh_key_path=$(task_var ssh_key_path) cookbook_src=$(task_var cookbook_src) remote_cookbook_dir=$(task_var remote_cookbook_dir) # ------------------------------------------------------------------ # Ensure the remote target directory exists # ------------------------------------------------------------------ ssh -i "${ssh_key_path}" -o StrictHostKeyChecking=no "${remote_user}@${remote_host}" \ "mkdir -p \"${remote_cookbook_dir}\"" # ------------------------------------------------------------------ # rsync the whole cookbook directory to the remote host # ------------------------------------------------------------------ rsync -avz -e "ssh -i ${ssh_key_path} -o StrictHostKeyChecking=no" \ "${cookbook_src}/" "${remote_user}@${remote_host}:${remote_cookbook_dir}/" # ------------------------------------------------------------------ # (Optional) Export the remote path as state for later tasks in this job # ------------------------------------------------------------------ update_state "{ \"remote_cookbook_dir\": \"${remote_cookbook_dir}\" }" ``` --- ## 6️⃣ Task 2 – run Chef – **`chef_deploy/tasks/run_chef/task.bash`** ```bash #!/usr/bin/env bash # chef_deploy/tasks/run_chef/task.bash # ------------------------------------------------------------------ # Pull in variables from the job (passed through run_task) # ------------------------------------------------------------------ remote_host=$(task_var remote_host) remote_user=$(task_var remote_user) ssh_key_path=$(task_var ssh_key_path) remote_cookbook_dir=$(task_var remote_cookbook_dir) chef_version=$(task_var chef_version) # ------------------------------------------------------------------ # Define where we will store the Chef run output locally (artifact) # ------------------------------------------------------------------ run_log="${HOME}/artifacts/chef_run_$(date +%s).log" # ------------------------------------------------------------------ # Build the remote command # ------------------------------------------------------------------ remote_cmd=$(cat <<'EOF' set -e # Install Chef client if not present (or enforce version) if ! command -v chef-client >/dev/null 2>&1; then curl -L https://omnitruck.chef.io/install.sh | sudo bash -s -- -v "${CHEF_VERSION}" fi # Run chef-client against the copied cookbook cd "${REMOTE_COOKBOOK_DIR}" chef-client --local-mode --runlist 'recipe[my_app::default]' \ --log_level info \ --log_location "${REMOTE_COOKBOOK_DIR}/chef_run.log" EOF ) # ------------------------------------------------------------------ # Export variables that the remote command expects # ------------------------------------------------------------------ export CHEF_VERSION="${chef_version}" export REMOTE_COOKBOOK_DIR="${remote_cookbook_dir}" # ------------------------------------------------------------------ # Execute the remote command via ssh and capture its output # ------------------------------------------------------------------ ssh -i "${ssh_key_path}" -o StrictHostKeyChecking=no "${remote_user}@${remote_host}" \ "bash -s" <<< "$(echo "$remote_cmd")" 2>&1 | tee "${run_log}" # ------------------------------------------------------------------ # (Optional) Pull the remote Chef log back to the runner for easier debugging # ------------------------------------------------------------------ scp -i "${ssh_key_path}" -o StrictHostKeyChecking=no \ "${remote_user}@${remote_host}:${remote_cookbook_dir}/chef_run.log" \ "${run_log}.remote" || true # ignore if remote log missing # ------------------------------------------------------------------ # Export the path of the locally‑saved log as state (already done in the job, # but we keep it here for completeness if the task runs standalone) # ------------------------------------------------------------------ update_state "{ \"chef_run_log\": \"${run_log}\" }" ``` > **What the script does** > 1. **Installs Chef** on the target (if missing) – you can pin a version via the `chef_version` param. > 2. **Runs `chef-client --local-mode`** against the transferred cookbook. > 3. Streams the remote console output into a **pipeline artifact** (`~/artifacts/chef_run_*.log`). > 4. Copies the remote log file back to the runner (optional, handy for later inspection). --- ## 7️⃣ Accessing the Chef run‑log from a **later job** (example) If another job needs the path of the log generated above, it can read it from the pipeline state: ```yaml # .dsci/jobs.yaml (adding a second job) jobs: - id: chef_deploy path: chef_deploy/ - id: report path: report_job/ ``` **`report_job/job.bash`** ```bash #!/usr/bin/env bash # report_job/job.bash # Pull the state exported by the previous job state_json=$(config) # full config JSON (includes _dsci_ state) log_path=$(echo "${state_json}" | jq -r '._dsci_.chef_deploy.chef_run_log') echo "=== Chef run log from previous job ===" cat "${log_path}" || echo "Log not found!" ``` --- ## 8️⃣ Summary of the solution | Component | Purpose | |-----------|---------| | **`jobs.yaml`** | Declares a single job (`chef_deploy`) and its input parameters. | | **`config.yaml`** | Provides sensible defaults that can be overridden at pipeline trigger time. | | **`job.bash`** | Orchestrates the two tasks, passes parameters, and publishes the run‑log path as state. | | **`copy_cookbook/task.bash`** | Uses `rsync` over SSH to copy the local Chef cookbook to the remote host; updates job‑local state. | | **`run_chef/task.bash`** | Installs (if needed) and runs `chef-client` on the remote host; stores output as an artifact and updates state. | | **Artifact directory (`~/artifacts/`)** | Holds the Chef run log so that any downstream job can read it. | | **State handling (`update_state`)** | Makes the log path available to later jobs via the `_dsci_` namespace. | All the heavy‑lifting is performed in **plain Bash**, which is perfect for SSH/rsync operations, while the DSCI SDK (`config`, `task_var`, `run_task`, `update_state`) handles parameter passing and state sharing without any extra YAML complexity. You can now drop the above files into your repository, push, and let DSCI execute the Chef cookbook on any reachable remote host. 🎉