test1.git | context.md
# What is DSCI
DSCI replaces complex CI/CD YAML with a tiny, high-level structural YAML file (jobs.yaml)
and executes all actual logic using standard Bash or Python in jobs (job.py/job.bash) and tasks (task.bash/task.py)
# Format
- Provide the file structure, the jobs.yaml, and the necessary tasks and jobs files
(e.g. tasks - task.bash/task.python, jobs - job.bash/job.py),
- Use relevant language (Bash or Python depending on the context).
# Constraints
## Job input parameters or job default configuration parameters
Job input parameters are optional and defined within `params:` block inside jobs.yaml file:
```
jobs:
-
id: job1
path: path1/
params:
foo: bar
```
Jobs default parameters are defined in job root directory, in `config.yaml` file:
`path1/config.yaml`:
```
foo: bar
```
Job input parameters if set override job default configuration parameters
## Access job input parameters / default configuration parameters
Use SDK `config()` function to read parameters inside jobs or tasks
### Python
In python job or tasks:
- Use `config()` function to access pipeline input parameters
You don't need to import any external libraries to make config() available in Python, this job is done by DSCI under the hood
Example:
```
cfg = config()
foo = cfg['foo']
```
### Bash
In bash jobs or tasks:
- Use `$(config name)` function to access pipeline input parameters, example:
```
foo = $(config foo)
```
# States
Jobs and tasks may have states allowing share data with other jobs and tasks
## To export state
In python tasks:
- Use SDK `update_state(dict)` function to export state, example:
```
state = { name: "value" }
update_state(state)
```
You don't need to import any external libraries to make update_state available in Python, this job is done by DSCI under the hood
In bash tasks:
- Use SDK `update_state name "value"` function to export state, example:
```
update_state name value
```
WARNING!!!
update_state does not work incrimentally, instead of (in Bash)
```
update_state name1 value1
update_state name1 value2
```
use
```
update_state '{ "name": "value1", "name2": "value2" }'
```
The same for Python, instead of
```
update_state({name1: "value1"})
update_state({name2: "value2"})
```
use
```
update_state({name1: "value1", name2: "value2"})
```
Or else you'll lose all the data in state beside the one you set
## To import state
### State from upstream jobs
#### Python
In import states from other upstream jobs use SDK `config()` function, example for Python:
```
config = config()
data = dict['_dsci_']['job1']['name']
```
You should _dsci_ prefixes in state object, when accessing state from other jobs.
#### Bash
To import state in Bash jobs or tasks use `config | jq` equivalent
```
data = config | jq '._dsci_.job1.name'
``
### State from upstream tasks within the same job
To import states from tasks defined within the same job use SDK `get_state()` function, example for Python:
```
state = config()
data = dict['name']
```
#### Bash
To import state from the same job in Bash use `get_state | jq` equivalent
data = get_state | jq '.name.'
# Files shares
If you need share files use artifacts/ directory to save/restore files between jobs/tasks
CRITICAL RULES:
1. DO NOT use complex YAML syntax, environment matrices, or third-party actions
2. Do not confuse DSCI with gitlab/github actions
3. The jobs.yaml file must ONLY list the job names and their execution entry points and optionally jobs input parameters
4. All build, test, and deploy logic must be written in clean, standard Bash/Python jobs/tasks
Use Bash for simple build/test instructions. Use Python for something more complex
5. Separate the jobs.yaml configuration and the script files clearly using Markdown code blocks labeled with filenames
6. Provide real solution even though there is no source code to build yet
7. Don't show how to run pipeline manually, I don't need that
# Documentation
Use following text as DSCI documentation
--- START OF DOCUMENTATION ---
# CICD
99%<sup>*</sup> less YAML pipeline engine
\* Yep - only tiny, simply structured YAML on the top (only to list jobs technically) everything else is general purpose programming languages
# Features
[Pros]
* no painful YAML pipelines, use general purpose programming languages for your pipelines instead
* repeatable steps are written as [plugins](/bash-plugins.md) on general purpose programming languages and used as is
* for developers - pipelines get run as normal scripts
[Limitations]
* no job dependencies
* ??? ( i don't know major ones, but let me know ;-)
# Show me the code
In your source code repository just drop few tasks under `.dsci` directory:
```
.
├── job_one
│ ├── job.py
│ └── tasks
│ ├── task_one
│ │ └── task.py
│ └── task_two
│ └── task.bash
├── job_two
│ ├── job.bash
│ └── tasks
│ └── task_one
│ └── task.bash
└── jobs.yaml
```
## Pipeline
*jobs.yaml*
```yaml
# runs job1, job2 sequentially
jobs:
-
id: job1
path: job_one/
-
id: job2
path: job_two/
```
## Job file ( job one )
*job_one/job.py*
```python
#!/bin/python3
run_task(
'task_one', {
'foo' : 'foo value',
'bar' : 'bar value'
}
);
run_task(
'task_two', {
'foo' : 'foo value',
'bar' : 'bar value'
}
);
```
## Task file ( task one )
*job_one/tasks/task_one/task.py*
```python
print(task_var("foo"))
print(task_var("bar"))
```
## Task file ( task two )
*job_one/tasks/task_two/task.bash*
```bash
#!/bin/bash
echo "hello from task one you passed: foo=${foo}, bar=${bar}"
```
Job file ( job two ):
*job_two/job.bash*
```bash
#!/bin/bash
run_task "task_one"
```
---
Etc.
## Passing states between tasks
Just use update\_state function within any tasks/* task to set some data:
```python
#!/usr/python3
update_state({
'out1' : 'out1 value',
'out2' : 'out2 value'
})
```
Then pick it up within any other tasks/* task by using get\_state function
```python
#!/bin/python
dict = get_state()
print(dict["out1"])
```
## Passing states between jobs
Just use update\_state function within any tasks/* task to set some data.
For example within job1, task_one:
```python
#!/usr/python3
update_state({
'out1' : 'out1 value',
'out2' : 'out2 value'
})
```
Then pick it up within any other jobs tasks/* task by using config function
```python
#!/bin/python
dict = config()
print(dict["_dsci_"]["job1"]["out1"])
```
# Programming languages supported
The same SDK for those programming languages:
* Bash
* Python
* Golang
* Php
* Ruby
* Powershell
* Raku
* Perl5
Choose the one you like and use it for pipeline. No extra code is required!
TBD - pipeline examples
# Using job plugins
There are a plenty of job [plugins](/bash-plugins.md) for common tasks:
- installing database, services, etc
For example:
*jobs.yaml*
```yaml
jobs:
-
# install mariadb database
id: db
plugin: mariadb
-
id: job1
path: job_one/
-
id: job2
path: job_two/
-
```
Job [plugins](/bash-plugins.md) act as native jobs, for example can use get\_state/update\_state function.
# Containers support
Jobs are executed either on ephemeral containers (alpine:latest) or on localhost
# Running pipelines locally
* enable debug option for a job, f.e.
```yaml
jobs:
-
id: job1
path: job_one/
debug: true
```
Then run job and copy job effective configuration from output.
* Paste configuration into some file, f.e.`.config.json`
* Run job locally
```bash
docker run --env SP6_TASK_CONFIG_FROM=.config.json -it --entrypoint /bin/bash -v $PWD:/opt/job dsci -c "cd /opt/job/; s6 --task-run ."
```
# Pipelines vs Jobs vs Tasks vs Plugins
- [Pipeline](/pipeline.md) is a list of jobs executed sequentially
- [Job](/job.md) is a list of tasks executed sequentially
- Isolation:
Jobs are executed on isolated environments, while [tasks](/task.md) within a certain job are executed in the same enviorment
- Environments are represented by ephemeral containers
- Jobs/tasks can share/pass states
- [Plugins](/bash-plugins.md) are the same as jobs, but reusable jobs published to https://sparrowhub.io
# Further reading
- [Jobs](/job.md)
- [Pipelines](/pipeline.md)
- [Demo Server](https://dsci.sparrowhub.io)
# Jobs
To create a job, create `job.$ext` file inside job folder, where $ext is one of supported by SDK languages extension, for example:
*job_one/job.python*
```python
run_task(
'task_one', {
'foo' : 'foo value',
'bar' : 'bar value'
}
);
run_task(
'task_two', {
'foo' : 'foo value',
'bar' : 'bar value'
}
);
```
Following is a list of job file names for different languages:
```
+------------+--------------+
| Language | File |
+------------+--------------+
| Raku | job.raku |
| Perl | job.pl |
| Bash | job.bash |
| Python | job.py |
| Ruby | job.rb |
| Powershell | job.ps1 |
| Php | job.php |
| Golang | job.go |
+------------+--------------+
```
Usually job is just a sequence of running [tasks](/task.md), but maybe any code:
*job_one/job.python*
```python
print("hello from job")
run_task(
'task_one', {
'foo' : 'foo value',
'bar' : 'bar value'
}
);
```
Task execution is done by calling `run_task()` function which is supported by all SDK languages, it takes the first mandatory parameter as a path inside `tasks/` directory where task files resides, and optionally tasks variables passed as dictionary
`run_task` function signatures for supported languages:
```
+------------+----------------------------------------------+
| Language | Signature |
+------------+----------------------------------------------+
| Raku | run_task(String,HASH) |
| Perl | run_task(SCALAR,HASHREF) |
| Bash | run_task TASK_NANE NAME VAL NAME2 VAL2 |
| Python | run_task(STRING,DICT) |
| Ruby | run_task(STRING,HASH) |
| Powershell | run_task(STRING,HASH) |
| Php | run_task(STRING,DICT) |
+------------+----------------------------------------------+
```
## Tasks variables
Tasks variables are handled inside task by using `task_var()` function available by all SDK languages:
Bash:
```bash
foo=$(task_var foo)
# in Bash you may also use shorted form:
echo $bar
```
Python:
```python
foo = task_var["foo"]
```
Raku
```raku
say task_var("foo");
```
`task_var` function signatures for supported languages:
```
+------------------+------------------------------------------------+
| Language | Signature |
+------------------+------------------------------------------------+
| Raku | task_var(STRING) |
| Perl | task_var(SCALAR) |
| Python | task_var(STRING) |
| Ruby | task_var(STRING) |
| Bash (1-st way) | $foo |
| Bash (2-nd way) | $(task_var foo) |
| Powershell | task_var(STRING) |
| Php | task_var(STRING) |
+------------------+------------------------------------------------+
```
# Default job parameters
If there is a file named `config.yaml` inside job directory it sets default jobs parameters:
```yaml
name: Alexey
occupation: IT
residency: Russia
```
Job parameters are available within tasks via `config()` function which is supported for all SDK languages:
```python
dict = config()
print(dict["name"])
print(dict["occupation"])
print(dict["residency"])
```
Job default parameters might be overridden by job parameters inside pipeline
Tasks
To create a task, create a task folder within `tasks/` directory and within this folder task file named as `task.$ext`, where $ext is one of supported by SDK languages extension, for example:
*tasks/task_one/task.python*
```python
print("hello from task")
```
*tasks/task_two/task.bash*
```bash
echo "hello from task"
```
Following is a list of tasks file names for different languages:
```
+------------+--------------+
| Language | File |
+------------+--------------+
| Raku | task.raku |
| Perl | task.pl |
| Bash | task.bash |
| Python | task.py |
| Ruby | task.rb |
| Powershell | task.ps1 |
| Php | task.php |
| Golang | task.go |
+------------+--------------+
```
Tasks are called from [job file](/job.md) using `run_task()` function, it takes the first input parameter as a path to sub directory within `tasks/` directory:
```python
#!/bin/python3
run_task("task_one"); # will run task from tasks/task_one/ folder
run_task("task_two"); # will run task from tasks/task_two/ folder
```
---
## Shared tasks
Sometimes it's sensible to share tasks across many jobs, to do so just create task inside `.dsci/shareds/tasks` folder, for example:
`dsci/shareds/tasks/build/task.bash` could be used as is a regular job task from job file using `run_task` method:
```python
#!/bin/python3
run_task("build"); # # will run task from dsci/shareds/tasks/build/ folder
run_task("task_one"); # will run task from tasks/task_one/ folder
run_task("task_two"); # will run task from tasks/task_two/ folder
```
If the same task exists within ~/.dsci/shared/tasks and tasks/, local job task will override shared task
## Single task job
Sometimes all you need is a single task job.
To create single task job just drop `task.$ext` file under job directory, no job file is required:
`.dsci/jobs.yaml`
```yaml
jobs:
-
id: job1
path: .
```
`.dsci/task.py`
```python
print("hello world")
```
Tasks may take some input variables by passing the second argument to `run_task()`function:
```python
run_task("task_one", { "foo" : "bar" });
```
Task variable is handled inside tasks by calling `task_var()` function
```python
foo = task_var("foo")
```
`run_task` function signatures for supported languages:
```
+------------+----------------------------------------------+
| Language | Signature |
+------------+----------------------------------------------+
| Raku | run_task(String,HASH) |
| Perl | run_task(SCALAR,HASHREF) |
| Bash | run_task TASK_NAME NAME VAL NAME2 VAL2 |
| Python | run_task(STRING,DICT) |
| Ruby | run_task(STRING,HASH) |
| Powershell | run_task(STRING,HASH) |
+------------+----------------------------------------------+
```
## Job parameters
Tasks may also get an access to job input parameters by using `config()` function, say job has a parameter named `param1`:
```yaml
param1: value1
```
then task may access the parameter like this:
Python example:
```python
job_params = config()
print(job_params["param1"])
```
`config()` function signature for supported languages:
```
+-------------+-------------------+
| Language | signature |
+-------------+-------------------+
| Raku | config() |
| Perl | config() |
| Bash(*) | config(string) |
| Python | config() |
| Ruby | config() |
| Powershell | config() |
| Php | config() |
+-------------+-------------------+
```
(*) Bash has following use case:
```bash
#!bash
param1=$(config param1)
echo $param1
```
---
Default job parameters
If there is a file named `config.yaml` inside job directory it sets default job parameters:
```yaml
param1: default_value
```
Inside task default parameters are accissble via `config` function
---
Tasks variables and job parameters serve the same purpose - configure your scripts, the difference is job parameters are set inside pipeline or in job default configuration file, while tasks variables are defined inside job file.
Also tasks variables are dynamic by their nature as defined using general programming languages, while job parameters are static and defined in plain YAML files.
## Exchange state between tasks
Tasks may exchange data between tasks within the same job by using `update_state()`,`get_state()` functions.
Say, task_one set two parameters (out1, out2) like this:
```python
#!/usr/python3
update_state({
'out1' : 'out1 value',
'out2' : 'out2 value'
})
```
Then any other task may pick it up by using `get_state()` function
```python
#!/bin/python
dict = get_state()
print(dict["out1"])
```
`update_state` function signatures for Sparrow6 supported languages:
```
+-------------+-----------------------------+
| Language | signature |
+-------------+-----------------------------+
| Raku | update_state(array|hash) |
| Perl | update_state(array|hash) |
| Bash(*) | update_state(key,value) |
| Python | update_state(array|dict) |
| Ruby | update_state(array|hash) |
| Powershell | update_state(array|hash) |
| Php | update_state(array|dict) |
+-------------+-----------------------------+
```
(*) Bash has a limited key/value support only:
```bash
#!bash
update_state "cnt" 100
```
`get_state()` function signature for supported languages:
```
+-------------+-------------------+
| Language | signature |
+-------------+-------------------+
| Raku | get_state() |
| Perl | get_state() |
| Bash | not supported |
| Python | get_state() |
| Ruby | get_state() |
| Powershell | get_state() |
| Php | get_state() |
+-------------+-------------------+
```
## Exchange state between jobs
To exchange data between different jobs the last executed `update_state()` results are taken and available for further consumption in other jobs.
Let's say we have `job_one/tasks/task_one/task.py` task ( within job with id `job1`) that define some state:
```python
update_state({
'message' : 'hello from job job_one'
})
```
Then any other job may read the data by using `get_state()`:
```python
state = get_state()
message = state["_dsci_"]["job1"]["message"]
```
Notice that key path is built from job ID, which is `job1`, it should also start with reserved key "\_dsci\_"
# Pipelines
To create a pipeline, create `jobs.yaml` file inside `.dsci` folder in repository root:
```yaml
# runs job1, job2 sequentially
jobs:
-
id: job1
path: job_one/
-
id: job2
path: job_two/
```
Jobs contain list of jobs to be executed sequentially. Every job has to have a unique id. If path is set job source is taken from path directory, see [job](job)
Job list may contain [plugins](/bash-plugins.md) instead of regular jobs:
```yaml
jobs:
-
id: build_essential
plugin: build-essential
```
[Plugins](/bash-plugins.md) are reusable jobs, see [http://sparrowhub.io](https://sparrowhub.io/search?q=all) for available _public_ plugins
## Parameters
Jobs and [plugins](/bash-plugins.md) may take input parameters:
```yaml
# runs job1, job2 sequentially
jobs:
-
id: job1
path: job_one/
params:
foo: bar
size: 10
colors:
- blue
- green
- red
-
id: database
plugin: mariadb
params:
db_user: alpine
db_pass: SecReet
db_name: products
```
Input parameters within job tasks are handled by the use of `config()` function available for all supported by SDK languages:
Bash:
```bash
foo=$(config foo)
```
Python:
```python
cfg = config()
print(cfg["foo"])
```
Raku
```raku
say config()<foo>;
```
---
## Single task job
Sometimes all you need is a single task job.
To create single task job just drop `task.$ext` file under job directory,
no job file is required:
`.dsci/jobs.yaml`
```yaml
jobs:
-
id: job1
path: .
```
`.dsci/task.py`
```python
print("hello world")
```
# Job conditions
Job conditions allow to skip or pass jobs on certain criteria.
Passing rules operate on job environment parameters is in `.<param> $condition` form, for example:
Skip job_one is commit message is equal to 'skip'
```yaml
jobs:
-
id: job_one
path: .
skip: .<message> eq "skip"
```
One can combine multiple job env parameters using standard AND/OR/||/&& operators from Boolean logic:
```yaml
jobs:
-
id: job_one
path: .
skip: .<message> eq "skip" or .<ref> eq "refs/heads/dev"
```
# Job environment parameters
## ref
Branch/Tag name. Example:
refs/heads/main
## message
Commit message
## repo_full_name
Full repository name. Example:
root/go-build-example
## sha
Commit SHA. Example:
689b7f1e30537514759ce40086fd2218fb95793d
## scm
Git repository URL. Example:
http://127.0.0.1:3000/root/go-build-example.git
## Regular expressions
One may use Raku regular expression to match against them.
Following example only passes job with branch name dev or master:
```yaml
jobs:
-
id: job_one
path: .
only: .<ref> ~~ "refs/heads/" [ master | main ]
```
## Parenthesis
One can use Parenthesis for grouping of logical expressions.
```yaml
jobs:
-
id: job_one
path: .
only: >
( .<ref> eq "refs/heads/main" and .<message> eq "release_prod" ) or
( .<ref> eq "refs/heads/dev" and .<message> eq "release_dev" )
```
## Operators
Following the list of operators one may use:
## eq
equivalent to string
## nq
not equivalent to string
## <, >, >=, <=, !=
less then, greater then, less or equal, greater or equal, not equal then an Int
## ~~
regular expression matching
## !~~
regular expression matching, negation form
## Regular expression modifiers
One can modify regular expression modifiers, to change matching logic.
Following example skip job if commit message is `skip_ci` or `Skip_Ci` or `SKIP_CI` - case insensitive
modifier
```yaml
jobs:
-
id: job_one
path: .
skip: m:i/skip_ci/
```
Follow [Raku regexs](https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://docs.raku.org/language/regexes) for details
## Global jobs conditions
To globally skip/pass all jobs use `global:` key in the root of pipeline configuration:
```yaml
global:
skip: .<message> ~~ m:/skip/
jobs:
-
id: job_one
path: .
```
# Job Artifacts
DSCI dramatically simplify working with artifacts
process inside pipelines.
To create artifact file just create it in `~/artifacts/`
directory in some task file, for example:
```bash
#!/bin/bash
echo "DSCI is cool" > ~/artifacts/message.txt
```
The the next job will see it in the same directory:
```python
#!/usr/bin/python3
from pathlib import Path
file_path = Path.home() / "artifacts/message.txt"
# Read the file created by previous job(s)
try:
with open(file_path, "r", encoding="utf-8") as file:
content = file.read()
print(content)
except FileNotFoundError:
print(f"Error: The file at {file_path} was not found.")
```
---
That's it. Artifacts de-facto is any file located in `~/artifacts` dir,
If some job decides to remove a file from `~/artifacts/` it won't be available for the next
jobs.
So artifacts works as a pipeline data buffer
--- END OF DOCUMENTATION ---