• 5 Posts
  • 18 Comments
Joined 3 years ago
Cake day: July 31st, 2023
  • Sp1983@programming.devOPtoProgramming@programming.dev•DSCI - CI engine with programming languages to write pipelines (no YAML)
    2·
    25 days ago

    … put those in separate scripts and call them from the YAML.

    This is exactly what I try to avoid, cause:

    1. many people (in my experience ) even don’t bother refactoring YAML spaghetti code to separate scripts and we end up unmaintainable codebase

    2. and even if one has to do such a refactoring what is the point of using YAML at all ?

    All I need just a collection of tasks/jobs written on languages of choice and I don’t need YAML “programming” language at all )

    PS And btw I don’t mind having a minimal amount of YAML as configuration layer and this is what is presented in DSCI, but only minimal ))

  • Sp1983@programming.devOPtoProgramming@programming.dev•DSCI - CI engine with programming languages to write pipelines (no YAML)
    1·
    25 days ago

    Ok, try to do it on GH actions, share states between tasks/jobs for example:

    tasks/task_one/task.py

    #!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

    tasks/task_two/task.py

    #!/usr/bin/python3

dict = get_state()
print(dict["out1"])
print(dict["out2"])

    Or share states between jobs:

    jobs/job1/task.py

    #!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

    jobs/job2/task.py

    #!/usr/bin/python3

dict = config()

print(dict["_dsci_"]["job1"]["out1"])
print(dict["_dsci_"]["job1"]["out2"])

    I can’t imagine how much boilerplate code (if this ever possible ) one needs to write to achieve that on YAML based pipelines (GH Actions/ etc)

    And don’t tell me about jobs artifacts ))

    UPDATE:

    Another good example is to run tasks conditionally, yes using old good if:

    #!/usr/bin/python3

if some_condition(foo, bar): 
    run_task(
       'task1', {
          'foo' : 'foo value',
          'bar' : 'bar value'
       }
    )

    The same could be quite awkward in YAML based code