Automation using Gitlab CI

 

 

Nils Werner, Lorenz Schmidt

nils.werner@audiolabs-erlangen.com

 

International Audio Laboratories Erlangen

Interactive Workshop

• Immediate questions: interrupt me
• Individual issues/anything not covered: ask me at the end

Content

Motivation

Gitlab CI Basics

• Basic terminology
• Architecture
• Parameterization

Examples

• Python Unittests
• LaTeX Compilation
• DNN Training

Motivation

Many tasks in programming are

• boring
• repetetive
• important

Boring tasks

• compilation
• testing
• ML training

Automation to the rescue

• Formalize tasks
• Run automatically, reproducibly, and reliably

In modern workflows

• Write tasks as program
  • make
  • pytest
  • train.py
• Run on every code change
  • git push
• Run in isolation
  • Re-create environment every time

Gitlab CI Basics

Basic Terminology

• Build automation: Automated compilation and testing
• Continous Integration (CI): Everybody frequently merging against a common main branch. Usually implies Build automation.
• Continous Delivery (CD): Automatically pushing code into production. Implies CI and Build automation.
• Gitlab: A Git server
• Gitlab CI: Continuous Integration built into Gitlab
• Gitlab Runner: A worker performing Gitlab CI tasks
• DevOps: Configuring your infrastructure as code (with revisions, testing, CI, CD...)
• MLOps: Configuring your ML experiments as code (with revisions, testing, preprocessing, training...)

Gitlab CI Infrastructure

• $1$ Gitlab instance
• $N \geq 1$ Gitlab Runner instances
• Gitlab Runners can be installed on any machine
• Gitlab Runners can be shared or private
• You don't need to be admin to add Gitlab Runner
• Gitlab Runners can be configured to run specific tasks only

Setting up Gitlab CI

1. Create .gitlab-ci.yml file
2. Push to Gitlab
3. Profit

Creating a .gitlab-ci.yml file

What data do you need?

• Your code (in same Git repo)
• The list of commands to run
• The environment
  • A Docker image to start from
  • Commands to install missing software

.gitlab-ci.yml Example

# .gitlab-ci.yml
test:3.8:                     # Task name
  image: python:3.8           # Task environment (Docker image)
  before_script:              # Task environment setup
    - apt-get install -yy libsndfile1-dev
    - pip install -U pip wheel
    - pip install -e .
  script:                     # Task execution
    - py.test

Advanced Features

• Multiple tasks in parallel/sequential
• Saving output data (artifacts)
• Secret variables
• Runner selection
• Website publishing
• Reusing definitions
• Caching
• Timed execution

Multiple parallel tasks

test:3.8:
  image: python:3.8
  script:
  # ...

test:3.9:
  image: python:3.9
  script:
  # ...

Multiple sequential tasks

stages:
  - tests
  - docs

test:3.8:
  stage: tests
  script:
  # ...

docs:
  stage: docs
  script:
  # ...

Saving artifacts

build:
  # ...

  artifacts:
    paths:
      - ClassicThesis.pdf

Reusing artifacts

build:
  # ...

  artifacts:
    paths:
      - ClassicThesis.pdf

test:
  # ...

  dependencies:
    - build           # `ClassicThesis.pdf` will be available

Environment Variables

variables:                  # Environment variables set in code
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

test:3.8:
   # ...

   script:
     - pip install numpy    # pip uses the variable $PIP_CACHE_DIR by default

Secret Variables

test:3.8:
   # ...

   script:
     - python $SECRET        # Defined in Gitlab settings, not in code

Runner selection

train:
    tags:                     # Runners with all tags will run this
      - cuda11
      - gtx1080
    # ...

    script:
      - python train.py

Website publishing

pages:                           # task name `pages` is important
  # ...                          # script must produce output in public/

  artifacts:
    paths:
    - public

Your repo https://git.audiolabs.uni-erlangen.de/$user/$repo/

Will be visible under https://$user.pages.audiolabs.uni-erlangen.de/$repo/

Reusing definitions

.job_template: &job_definition
  # ...
  script:
    - py.test

test:3.8:
  image: python:3.8
  <<: *job_definition

test:3.9:
  image: python:3.9
  <<: *job_definition

Caching

cache:
  paths:
    - downloaded-data/  # Will be restored on subsequent runs

test:3.8:
  image: python:3.8
  script:
  # ...

Run conditions

test:3.8:
  image: python:3.8
  script:
  # ...

  only:
  - master

Only runs on the master branch

Examples

• nils/ci-examples
  • Everything
• nils/s-tdar-paper-2019
  • LaTeX Compilation
  • Output saving
• nils/mlops
  • Runner selection
• nils/subband-merging
  • Reuse of definitions
  • Website publishing
• nils/impfbot
  • Secret variables
  • Timed execution
• python/pyrir
  • Multiple stages
  • Website publishing

Best practices

• Keep it simple
• Write tests
• Use pytest
• pip install -U pip wheel first
• apt-get update -yy first
• Use popular Docker images
• Keep secrets in variables
• Keep installation-specific values in variables
• Use caching for large installs (LaTeX is 3GB of downloads!)

How to get started

# .gitlab-ci.yml
test:3.8:                     # Task name
  image: python:3.8           # Task environment (Docker image)
  before_script:              # Task environment setup
    - apt-get install -yy libsndfile1-dev
    - pip install -U pip wheel
    - pip install -e .
  script:                     # Task execution
    - py.test

Read More

Some Useful Links

https://docs.gitlab.com/ee/ci/
https://git.audiolabs.uni-erlangen.de/nils/ci-examples

Contact Me:

nils.werner@​audiolabs-erlangen.com

@nlswrnr

https://www.audiolabs-erlangen.de/fau/assistant/werner

Thank you!

Discussion points

• Should we use Gitlab Runners on Pascal & Tesla?