6.1. CI/CD Devtools Ecosystem

6.1.1. Agility

• Pair Programming in PyCharm: https://www.jetbrains.com/help/pycharm/code-with-me.html

• Further Reading: https://dev.astrotech.io/agile/index.html

6.1.2. Ecosystem

• Further Reading: https://dev.astrotech.io/summary/index.html

6.1.3. Version Control System

• Git

• Git Flow

• Github Flow

• Github

• Bitbucket

• GitLab

• GitOps: https://www.gitops.tech

• FluxCD: https://github.com/fluxcd/flux

• Further Reading: https://dev.astrotech.io/git/index.html

6.1.4. Virtualization

• Docker

• LXC - Linux Containers

• OCI - Open Container Initiative

• Kubernetes

• Containerd

• OpenShift

• Open Stack

• Amazon EKS, ECS

• Further Reading: https://dev.astrotech.io/docker/index.html

6.1.5. Continuous Integration / Delivery

• Jenkins

• Github Actions

• Bitbucket Pipelines

• CircleCI

• Travis

• GitLab

• Further Reading: https://dev.astrotech.io/jenkins/index.html

6.1.6. Quality Assurance

• SonarQube

• SonarLint 1

• SonarScanner

• SonarCloud

• Coverage

• PEP-8

• PyLint

• Black

• Further Reading: https://dev.astrotech.io/sonarqube/index.html

• Further Reading: https://python.astrotech.io/devops/ci-cd/tools.html#static-analysis

• Further Reading: https://python.astrotech.io/devops/ci-cd/code-style.html

• Further Reading: https://python.astrotech.io/devops/ci-cd/coverage.html

• Further Reading: https://python.astrotech.io/devops/ci-cd/static-analysis.html

6.1.7. Issue Tracker

• Jira

• Gitlab

• Github issues

• Jira Integration: https://jira.astrotech.io/end-user/automation.html

• Further Reading: https://dev.astrotech.io/jira/index.html

6.1.8. SSH

• Further Reading: https://dev.astrotech.io/linux/index.html

6.1.9. Testing

• Further Reading: https://test.astrotech.io

6.1.10. Mutation Testing

• mutmut (actively maintained) https://github.com/boxed/mutmut

• mutpy (last commit 2019) https://github.com/mutpy/mutpy

• https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78

• https://www.geeksforgeeks.org/software-testing-mutation-testing/

Following code:

>>> def is_adult(age: int) -> bool:
...     if age >= 18:       # original line
...         return True
...     else:
...         return False

Will be modified to:

>>> def is_adult(age: int) -> bool:
...     if age > 18:        # mutated line
...         return True
...     else:
...         return False

And then all tests will be executed to check if you have good tests to cover for such change

mutmut 2.0 creates the following mutants (source):

• Operator mutations: About 30 different patterns like replacing + by - , * by ** and similar, but also > by >= .

• Keyword mutations: Replacing True by False , in by not in and similar.

• Number mutations: You can write things like 0b100 which is the same as 4, 0o100, which is 64, 0x100 which is 256, .12 which is 0.12 and similar. The number mutations try to capture mistakes in this area. mutmut simply adds 1 to the number.

• Name mutations: The name mutations capture copy vs deepcopy and "" vs None .

• Argument mutations: Replaces keyword arguments one by one from dict(a=b) to dict(aXXX=b).

• or_test and and_test: and ↔ or

• String mutation: Adding XX to the string.

Those can be grouped into three very different kinds of mutations:

• value mutations (string mutation, number mutation),

• decision mutations (switch if-else blocks, e.g. the or_test / and_test and the keyword mutations),

• statement mutations (removing or changing a line of code).

$ ln -s /usr/bin/python3 /home/ubuntu/bin/python
$ pip install mutmut
$ mutmut --help

$ mutmut run
$ mutmut results
$ mutmut show 2
$ mutmut show 22

$ mutmut html
$ python3 -m http.server 8080 --directory html

$ mutmut junitxml

$ rm -fr .mutmut-cache

Code 6.70. setup.cfg

[mutmut]
paths_to_mutate=src/
backup=False
runner=python -m hammett -x
tests_dir=tests/
dict_synonyms=Struct, NamedStruct

some_code_here()  # pragma: no mutate

In order to better integrate with CI/CD systems, mutmut supports the generation of a JUnit XML report (using https://pypi.org/project/junit-xml/). This option is available by calling mutmut junitxml. In order to define how to deal with suspicious and untested mutants, you can use

$ mutmut junitxml --suspicious-policy=ignore --untested-policy=ignore

The possible values for these policies are:

• ignore: Do not include the results on the report at all

• skipped: Include the mutant on the report as "skipped"

• error: Include the mutant on the report as "error"

• failure: Include the mutant on the report as "failure"

If a failed mutant is included in the report, then the unified diff of the mutant will also be included for debugging purposes.

6.1.11. BDD Testing

• Lettuce: http://lettuce.it/index.html

• Cucumber: https://cucumber.io

• Behave: https://behave.readthedocs.io/en/stable/tutorial.html

Example:

Feature: showing off behave

  Scenario: run a simple test
     Given we have behave installed
      When we implement a test
      Then behave will test it for us!

from behave import *

@given('we have behave installed')
def step_impl(context):
    pass

@when('we implement a test')
def step_impl(context):
    assert True is not False

@then('behave will test it for us!')
def step_impl(context):
    assert context.failed is False

$ behave
Feature: showing off behave # features/tutorial.feature:1

  Scenario: run a simple test        # features/tutorial.feature:3
    Given we have behave installed   # features/steps/tutorial.py:3
    When we implement a test         # features/steps/tutorial.py:7
    Then behave will test it for us! # features/steps/tutorial.py:11

1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
3 steps passed, 0 failed, 0 skipped, 0 undefined

Parameters:

Scenario: look up a book
  Given I search for a valid book
   Then the result page will include "success"

Scenario: look up an invalid book
  Given I search for a invalid book
   Then the result page will include "failure"

@then('the result page will include "{text}"')
def step_impl(context, text):
    if text not in context.response:
        fail('%r not in %r' % (text, context.response))

Step Data:

Scenario Outline: Blenders
   Given I put <thing> in a blender,
    when I switch the blender on
    then it should transform into <other thing>

 Examples: Amphibians
   | thing         | other thing |
   | Red Tree Frog | mush        |

 Examples: Consumer Electronics
   | thing         | other thing |
   | iPhone        | toxic waste |
   | Galaxy Nexus  | toxic waste |

6.1.12. Load Testing

• Gatling: https://gatling.io

• Locust: https://locust.io/

• JMeter: https://jmeter.apache.org

Code 6.71. https://gatling.io/docs/gatling/tutorials/quickstart/#gatling-scenario-explained

package computerdatabase // 1

import scala.concurrent.duration._

// 2
import io.gatling.core.Predef._
import io.gatling.http.Predef._

class BasicSimulation extends Simulation { // 3

  val httpProtocol = http // 4
    .baseUrl("http://computer-database.gatling.io") // 5
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // 6
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")

  val scn = scenario("BasicSimulation") // 7
    .exec(
      http("request_1") // 8
        .get("/")
    ) // 9
    .pause(5) // 10

  setUp( // 11
    scn.inject(atOnceUsers(1)) // 12
  ).protocols(httpProtocol) // 13
}

What does it mean?

1. The optional package.

2. The required imports.

3. The class declaration. Note that it extends Simulation.

4. The common configuration to all HTTP requests.

5. The baseUrl that will be prepended to all relative urls.

6. Common HTTP headers that will be sent with all the requests.

7. The scenario definition.

8. An HTTP request, named request_1. This name will be displayed in the final reports.

9. The url this request targets with the GET method.

10. Some pause/think time.

6.1.13. Testing UI

• Selenium: https://www.selenium.dev

from selenium import webdriver
from selenium.webdriver.common.by import By


def test_eight_components():
    driver = webdriver.Chrome()

    driver.get("https://google.com")

    title = driver.title
    assert title == "Google"

    driver.implicitly_wait(0.5)

    search_box = driver.find_element(by=By.NAME, value="q")
    search_button = driver.find_element(by=By.NAME, value="btnK")

    search_box.send_keys("Selenium")
    search_button.click()

    search_box = driver.find_element(by=By.NAME, value="q")
    value = search_box.get_attribute("value")
    assert value == "Selenium"

    driver.quit()

6.1.14. Testing microservices

• Further Reading: https://arch.astrotech.io

• Source: https://martinfowler.com/articles/microservice-testing/

6.1.15. Provisioning

• Ansible

• Puppet

• Chef

• Salt, SaltStack

• Vagrant

• Further Reading: https://dev.astrotech.io/puppet/index.html

• Further Reading: https://dev.astrotech.io/ansible/index.html

• Further Reading: https://dev.astrotech.io/vagrant/index.html

6.1.16. Setup

git clone https://github.com/sages-pl/src-python /home/ubuntu/src
sudo apt update
sudo apt install -y uidmap
curl https://get.docker.com/rootless |sh
echo 'export PATH=/home/ubuntu/.local/bin:$PATH' >> ~/.profile
echo 'export DOCKER_HOST=unix:///run/user/1000/docker.sock' >> ~/.profile
echo 'export IP=$(curl -s ipecho.net/plain)' >> ~/.profile
source ~/.profile
docker network create ecosystem

Gitea:

cat > /home/ubuntu/bin/run-gitea << EOF

docker run \\
    --name gitea \\
    --detach \\
    --restart always \\
    --env USER_UID=1000 \\
    --env USER_GID=1000 \\
    --env GITEA__server__ROOT_URL=http://$IP:3000/ \\
    --env GITEA__database__DB_TYPE=sqlite3 \\
    --env GITEA__database__PATH=/var/lib/gitea/data/gitea.db \\
    --env GITEA__database__HOST=... \\
    --env GITEA__database__NAME=... \\
    --env GITEA__database__USER=... \\
    --env GITEA__database__PASSWD=... \\
    --dns 8.8.8.8 \\
    --network ecosystem \\
    --publish 3000:3000 \\
    --publish 2222:22 \\
    --volume gitea_data:/var/lib/gitea \\
    --volume gitea_config:/etc/gitea \\
    --volume /etc/timezone:/etc/timezone:ro \\
    --volume /etc/localtime:/etc/localtime:ro \\
    gitea/gitea:latest-rootless

echo "Gitea running on: http://$IP:3000/"

EOF

chmod +x /home/ubuntu/bin/run-gitea
run-gitea

Jenkins:

cat > /home/ubuntu/bin/run-jenkins << EOF

chmod o+rw /run/user/1000/docker.sock
sudo ln -s /usr/bin/python3 /usr/bin/python
sudo ln -s /home/ubuntu/.local/share/docker/volumes/jenkins_data/_data/ /var/jenkins_home

docker run \\
    --name jenkins \\
    --detach \\
    --restart always \\
    --network ecosystem \\
    --publish 8080:8080 \\
    --volume jenkins_data:/var/jenkins_home \\
    --volume /run/user/1000/docker.sock:/var/run/docker.sock \\
    jenkinsci/blueocean:latest

docker exec -u root jenkins apk add python3 py3-pip

echo "Jenkins running on: http://$IP:8080/"

EOF

chmod +x /home/ubuntu/bin/run-jenkins
run-jenkins

SonarQube:

cat > /home/ubuntu/bin/run-sonarqube << EOF

docker run \\
    --name sonarqube \\
    --detach \\
    --restart always \\
    --network ecosystem \\
    --publish 9000:9000 \\
    --volume sonarqube_data:/opt/sonarqube/data \\
    --volume sonarqube_logs:/opt/sonarqube/logs \\
    --volume sonarqube_extensions:/opt/sonarqube/extensions \\
    sonarqube

echo "SonarQube running on: http://$IP:9000/"

EOF

chmod +x /home/ubuntu/bin/run-sonarqube
run-sonarqube

SonarScanner:

• sonar-project.properties

• Further Reading: https://dev.astrotech.io/sonarqube/sonarscanner.html

• Further Reading: https://python.astrotech.io/devops/ci-cd/static-analysis.html

docker pull sonarsource/sonar-scanner-cli

Docker Registry:

cat > /home/ubuntu/bin/run-registry << EOF

docker run \\
    --detach \\
    --restart always \\
    --name registry \\
    --net ecosystem \\
    --publish 5000:5000 \\
    --volume registry_data:/var/lib/registry \\
    registry:2

echo "Registry running on: http://$IP:5000/"

EOF

chmod +x /home/ubuntu/bin/run-registry
run-registry

Registry UI:

cat > /home/ubuntu/registry-ui.yml << EOF

listen_addr: 0.0.0.0:8888
base_path: /

registry_url: http://registry:5000
verify_tls: true

# registry_username: user
# registry_password: pass

# The same one should be configured on Docker registry as Authorization Bearer token.
event_listener_token: token
event_retention_days: 7

event_database_driver: sqlite3
event_database_location: data/registry_events.db
# event_database_driver: mysql
# event_database_location: user:password@tcp(localhost:3306)/docker_events

cache_refresh_interval: 10

# If users can delete tags.
# If set to False, then only admins listed below.
anyone_can_delete: false

# Users allowed to delete tags.
# This should be sent via X-WEBAUTH-USER header from your proxy.
admins: []

# Debug mode. Affects only templates.
debug: true

# How many days to keep tags but also keep the minimal count provided no matter how old.
purge_tags_keep_days: 90
purge_tags_keep_count: 2

EOF

cat > /home/ubuntu/bin/run-registry-ui << EOF

docker run \\
    --name registry-ui \\
    --detach \\
    --restart always \\
    --network ecosystem \\
    --publish 8888:8888 \\
    --volume /home/ubuntu/registry-ui.yml:/opt/config.yml:ro \\
    quiq/docker-registry-ui

echo "Registry UI running on: http://$IP:8888/"

EOF

chmod +x /home/ubuntu/bin/run-registry-ui
run-registry-ui

Files:

cat > /home/ubuntu/src/Dockerfile << EOF
FROM python:3.10
COPY game.pyz /game.pyz
CMD python3 /game.pyz
EOF

cat > /home/ubuntu/src/sonar-project.properties << EOF
## Sonar Server
sonar.host.url=http://sonarqube:9000/
sonar.login=TOKEN

## Software Configuration Management
sonar.scm.enabled=true
sonar.scm.provider=git

## SonarScanner Config
sonar.sourceEncoding=UTF-8
sonar.verbose=false
sonar.log.level=INFO
sonar.showProfiling=false
sonar.projectBaseDir=/usr/src/
sonar.working.directory=/tmp/

## Quality Gates
sonar.qualitygate.wait=true
sonar.qualitygate.timeout=300

## About Project
sonar.projectKey=mypythonproject
sonar.projectName=MyPythonProject

## Python
sonar.language=py
sonar.python.version=3.10
sonar.sources=src
sonar.tests=test
sonar.inclusions=**/*.py
sonar.exclusions=**/migrations/**,**/*.pyc,**/__pycache__/**
sonar.python.xunit.skipDetails=false
sonar.python.xunit.reportPath=.tmp/xunit.xml
sonar.python.coverage.reportPaths=.tmp/coverage.xml,./cobertura.xml
sonar.python.bandit.reportPaths=.tmp/bandit.json
sonar.python.pylint.reportPaths=.tmp/pylint.txt
sonar.python.flake8.reportPaths=.tmp/flake8.txt

EOF

cat > /home/ubuntu/src/Jenkinsfile << EOF
pipeline {
  agent any
  triggers { pollSCM('* * * * *') }

  stages {
    stage('Env Prepare')            { steps { sh 'run/env-prepare' }}
    stage('Env Setup')              { steps { sh 'run/env-setup' }}
    stage('Env Debug')              { steps { sh 'run/env-debug' }}

    stage('Test') {
    parallel {
        stage('Test Code Style')    { steps { sh 'run/test-codestyle' }}
        stage('Test Functional')    { steps { sh 'run/test-functional' }}
        stage('Test Integration')   { steps { sh 'run/test-integration' }}
        stage('Test Lint')          { steps { sh 'run/test-lint' }}
        stage('Test Load')          { steps { sh 'run/test-load' }}
        stage('Test Mutation')      { steps { sh 'run/test-mutation' }}
        stage('Test Regression')    { steps { sh 'run/test-regression' }}
        stage('Test Security')      { steps { sh 'run/test-security' }}
        stage('Test Smoke')         { steps { sh 'run/test-smoke' }}
        stage('Test Static')        { steps { sh 'run/test-static' }}
        stage('Test UI')            { steps { sh 'run/test-ui' }}
        stage('Test Unit')          { steps { sh 'run/test-unit' }}
    }}
    stage('Test Report')            { steps { sh 'run/test-report' }}

    stage('Artifact Prepare')       { steps { sh 'run/artifact-prepare' }}
    stage('Artifact Build')         { steps { sh 'run/artifact-create' }}
    stage('Artifact Publish')       { steps { sh 'run/artifact-publish' }}
    stage('Artifact Cleanup')       { steps { sh 'run/artifact-cleanup' }}

    stage('Deploy Dev')             { steps { sh 'run/deploy-dev' }}
    stage('Deploy Test')            { steps { sh 'run/deploy-test' }}
    stage('Deploy Preprod')         { steps { sh 'run/deploy-preprod' }}
    stage('Deploy Prod')            { steps { sh 'run/deploy-prod' }}
  }
}

// To run all:
// grep -Po "^[^/].*sh '\K.+(?=')" Jenkinsfile |sh -x

EOF

Tests:

cd /home/ubuntu/src
mkdir -p run/
touch run/test-codestyle
touch run/test-coverage
touch run/test-functional
touch run/test-integration
touch run/test-lint
touch run/test-load
touch run/test-mutation
touch run/test-regression
touch run/test-report
touch run/test-security
touch run/test-smoke
touch run/test-static
touch run/test-ui
touch run/test-unit
touch run/artifact-prepare
touch run/artifact-create
touch run/artifact-publish
touch run/artifact-cleanup
touch run/deploy-dev
touch run/deploy-test
touch run/deploy-preprod
touch run/deploy-prod
chmod +x run/*

cat > run/env-prepare << EOF
env |sort
EOF

cat > run/env-setup << EOF
python3 -m pip install --upgrade -r requirements.dev
EOF

cat > run/env-debug << EOF
which python3
python3 --version
python3 -m pip freeze
EOF

cat > run/test-codestyle << EOF
export PYTHONPATH=src
python3 -m flake8 --exit-zero --doctest --output-file=.tmp/flake8.txt src
EOF

cat > run/test-coverage << EOF
export PYTHONPATH=src
python3 -m coverage run src
python3 -m coverage xml -o .tmp/coverage.xml
EOF

cat > run/test-functional << EOF
echo 'Not Implemented'
EOF

cat > run/test-integration << EOF
export PYTHONPATH=src
python3 -m doctest -v test/*.py
EOF

cat > run/test-lint << EOF
export PYTHONPATH=src
python3 -m pylama --verbose --async src || true
python3 -m pylint --exit-zero --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" --output=.tmp/pylint.txt --disable=C0114,C0115,C0116,E0401,C0103 src
EOF

cat > run/test-load << EOF
echo 'Not Implemented'
EOF

cat > run/test-mutation << EOF
rm -fr .mutmut-cache
mutmut run || true
mutmut results
mutmut junitxml --suspicious-policy=ignore --untested-policy=ignore > .tmp/xunit.xml
EOF

cat > run/test-regression << EOF
echo 'Not Implemented'
EOF

cat > run/test-report << EOF
docker run --rm --net ecosystem -v $(pwd):/usr/src sonarsource/sonar-scanner-cli
EOF

cat > run/test-security << EOF
export PYTHONPATH=src
python3 -m bandit --format json --output=.tmp/bandit.json --recursive src
EOF

cat > run/test-smoke << EOF
echo 'Not Implemented'
EOF

cat > run/test-static << EOF
export PYTHONPATH=src
python3 -m mypy --ignore-missing-imports --cobertura-xml-report=.tmp src || test
EOF

cat > run/test-ui << EOF
echo 'Not Implemented'
EOF

cat > run/test-unit << EOF
export PYTHONPATH=src
python3 -m unittest discover -v test
EOF

cat > run/artifact-prepare << EOF
python3 -m pip install --upgrade --no-cache-dir -r requirements.prod --target src
rm -fr src/*.dist-info
python3 -m compileall -f src
# find src -name '*.py' -not -name '__main__.py' -not -name '__init__.py' -delete  # not working for now
python3 -m zipapp --python="/usr/bin/env python3" --output=game.pyz src
EOF

cat > run/artifact-create << EOF
docker build . -t localhost:5000/myapp:$(git log -1 --format='$h')
EOF

cat > run/artifact-publish << EOF
docker push localhost:5000/myapp:$(git log -1 --format='$h')
EOF

cat > run/artifact-cleanup << EOF
docker rmi localhost:5000/myapp:$(git log -1 --format='$h')
EOF

cat > run/deploy-dev << EOF
echo 'Not Implemented'
EOF

cat > run/deploy-test << EOF
echo 'Not Implemented'
EOF

cat > run/deploy-preprod << EOF
echo 'Not Implemented'
EOF

cat > run/deploy-prod << EOF
echo 'Not Implemented'
EOF

6.1.17. References

1

https://www.sonarlint.org