dda-devops-build/README.md

dda-devops-build

| team@social.meissa-gmbh.de | Website & Blog

dda-devops-build integrates all the tools we use to work with clouds & provide some nice functions around.

Tools we support are

• terraform: for setting up the plain infrastructure around.
• docker: for creating images
• c4k: for generating kubernetes manifests
• provs: for setting up small single-node k3s clusters
• gopass: for credential management on devops computers
• cloud providers: hetzner, digitalocean, aws

In addition we provide a ReleaseMixin for release related tasks like tag / publish & version-bump

classDiagram
    class DevopsBuild {
        name()
        build_path()
        initialize_build_dir()
    }


    class DevopsTerraformBuild {
        terraform_build_commons_path()
        project_vars()
        initialize_build_dir()
        post_build()
        read_output_json()
        plan()
        plan_fail_on_diff()
        apply(auto_approve=False)
        refresh()
        destroy(auto_approve=False)
        tf_import(tf_import_name, tf_import_resource,)
        print_terraform_command(terraform)
    }

    class DevopsImageBuild {
        def initialize_build_dir()
        image()
        drun()
        dockerhub_login()
        dockerhub_publish()
        test()
    }

    class ReleaseMixin {
        prepare_release()
        tag_and_push_release()
    }
    
    class ProvsK3sBuild {
        def update_runtime_config(dns_record)
        write_provs_config()
        provs_apply(dry_run=False)
    }

    class C4kBuild {
        def update_runtime_config(dns_record)
        def write_c4k_config()
        def write_c4k_auth()
        c4k_apply(dry_run=False)
    }

    DevopsBuild <|-- DevopsImageBuild
    DevopsBuild <|-- DevopsTerraformBuild
    DevopsBuild <|-- ReleaseMixin
    DevopsBuild <|-- ProvsK3sBuild
    DevopsBuild <|-- C4kBuild

Principles we follow are:

• Seperate build artefacts from version controlled code
• Domain Driven Design - in order to stay sustainable

Setup

Ensure that yout python3 version is at least Python 3.10

sudo apt install python3-pip
pip3 install -r requirements.txt
export PATH=$PATH:~/.local/bin

Example Build

lets assume the following project structure

my-project
   | -> my-module
   |       | -> build.py
   |       | -> some-terraform.tf
   | -> an-other-module
   | -> target  (here will the build happen)
   |       | -> ...

from pybuilder.core import task, init
from ddadevops import *

name = 'my-project'
MODULE = 'my-module'
PROJECT_ROOT_PATH = '..'

class MyBuild(DevopsTerraformBuild):
    pass


@init
def initialize(project):
    project.build_depends_on('ddadevops>=0.5.0')
    account_name = 'my-aws-account-name'
    account_id = 'my-aws-account-id'
    stage = 'my stage i.e. dev|test|prod'
    additional_vars = {'var_to_use_insied_terraform': '...'}
    additional_var_files = ['variable-' + account_name + '-' + stage + '.tfvars']
    config = create_devops_terraform_build_config(stage, PROJECT_ROOT_PATH,
                                                  MODULE, additional_vars,
                                                  additional_tfvar_files=additional_var_files)
    build = MyBuild(project, config)
    build.initialize_build_dir()


@task
def plan(project):
    build = get_devops_build(project)
    build.plan()


@task
def apply(project):
    build = get_devops_build(project)
    build.apply()

@task
def destroy(project):
    build = get_devops_build(project)
    build.destroy()

@task
def tf_import(project):
    build = get_devops_build(project)
    build.tf_import('aws_resource.choosen_name', 'the_aws_id')

Feature aws-backend

Will use a file backend.dev.live.properties where dev is the [account-name], live is the [stage].

the backend.dev.live.properties file content:

key = ".."
region = "the aws region"
profile = "the profile used for aws"
bucket = "the s3 bucket name"
kms_key_id = "the aws key id"

the build.py file content:

class MyBuild(AwsBackendPropertiesMixin, DevopsTerraformBuild):
    pass


@init
def initialize(project):
    project.build_depends_on('ddadevops>=1.0')
    account_name = 'my-aws-account-name'
    account_id = 'my-aws-account-id'
    stage = 'my stage i.e. dev|test|prod'
    additional_vars = {}
    config = create_devops_terraform_build_config(stage, PROJECT_ROOT_PATH,
                                                  MODULE, additional_vars)
    config = add_aws_backend_properties_mixin_config(config, account_name)
    build = MyBuild(project, config)
    build.initialize_build_dir()

Feature aws-mfa-assume-role

In order to use aws assume role in combination with the mfa-tool (pip install mfa):

the build.py file content:

class MyBuild(class MyBuild(AwsMfaMixin, DevopsTerraformBuild):
    pass


@init
def initialize(project):
    project.build_depends_on('ddadevops>=1.0')
    account_name = 'my-aws-account-name'
    account_id = 'my-aws-account-id'
    stage = 'my stage i.e. dev|test|prod'
    additional_vars = {}
    config = create_devops_terraform_build_config(stage, PROJECT_ROOT_PATH,
                                                  MODULE, additional_vars)
    config = add_aws_backend_properties_mixin_config(config, account_name)
    config = add_aws_mfa_mixin_config(config, account_id, 'eu-central-1',
                                      mfa_role='my_developer_role',
                                      mfa_account_prefix='company-',
                                      mfa_login_account_suffix='users_are_defined_here')
    build = MyBuild(project, config)
    build.initialize_build_dir()

@task
def access(project):
    build = get_devops_build(project)
    build.get_mfa_session()

Feature DdaImageBuild

The docker build supports image building, tagging, testing and login to dockerhost. For bash based builds we support often used script-parts as predefined functions see install_functions.sh.

A full working example: doc/example/50_docker_module

Feature AwsRdsPgMixin

The AwsRdsPgMixin provides

• execute_pg_rds_sql - function will optionally resolve dns-c-names for trusted ssl-handshakes
• alter_db_user_password
• add_new_user
• deactivate_user

the build.py file content:

class MyBuild(..., AwsRdsPgMixin):
    pass


@init
def initialize(project):
    project.build_depends_on('ddadevops>=1.0')

    ...
    config = add_aws_rds_pg_mixin_config(config,
                                         stage + "-db.bcsimport.kauf." + account_name + ".breuni.de",
                                         "kauf_bcsimport",
                                         rds_resolve_dns=True,)
    build = MyBuild(project, config)
    build.initialize_build_dir()

@task
def rotate_credentials_in(project):
    build = get_devops_build(project)
    build.alter_db_user_password('/postgres/support')
    build.alter_db_user_password('/postgres/superuser')
    build.add_new_user('/postgres/superuser', '/postgres/app', 'pg_group_role')


@task
def rotate_credentials_out(project):
    build = get_devops_build(project)
    build.deactivate_user('/postgres/superuser', 'old_user_name')

Releasing and updating

Publish snapshot

1. every push will be published as dev-dependency

Release

adjust version no in build.py to release version no.
git commit -am "release"
git tag -am "release" [release version no]
git push --follow-tags
increase version no in build.py
git commit -am "version bump"
git push
pip3 install --upgrade --user ddadevops

License

Copyright © 2021 meissa GmbH Licensed under the Apache License, Version 2.0 (the "License")