DEP 2 - DebOps code standards

DEP:2
Title:DebOps code standards
Author:Maciej Delmanowski, Robin Schneider, Reto Gantenbein
Status:Accepted
Type:Standards Track
Created:2016-11-05
Post-History:none

Abstract

This document defines code guidelines for Ansible roles included in DebOps. These guidelines are provided for Ansible role authors to ensure that the roles included in the project are interoperable and extensible.

Goals of the Policy

The DebOps code is comprised of Ansible roles which define data models and specific tasks that should be performed on hosts to achieve desired results (installation and configuration of a service or application, interaction with third-party software and services, etc.), Ansible playbooks which define what roles should be executed on which hosts, and Ansible inventory which defines what hosts Ansible should interact with, what playbooks to apply to these hosts and what parameters to use in the roles.

This Policy describes how the Ansible roles and playbooks should be written, what ways can be used to combine two or more roles together and how the roles should be documented.

Summary

Here's a basic set of principles to be aware while writing roles:

  • YAML files MUST use 2 space indents and end with .yml.
  • Use spaces around Jinja variable syntax as in {{ var }}.
  • When possible use 80 characters line width.
  • Write everything to be compatible with Python v2 and v3.

Ansible role: defaults

Ansible role: tasks

  • Make sure the task execution is idempotent.
  • Describe each task and include with the name option.
  • Use native YAML syntax for task definition formatting.
  • If some tasks should only be executed under certain circumstances, group them together and conditionally include the task list.
  • Set a minimal Ansible version in meta/main.yml according to the modules and task parameters used.
  • Disable debug mechanisms such as the debug or ignore_errors statements in the master branch.
  • Use tags to make individually usable tasks better accessible.

Ansible role: templates

  • If possible follow the directory structure of the target file when storing the Jinja2 template.
  • Properly indent Jinja2 loop constructs.

Ansible role: dependencies

  • Define role dependencies as "soft" dependencies via playbook and make them conditional if possible.
  • Avoid the use of "hard" role dependencies specified in meta/main.yml.
  • Don't directly include variables of other roles. Instead use Ansible facts to share configuration state between roles.
  • Provide dedicated inventory variables if the role configuration is meant to be extended by dependent roles.

Ansible role overview

Ansible roles are the basic building block of DebOps infrastructure. Due to the constraints put on them by the DebOps project, they need to be written in a certain way as to maximize to user's ability to use them through the Ansible inventory without a requirement to modify the role's code as well as offer the most amount of reusability so that other Ansible roles can utilize them if necessary.

Each role SHOULD focus on a specific service or application. Roles can be composed together inside playbooks if needed, so there's no need to put different services together in the same role. An exception here are very similar services like the different NTP daemons. This has the advantage to only having one set of default variables regardless of the particular chosen service and it makes changing the particular service very easy.

Ansible role default variables

Naming convention

DebOps roles MUST use a special variable naming scheme to indicate a "namespace" of a given role default variables. The variable MUST contain the role name, followed by two underscore characters, followed by the rest of the variable name. For example, the variable which defines the name of the nginx UNIX user account MUST be defined as (or similar):

nginx__user: 'www-data'

For another example, the list of APT packages to install on a particular host using the debops.apt_install role MUST be defined as (or similar):

apt_install__host_packages: [ 'bash' ]

Variables which are meant to be dependent variables on a provider role, MUST additionally contain the word dependent in their variable name. E.g.:

apt_preferences__dependent_list: []

Variables which are meant to define dependent variables in the consumer role are named after the dependent variable of the provider role prefixed with the consume role name. E.g.

nginx__apt_preferences__dependent_list:
  - package: 'nginx nginx-*'
    backports: [ 'wheezy', 'precise' ]
    by_role: 'debops.nginx'

Variable documentation

For each role the DebOps documentation will include a page which documents the default variables. This page is generated from the role's defaults/main.yml file with help of yaml2rst. The entire comment of the defaults file is thereby interpreted as reStructuredText and then rendered via Sphinx.

Each variable comment is started with a .. envvar:: reference anchor followed by the name of the variable. This construct allows any documentation page to reference the variable via :envvar:`varname` which Sphinx will translate into a link pointing to the variable description. Below the anchor comment should describe the purpose of the variable, accepted values, side effects and so on. Within the comment all reStructuredText constructs supported by Sphinx can be used. An example variable definition would eventually look like this:

# .. envvar:: ferm__flush [[[
#
# Should ferm-rules be flushed when :program:`ferm` is disabled? The default is true,
# but you may need set both :envvar:`ferm__enabled` and this to ``False`` if you are
# running in some container and are not allowed to change :command:`iptables`.
ferm__flush: '{{ ferm__enabled | bool }}'
                                                               # ]]]

Related default variables should be grouped to sections for a better overview and easier navigation. For example it makes sense to distinguish packaging and network related variables. Of course additional or different section titles might be meaningful for the individual role:

# APT packages [[[
# ----------------

[... packaging related variables ...]

                                                               # ]]]
# Network configuration [[[
# -------------------------

[... networking related variables ...]
                                                               # ]]]

Dependent variables

DebOps is designed in a way that there are divided responsibilities between the roles. Each role has its clear task to fulfill. For example an application role must never care about the firewall configuration by itself. There is a dedicated role for firewall configuration. And the application role needs a way to tell the firewall role which access rules to configure. This is done via dependent variables.

Each role providing a feature which MAY be consumed by other roles MUST provide a dedicated dependent variable for the configuration of this feature. As that variable is always defined in the context of the consuming role, its value MUST NOT modify a shared configuration state (e.g. configuration template). The provider role might be executed multiple times depending on the role dependency configuration of the consuming roles.

Caution

A role providing dependent variables MUST be able to handle multiple role executions with different values of the dependent variables without mutual interference.

Ideally the dependent variable SHOULD accept a list of YAML dictionaries where one property MUST be the state (present or absent) of the configuration. The provider role SHOULD then manage an individual configuration file per list item which allows it to selectively add or remove configuration states.

Additionally, the by_role property (string) SHOULD be accepted which can be used to indicate the role responsible for a given item. by_role MUST be given in the form of ROLE_OWNER.ROLE_NAME.

Example:

The debops.apt_preferences role implements a feature to set APT package pinning configurations. Each role requiring a specific version of a package to be available can ask the apt_preferences to do the corresponding configuration. For this, apt_preferences provides the following dependent variable:

# List of :man:`apt_preferences(5)` pins to configure in
# :file:`/etc/apt/preferences.d/`.  This variable is meant to be used from a
# role dependency in :file:`role/meta/main.yml` or in a playbook.
apt_preferences__dependent_list: []

The consumer role, in this case debops.nginx, would then define an own variable which defines the necessary pinning information:

nginx__apt_preferences__dependent_list:
  - package: 'nginx nginx-*'
    backports: [ 'wheezy', 'precise' ]
    by_role: 'debops.nginx'

In the playbook a soft dependency can be specified where the dependent variable is passed to the provider:

- name: Manage nginx webserver
  hosts: 'debops_service_nginx'
  become: True

  roles:

    - role: debops.apt_preferences
      apt_preferences__dependent_list:
        - '{{ nginx__apt_preferences__dependent_list }}'

    - role: debops.nginx

By including the configuration for the debops.apt_preferences role in the nginx default variables the user is able to change it through the Ansible inventory without the need to modify any of the involved roles or the playbook.

Inventory level scoped variables

The Ansible inventory allows managing hosts, such as executing playbooks or scoping variables, in three different levels:

  • All hosts in the inventory
  • All hosts which are member of a common user defined group
  • One host individually

When evaluating variable values Ansible would override variables when they are defined in a more specific level. This easily becomes an issue when dealing with lists as variable value.

To mitigate this DebOps role authors SHOULD provide three different variables, let's call it inventory level scoped variables, for the same configuration property. They are meant to be defined by the user in the respective inventory level context. The role then has to make sure that they are merged appropriately so that the configuration values of all levels are respected.

Example:

The debops.users role allows to define user accounts which should be created on a machine. For accounts which should be created on every machine, the user can define the following variable in the global level of the inventory (e.g. inventory/groups_vars/all/users.yml):

# List of user accounts to manage on all hosts in Ansible inventory.
users__accounts: []

Users which should only be created on a group of servers are defined in the related variable in group level inventory (e.g. inventory/group_vars/group_name/users.yml):

# List of UNIX user accounts to manage on hosts in specific Ansible inventory
# group.
users__group_accounts: []

And the same for host-specific users which are defined in the related host level inventory variable in (e.g. inventory/host_vars/hostname/users.yml):

# List of UNIX user accounts to manage on specific hosts in Ansible inventory.
users__host_accounts: []

Ansible role tasks

Ansible tasks are doing the actual work namely querying and modifying the target host. Each task defines a Ansible module invocation with a number of general and module specific options.

Description

Each task MUST have the name option set with a meaningful description. This allows to quickly reason about the change impact when running a playbook and classify the progress in case of an abortion.

Example:

- name: Check if password history database exists
  stat:
    path: '/etc/security/opasswd'
  register: auth__register_opasswd

The same is true for the include statement. Especially for conditional includes which may be skipped it's helpful for identifying which features of the role have been left out.

Example:

- name: Configure acme-tiny support
  include: acme_tiny.yml
  when: (pki__enabled|bool and
         (pki__acme|bool or pki__acme_install|bool))

Conditions

It might be often necessary that task execution depends on a certain condition using the when statement. In many cases the condition is simple and straight forward, for example when depending on the existence of a file. Other times the condition might be more complex, for example when depending on a state of other role configurations. In this case the expression SHOULD be defined in a default variable. This would give the user the ability to override the decision and have better control about the role's behavior.

Example:

- name: Add database server user to specified groups
  user:
    name: 'mysql'
    groups: '{{ mariadb_server__append_groups | join(",") | default(omit) }}'
    append: True
    createhome: False
  when: mariadb_server__pki|bool

Here the condition mariadb_server__pki is a extensive evaluation of the current state. The related default variable is defined in defaults/main.yml as following:

# Enable or disable support for SSL in MariaDB (using ``debops.pki``).
mariadb_server__pki: '{{ mariadb_server__pki_realm in ansible_local.pki.known_realms|d([]) }}'

If the user doesn't agree with the defined condition, the variable can simply be redefined in the Ansible inventory without the need to modify the Ansible code of the role.

YAML syntax

Task definitions MUST use the native YAML syntax formatting. Ansible accepts various ways to define Ansible tasks. However, there are several advantages by agreeing on the YAML syntax:

  • Unified coding style
  • Vertical formatting is easier to read
  • Supports complex parameter values (e.g. nested dictionaries)

Example:

Instead of ...

- name: Create plugin path
  file: path={{ elasticsearch__path_plugins }} state=directory

... use the correct YAML syntax:

- name: Create plugin path
  file:
    path: '{{ elasticsearch__path_plugins }}'
    state: 'directory'

Disable debug statements

Role authors MUST NOT unconditionally use Ansible debug mechanisms such as the debug module or the ignore_errors task statement in code which is used for normal operations. Released code is expected to be functional under every possible circumstance otherwise it is considered to be a bug which must be fixed on a best effort basis.

For fragile or complex code paths it might be acceptable to use the debug statement with an increased verbosity level. This will only show the message, if ansible-playbook is executed with one or more --verbose options. For example:

- name: Show intermediate value from a lookup query
  debug:
    var: '{{ lookup("template", "lookups/fancy_lookup.j2") }}'
    verbosity: 1

Ansible role dependencies

Soft role dependencies

Role dependencies are considered "soft" if they are defined in the roles list of a playbook. This approach offers a higher flexibility as the user can choose which roles to run and which features to include.

Whenever possible DebOps role authors MUST specify role dependencies via playbook instead of "hard" dependencies in the meta/main.yml.

It's also possible to pass configuration values to a dependent role if the involved role is offering dedicated dependent variables for this purpose. With soft dependencies custom playbook authors are therefore free to pass values from arbitrary sources according to their requirements.

When a role that is used as a soft dependency already contains its own dependencies, all of them SHOULD be included in the playbook to ensure that the required functionality (firewall access, APT preferences, etc.) is provided and configured as needed by the dependent role.

Example:

This is an example playbook for debops.slapd defining soft dependencies:

---

- name: Manage OpenLDAP service
  hosts: [ 'debops_service_slapd' ]
  become: True

  roles:

    - role: debops.ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ slapd__ferm__dependent_rules }}'

    - role: debops.tcpwrappers
      tags: [ 'role::tcpwrappers' ]
      tcpwrappers__dependent_allow:
        - '{{ slapd__tcpwrappers__dependent_allow }}'

    - role: debops.slapd
      tags: [ 'role::slapd' ]

Hard role dependencies

Role dependencies are considered "hard" if they are defined in the dependencies list in meta/main.yml. DebOps role authors MUST avoid the use of hard role dependencies for the following reasons:

  • Hard role dependencies must always be installed on the Ansible controller even when their execution is conditionally triggered via when statement.
  • It hinders the independent use of the role in a custom playbook or outside of DebOps where playbook authors might rely on a different role for a certain feature or decide not to use a certain feature at all.
  • The playbook execution flow is more difficult to reason about as hard dependencies are defined outside of the playbook.

Generally role dependencies MUST be defined as "soft" dependencies via playbook unless the tight coupling to another role is unavoidable for implementing the required functionality. A reasonable exception is for example the debops.secret role which defines a common path for the lookup("password") plugin.

Ansible role facts

Ansible facts are small things that are automatically discovered by the Ansible ansible.builtin.setup module on a target host when a playbook is executed. They can be used by Ansible role and playbook authors through normal variables. The default facts provided are indicated by the ansible_ namespace. It's possible to define custom facts through JSON or INI files or scripts returning such output in the /etc/ansible/facts.d directory of a host. These facts are then available as key/value pairs under the ansible_local variable.

Share configuration state with other roles

If a role needs to know a configuration state of another role it MUST NOT access inventory variables of the source role. This impedes the portability of a role and effectively makes the source role its hard dependency. Instead, roles SHOULD expose public data structures as needed for other roles to use as Ansible local facts. This ensures that the data used by other roles is available at all times, and therefore idempotent.

To set a local fact the role MUST define a template or copy task which writes a facts file /etc/ansible/facts.d/rolename.fact.

Example:

If a role needs to make a decision based on the fact if the firewall managed by debops.ferm is enabled or not, it MUST NOT check the value of ferm__enabled but query the local fact of the ferm role:

is_firewall_enabled: '{{ ansible_local.ferm.enabled|d("unknown") }}'

To successfully read the local fact of another role the latter obviously must have run before. Always consider the case that the fact may be undefined and fallback to a meaningful default value.

The debops.ferm role itself defines the facts via a Jinja2 template such as:

[...]
{
"enabled": "{{ ferm__enabled     | bool | lower }}",
"forward": "{{ ferm__tpl_forward | bool | lower }}",
"ansible_controllers": {{ ferm__tpl_ansible_controllers_result | to_nice_json }}
}

Python compatibility

The Python language is used on several levels in a direct and indirect fashion:

  • debops-tools on the controller (direct)
  • saved facts on the target hosts (direct)
  • Jinja templates (indirect)
  • Jinja within yaml (indirect)

In all incarnations the syntax MUST be written to be compatible with both Python Version 2 and 3. Additionally code MUST comply to the PEP8 coding style guide.

Some useful hints

  • Always use unicode strings explicitly:

Example:

Instead of ...

foo = ''

... use the unicode literal

foo = u''
  • Always use brackets for "print()"ing.
  • Always use "dict.items()" in favor of "dict.iteritems()" which is deprecated.
  • Be aware that "dict.keys()" behaves differently in Python 3. Especially with Jinja-filters.

Example:

Instead of ...

foo = bar.keys()

... or ...

foo: {{ bar.keys() | to_nice_json }}

... use the list filter/method

foo = list(bar)

... or ...

foo: {{ bar | list | to_nice_json }}