map.jinja: gather formula configuration values

Newest map.jinja only available in a few formulas so far

This document is specific to the newest map.jinja incarnation, also known as the v5 map.jinja.

At the time of writing, this is only available in a handful of formulas, as well as the template-formula itself.

The documentation explains the use of a map.jinja to gather parameters values for a formula.

As pillars are rendered on the Salt master for every minion, this increases the load on the master as the pillar values and the number of minions grows.

As a good practice, you should:

  • store non-secret data in YAML files distributed by the fileserver

  • store secret data in:

Current best practice is to let map.jinja handle parameters from all sources, to minimise the use of pillars, grains or configuration from sls files and templates directly.

Table of Contents

1. For formula users

1.1. Quick start: configure per role and per DNS domain name values

We will see a quick setup to configure the TEMPLATE formula for different DNS domain names and several roles.

For this example, I’ll define 2 kinds of fileserver sources:

  1. formulas git repositories with hard-coded version reference to avoid breaking my setup randomly at upstream update. they are the last sources where files are looked up

  2. parameters of the formulas in the file backend roots

1.1.1. Configure the fileserver backends

I configure the fileserver backends to serve:

  1. files from roots first

  2. gitfs repositories last

Create the file /etc/salt/master.d/fileserver.conf and restart the master:

---
##
## file server
##
fileserver_backend:
  # parameters values and override
  - roots
  # formulas
  - gitfs

# The files in this directory will take precedence over git repositories
file_roots:
  base:
    - /srv/salt

# List of formulas I'm using
gitfs_remotes:
  - https://github.com/saltstack-formulas/template-formula.git:
    - base: v4.1.1
  - https://github.com/saltstack-formulas/openssh-formula.git:
    - base: v2.0.1
...

1.1.2. Create per DNS configuration for TEMPLATE formula

Now, we can provides the per DNS domain name configuration files for the TEMPLATE formulas under /srv/salt/TEMPLATE/parameters/.

We create the directory for dns:domain grain and we add a symlink for the domain grain which is extracted from the minion id:

mkdir -p /srv/salt/TEMPLATE/parameters/dns:domain/
ln -s dns:domain /srv/salt/TEMPLATE/parameters/domain

We create a configuration for the DNS domain example.net in /srv/salt/TEMPLATE/parameters/dns:domain/example.net.yaml:

---
values:
  config: /etc/template-formula-example-net.conf
...

We create another configuration for the DNS domain example.com in the Jinja YAML template /srv/salt/TEMPLATE/parameters/dns:domain/example.com.yaml.jinja:

---
values:
  config: /etc/template-formula-{{ grains['os_family'] }}.conf
...

1.1.3. Create per role configuration for TEMPLATE formula

Now, we can provides the per role configuration files for the TEMPLATE formulas under /srv/salt/TEMPLATE/parameters/.

We create the directory for roles:

mkdir -p /srv/salt/TEMPLATE/parameters/roles

We will define 2 roles:

  • TEMPLATE/server

  • TEMPLATE/client

We create a configuration for the role TEMPLATE/server in /srv/salt/TEMPLATE/parameters/roles/TEMPLATE/server.yaml:

---
values:
  config: /etc/template-formula-server.conf
...

We create another configuration for the role TEMPLATE/client in /srv/salt/TEMPLATE/parameters/roles/TEMPLATE/client.yaml:

---
values:
  config: /etc/template-formula-client.conf
...

1.1.4. Enable roles and the dns:domain and domain grains for map.jinja

We need to redefine the sources for map.jinja to load values from our new configuration files, we provide a global configuration for all our minions.

We create the global parameters file /srv/salt/parameters/map_jinja.yaml:

---
values:
  sources:
    # default values
    - "Y:G@osarch"
    - "Y:G@os_family"
    - "Y:G@os"
    - "Y:G@osfinger"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"

    # Roles activate/deactivate things
    # then thing are configured depending on environment
    # So roles comes before `dns:domain`, `domain` and `id`
    - "Y:C@roles"

    # DNS domain configured (DHCP or resolv.conf)
    - "Y:G@dns:domain"

    # Based on minion ID
    - "Y:G@domain"

    # default values
    - "Y:G@id"
...

The syntax is explained later at Sources of configuration values.

1.1.5. Bind roles to minions

We associate roles grains to minion using grains.append.

For the servers:

salt 'server-*' grains.append roles TEMPLATE/server

For the clients:

salt 'client-*' grains.append roles TEMPLATE/client

Note

Since we used Y:C@roles, map.jinja will do a salt['config.get']('roles') to retrieve the roles so you could use any other method to bind roles to minions (pillars or SDB) but grains seems to be the preferred method.

1.1.6. Note for Microsoft Windows systems

If you have a minion running under windows, you can’t use colon : as a delimiter for grain path query (see bug 58726) in which case you should use an alternate delimiter:

Modify /srv/salt/parameters/map_jinja.yaml to change the query for dns:domain to define the alternate delimiter:

---
values:
  sources:
    # default values
    - "Y:G@osarch"
    - "Y:G@os_family"
    - "Y:G@os"
    - "Y:G@osfinger"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"

    # Roles activate/deactivate things
    # then thing are configured depending on environment
    # So roles comes before `dns:domain`, `domain` and `id`
    - "Y:C@roles"

    # DNS domain configured (DHCP or resolv.conf)
    - "Y:G:!@dns!domain"

    # Based on minion ID
    - "Y:G@domain"

    # default values
    - "Y:G@id"
...

And then, rename the directory:

mv /srv/salt/TEMPLATE/parameters/dns:domain/  '/srv/salt/TEMPLATE/parameters/dns!domain/'

1.2. Format of configuration YAML files

When you write a new YAML file, note that it must conform to the following layout:

  • a mandatory values key to store the configuration values

  • two optional keys to configure the use of salt.slsutil.merge

    • an optional strategy key to configure the merging strategy, for example strategy: 'recurse', the default is smart

    • an optional merge_lists key to configure if lists should be merged or overridden for the recurse and overwrite strategy, for example merge_lists: 'true'

Here is a valid example:

---
strategy: 'recurse'
merge_lists: 'false'
values:
  pkg:
    name: 'some-package'
  config: '/path/to/a/configuration/file'
...

1.2.1. Using Jinja2 YAML template

You can provide a Jinja2 YAML template file with a name suffixed with .yaml.jinja, it must produce a YAML file conform to the Format of configuration YAML files, for example:

---
strategy: 'overwrite'
merge_lists: 'true'
values:
{%- if grains["os"] == "Debian" %}
  output_dir: /tmp/{{ grains["id"] }}
{%- endif %}
...

1.3. Sources of configuration values

The map.jinja file aggregates configuration values from several sources:

For the values loaded from YAML files, map.jinja will automatically try to load a Jinja2 template with the same name as the YAML file with the addition of the .jinja extension, for example foo/bar/quux.yaml.jinja.

After loading values from all sources, it will try to include the salt://parameters/post-map.jinja Jinja file if it exists which can post-process the mapdata variable.

1.3.1. Configuring map.jinja sources

The map.jinja file uses several sources where to lookup parameter values. The list of sources can be configured in two places:

  1. globally

    1. with a plain YAML file salt://parameters/map_jinja.yaml

    2. with a Jinja2 YAML template file salt://parameters/map_jinja.yaml.jinja

  2. per formula

    1. with a plain YAML file salt://{{ tplroot }}/parameters/map_jinja.yaml

    2. with a Jinja2 YAML template file salt://{{ tplroot }}/parameters/map_jinja.yaml.jinja

Note

The map.jinja configuration files must conform to the format of configuration YAML files.

Each source definition has the form [<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY> where <TYPE> can be one of:

The YAML type option can define the query method to lookup the key value to build the file name:

The C, G or I types can define the SUB option to store values in the sub key mapdata.<KEY> instead of directly in mapdata.

All types can define the <DELIMITER> option to use an alternate delimiter of the <KEY>, for example: on windows system you can’t use colon : for YAML file path name and you should use something else like exclamation mark !.

Finally, the <KEY> describes what to lookup to either build the YAML filename or gather values using one of the query methods.

Note

For the YAML type:

  • if the <KEY> can’t be looked up, then it’s used a literal string path to a YAML file, for example: any/path/can/be/used/here.yaml will result in the loading of salt://{{ tplroot }}/parameters/any/path/can/be/used/here.yaml if it exists

  • map.jinja will automatically try to load a Jinja2 template, after the corresponding YAML file, with the same name as the YAML file extended with the .jinja extension, for example any/path/can/be/used/here.yaml.jinja

The built-in map.jinja sources are:

- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "Y:G@id"

This is strictly equivalent to the following map_jinja.yaml.jinja:

values:
  sources:
    - "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml"
    - "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml.jinja"
    - "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml"
    - "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml.jinja"
    - "parameters/os/{{ salt['grains.get']('os') }}.yaml"
    - "parameters/os/{{ salt['grains.get']('os') }}.yaml.jinja"
    - "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml"
    - "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml.jinja"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"
    - "parameters/id/{{ salt['grains.get']('id') }}.yaml"
    - "parameters/id/{{ salt['grains.get']('id') }}.yaml.jinja"

1.3.2. Loading values from the configuration sources

For each configuration source defined, map.jinja will:

  1. load values depending on the source type:

    • for YAML file sources

      • if the <KEY> can be looked up:

        • load values from the YAML file named salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml if it exists

        • load values from the Jinja2 YAML template file named salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml.jinja if it exists

      • otherwise:

        • load the YAML file named salt://{{ tplroot }}/parameters/<KEY>.yaml if it exists

        • load the Jinja2 YAML template file named salt://{{ tplroot }}/parameters/<KEY>.yaml.jinja if it exists

    • for C, G or I source type, lookup the value of salt['<QUERY_METHOD>']('<KEY>')

  2. merge the loaded values with the previous ones using salt.slsutil.merge

There will be no error if a YAML or Jinja2 file does not exists, they are all optional.

1.3.3. Configuration values from salt['config.get']

For sources with of type C declared in map_jinja:sources, you can configure the merge option of salt['config.get'^] by defining per formula strategy configuration key (retrieved with salt['config.get'](tplroot ~ ':strategy') with one of the following values:

  • recurse merge recursively dictionaries. Non dictionary values replace already defined values

  • overwrite new value completely replace old ones

By default, no merging is done, the first value found is returned.

1.3.4. Global view of the order of preferences

To summarise, here is a complete example of the load order of formula configuration values for an AMD64 Ubuntu 18.04 minion named minion1.example.net for the libvirt formula:

  1. parameters/defaults.yaml

  2. parameters/defaults.yaml.jinja

  3. parameters/osarch/amd64.yaml

  4. parameters/osarch/amd64.yaml.jinja

  5. parameters/os_family/Debian.yaml

  6. parameters/os_family/Debian.yaml.jinja

  7. parameters/os/Ubuntu.yaml

  8. parameters/os/Ubuntu.yaml.jinja

  9. parameters/osfinger/Ubuntu-18.04.yaml

  10. parameters/osfinger/Ubuntu-18.04.yaml.jinja

  11. salt['config.get']('libvirt:lookup')

  12. salt['config.get']('libvirt')

  13. parameters/id/minion1.example.net.yaml

  14. parameters/id/minion1.example.net.yaml.jinja

Remember that the order is important, for example, the value of key1:subkey1 loaded from parameters/os_family/Debian.yaml is overridden by a value loaded from parameters/id/minion1.example.net.yaml.

2. For formula authors and contributors

2.1. Dependencies

map.jinja requires:

  • salt minion 2018.3.3 minimum to use the traverse jinja filter

  • to be located at the root of the formula named directory (e.g. libvirt-formula/libvirt/map.jinja)

  • the libsaltcli.jinja library, stored in the same directory, to disable the merge option of salt['config.get'^] over salt-ssh

  • the libmapstack.jinja library to load the configuration values

  • the libmatchers.jinja library used by libmapstack.jinja to parse compound like matchers

2.2. Use formula configuration values in sls

The map.jinja exports a unique mapdata variable which could be renamed during import.

Here is the best way to use it in an sls file:

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}

test-does-nothing-but-display-TEMPLATE-as-json:
  test.nop:
    - name: {{ TEMPLATE | json }}

2.3. Use formula configuration values in templates

When you need to process salt templates, you should avoid calling salt['config.get'^] (or salt['pillar.get'^] and salt['grains.get'^]) directly from the template. All the needed values should be available within the mapdata variable exported by map.jinja.

Here is an example based on template-formula/TEMPLATE/config/file.sls:

# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_package_install = tplroot ~ '.package.install' %}
{%- from tplroot ~ "/map.jinja" import mapdata as TEMPLATE with context %}
{%- from tplroot ~ "/libtofs.jinja" import files_switch with context %}

include:
  - {{ sls_package_install }}

TEMPLATE-config-file-file-managed:
  file.managed:
    - name: {{ TEMPLATE.config }}
    - source: {{ files_switch(['example.tmpl'],
                              lookup='TEMPLATE-config-file-file-managed'
                 )
              }}
    - mode: 644
    - user: root
    - group: {{ TEMPLATE.rootgroup }}
    - makedirs: True
    - template: jinja
    - require:
      - sls: {{ sls_package_install }}
    - context:
        TEMPLATE: {{ TEMPLATE | json }}

This sls file expose a TEMPLATE context variable to the jinja template which could be used like this:

########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################

This is another example file from SaltStack template-formula.

# This is here for testing purposes
{{ TEMPLATE | json }}

winner of the merge: {{ TEMPLATE['winner'] }}