nifi-formula

A salt formula to deploy Apache NiFi

Configure Apache NiFi in 3 different configurations:

Standalone
Cluster with embedded Zookeeper
Cluster with external Zookeeper

Configures systemd and sysctl limits automatically. See the pillar.example for detailed list of pillars.

1. General notes

See the full SaltStack Formulas installation and usage instructions.

If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.

If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning.

See Formula Versioning Section for more details.

If you need (non-default) configuration, please pay attention to the pillar.example file and/or Special notes section.

2. Contributing to this repo

Commit message formatting is significant!!

Please see How to contribute for more details.

3. Special notes

3.1. Requirements

CentOS, Ubuntu, Debian
Cluster configurations require hosts setup with FQDN and DNS.

3.2. Sample Pillars

3.2.1. Pillars - Standalone Node

By Default all of the settings will configure a standalone NiFi Node

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  nifi:
    cluster.is.node: 'false'
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000

3.2.2. Pillars - 3 Node Cluster with Embedded Zookeeper

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000
  nifi:
    # cluster node properties (only configure for cluster nodes) #
    cluster.is.node: 'true'
    cluster.node.address: {{ grains['fqdn'] }}
    cluster.node.protocol.port: '1111'
    cluster.flow.election.max.wait.time: '1 mins'
    zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
    state.management.embedded.zookeeper.start: 'true'
    web.http.host: {{ grains['fqdn'] }}
  zookeeperproperties:
    # To configure Zookeeper.properties set 'state.management.embedded.zookeeper.start' to 'true' above, and then define your Embedded Zookeeper servers here.
    customservers:
      Node1:
        hostname: nifi-1.localdomain
        zookeeper_myid: 1
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'
      Node2:
        hostname: nifi-2.localdomain
        zookeeper_myid: 2
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'
      Node3:
        hostname: nifi-3.localdomain
        zookeeper_myid: 3
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'

3.2.3. Pillars - 3 Node Cluster with External Zookeeper Servers

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000
  nifi:
    # cluster node properties (only configure for cluster nodes) #
    cluster.is.node: 'true'
    cluster.node.address: {{ grains['fqdn'] }}
    cluster.node.protocol.port: '1111'
    cluster.flow.election.max.wait.time: '1 mins'
    zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
    state.management.embedded.zookeeper.start: 'true'
    web.http.host: {{ grains['fqdn'] }}

4. Available states

4.1. `nifi`

Meta-state (This is a state that includes other states).

This installs the nifi package, manages the nifi configuration file and then starts the associated nifi service.

4.2. `nifi.package`

This state will install the nifi package only. This downloads the tar.gz file from the downloadurl and deploys it to servers.

4.3. `nifi.config`

This state will configure the nifi service and has a dependency on nifi.install via include list.

4.4. `nifi.service`

This state will start the nifi service and has a dependency on nifi.config via include list.

5. Testing

Linux testing is done with kitchen-salt.

5.1. Requirements

Ruby
Docker

$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

5.2. `bin/kitchen converge`

Creates the docker instance and runs the nifi main state, ready for testing.

5.3. `bin/kitchen verify`

Runs the inspec tests on the actual instance.

5.4. `bin/kitchen destroy`

Removes the docker instance.

5.5. `bin/kitchen test`

Runs all of the stages above in one go: i.e. destroy + converge
verify + destroy.

5.6. `bin/kitchen login`

Gives you SSH access to the instance for manual testing.