Data storage and caching

The Salt mine

The Salt Mine is used to collect arbitrary data from Minions and store it on the Master. This data is then made available to all Minions via the salt.modules.mine module. Mine data is gathered on the Minion and sent back to the Master where only the most recent data is maintained (if long term data is required use returners or the external job cache).

To enable the Salt Mine the mine_functions option needs to be applied to a Minion. This option can be applied via the Minion’s configuration file, or the Minion’s Pillar. The mine_functions option dictates what functions are being executed and allows for arguments to be passed in. If no arguments are passed, an empty list must be added:

mine_functions:
  test.ping: []
  network.ip_addrs:
    interface: eth0
    cidr: '10.0.0.0/8'

Mine vs grains

Mine data is designed to be much more up-to-date than grain data. Grains are refreshed on a very limited basis and are largely static data. Mines are designed to replace slow peer publishing calls when Minions need data from other Minions. Rather than having a Minion reach out to all the other Minions for a piece of data, the Salt Mine, running on the Master, can collect it from all the Minions every Mine Interval, resulting in almost fresh data at any given time, with much less overhead.

Minion data cache

The Minion data cache contains the Salt Mine data, minion grains and minion pillar information cached on the Salt Master. By default, Salt uses the localfs cache module to save the data in a msgpack file on the Salt Master.

Pluggable cache modules

While the default Minion data cache is the localfs cache, other external data stores can also be used to store this data such as the consul, etcd or redis module. To configure a Salt Master to use a different data store, the cache setting needs to be established:

cache: etcd

The pluggable data cache streamlines using various Salt topologies such as a Multi-Master or Salt Syndics configuration by allowing the data stored on the Salt Master about a Salt Minion to be available to other Salt Syndics or Salt Masters that a Salt Minion is connected to.

Master job cache

The Salt master maintains a job cache of all job executions which can be queried via the jobs runner. This job cache is called the Default Job Cache.

Managing the Job cache

A number of options are available when configuring the job cache. The default caching system uses local storage on the Salt Master and can be found in the job cache directory (on Linux systems this is typically /var/cache/salt/master/jobs). The default caching system is suitable for most deployments as it does not typically require any further configuration or management.

The default job cache is a temporary cache and jobs will be stored for 24 hours. If the default cache needs to store jobs for a different period the time can be easily adjusted by changing the keep_jobs parameter in the Salt Master configuration file. The value passed in is measured via hours:

keep_jobs: 24

Storing jobs in an external job cache

After a job executes, job results are returned to the Salt Master by each Salt Minion. These results are stored in the Default Job Cache.

In addition to the Default Job Cache, Salt provides two additional mechanisms to send job results to other systems (databases, local syslog, and others):

Note

Read more at Salt documentation: External cache.

Lab: Get data from Mine

One way to use data from Salt Mine is in a State. The values can be retrieved via Jinja and used in the SLS file. The following lab will setup HAProxy configuration file and pulls IP addresses from all Minions with the “web” grain to add them to the pool of load balanced servers.

Edit /etc/salt/minion.d/minion.conf:

mine_interval: 5

Edit /srv/salt/haproxy.sls:

haproxy_config:
  file.managed:
    - name: /etc/haproxy/config
    - source: salt://haproxy_config
    - template: jinja

Edit /srv/salt/haproxy_config:

{% for server, addrs in salt['mine.get']('roles:web', 'network.ip_addrs', expr_form='pillar').items() %}
server {{ server }} {{ addrs[0] }}:80 check
{% endfor %}