Pillar Data Doesn't Get Updated within Custom Execution Module

Question

I am writing a custom Salt execution module that runs some basic tests against state files that contain a YAML file with different tests and parameters. The module will read these YAML files, and set grains accordingly to the testing requirement and then run a state.show_sls against the state using the grain data provided. This works well until I need something from pillar. The grains get set, however the pillar data never gets updated according to the new grains set. We also run a masterless setup, so the minion does all the work.

# pillar/top.sls
base:
  'G@roles:bind and G@location:MyTown':
    - match: compound
    - bind.MyTown

Within my execution module I set:

#_/modules/my_execution_module.py
def all():
    tests = _get_tests()
    for test in tests:
        __grains__.update(test['grains'])
        __pillar__.update(__grains__)
        print(__salt__['grains.get']('roles'))      # Comes back with expected data.
        print(__salt__['grains.get']('location'))   # Comes back with expected data.
        print(__salt__['pillar.get']('bind_conf'))  # This comes back as blank.
        __salt__['state.show_sls'](test['state'])

Running the same test, but running all the salt commands individually to set the grains and print the grain/pillar information works as expected. It's only within this custom module that, while it sets the grains, does not update the pillar information.

salt-call.bat --versions
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.18
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)]
  python-gnupg: 0.4.7
        PyYAML: 5.4.1
         PyZMQ: 19.0.0
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.22621 SP0 Multiprocessor Free

OrangeDog · Accepted Answer · 2023-02-17 15:42:00Z

1

Both grains and pillar are cached and the refreshes are asynchronous.

Running this after changing the grains should hopefully work:

__salt__["saltutil.refresh_grains"](refresh_pillar=False)
__salt__["saltutil.refresh_pillar"](wait=True)

Trying to add the grains into the pillar (__pillar__.update(__grains__)) isn't going to work though. It will always be replaced with the latest compiled pillar.

Further information: https://docs.saltproject.io/en/3005/topics/pillar/#in-memory-pillar-data-vs-on-demand-pillar-data

It turns out that in a masterless setup the refresh never completes, because there's no master to do it.

A hack like this works, but you'll need a bit more in case the new pillar has some keys removed.

__pillar__.update(__salt__["pillar.items"]())

Consider whether you actually need pillar in a masterless setup, or whether this could be a runner or orchestration instead of an execution module.

edited Feb 17, 2023 at 15:42

answered Feb 17, 2023 at 9:48

OrangeDog

39.3k18 gold badges141 silver badges227 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

RogueOverride Over a year ago

Thanks @OrangeDog, thanks for the tip and documentation. However when adding both these lines after doing the grain update, it still doesn't appear to update the pillar that is in-memory for the module. I keep getting the following from the pillar refresh: [WARNING ] Pillar refresh did not complete within timeout 30 I even tried putting this timeout to 600 seconds, but it timed out with that as well.

OrangeDog Over a year ago

@RogueOverride I'm guessing this is because it's masterless. It doesn't make much sense to be using pillar without a master.

OrangeDog Over a year ago

Maybe a hack like __pillar__.update(__salt__["pillar.items"]()) would do it.

OrangeDog Over a year ago

Otherwise, an execution module is just not going to work for this. You may have better luck with a runner or orchestration.

RogueOverride Over a year ago

actually, your hack worked in this case, __pillar__.update(__salt__["pillar.items"]()). I did start to think about it being masterless and that pillars are kind of pointless on a masterless setup. However, we came from a master setup and converted over to a masterless. As of right now, pillars are there out of convenience and clarity to the team. Thank you for this hack, this works.

Collectives™ on Stack Overflow

Pillar Data Doesn't Get Updated within Custom Execution Module

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related