Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: Salt

  • Salt Minion: ExecStart=/usr/bin/salt-minion (code=exited, status=1/FAILURE)

    Royalty-free cartoon styled clip art graphic of a salt shaker character This image is available as an EPS file for an extra $20 fee after purchasing the high resolution. In order to obtain the EPS you will need to contact customer service. You can obtain the EPS files for the whole collection by purchasing the large collection and paying an additional $100.

    Ran into this error today when trying to rebuild my CEPH cluster. After removing all CEPH packages I was working through the Red Hat CEPH 1.3 install guide and was running the command below.

    # salt '*' state.highstate

    The output of the command above indicated that one of the minions was not healthy.

    osd01.lab.localdomain:
    Minion did not return. [Not connected]

    So I logged into the problematic minion and attempted to start salt-minion manually.

    # systemctl start salt-minion

    Salt-minion failed to start and barfed out the errors below.

    salt-minion.service - The Salt Minion
    Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled)
    Active: failed (Result: exit-code) since Wed 2015-08-19 09:17:08 EDT; 35min ago
    Process: 841 ExecStart=/usr/bin/salt-minion (code=exited, status=1/FAILURE)
    Main PID: 841 (code=exited, status=1/FAILURE)
    CGroup: /system.slice/salt-minion.service
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: File "/usr/lib/python2.7/site-packages/salt/payload.py", line 204, in send_auto
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: return self.send(enc, load, tries, timeout)
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: File "/usr/lib/python2.7/site-packages/salt/payload.py", line 196, in send
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: return self.serial.loads(self.socket.recv())
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: File "/usr/lib/python2.7/site-packages/salt/payload.py", line 95, in loads
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: return msgpack.loads(msg, use_list=True)
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: File "msgpack/_unpacker.pyx", line 142, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:142)
    Aug 19 09:17:08 osd01.lab.localdomain salt-minion[841]: msgpack.exceptions.ExtraData: unpack(b) received extra data.
    Aug 19 09:17:08 osd01.lab.localdomain systemd[1]: salt-minion.service: main process exited, code=exited, status=1/FAILURE
    Aug 19 09:17:08 osd01.lab.localdomain systemd[1]: Unit salt-minion.service entered failed state.

    After a bit of googling I was able to figure out that the issue was related to the local salt cache files. so I took the scorched earth approach and removed them. Note that this was performed on the minion – osd01.

    # cd /var/cache/salt/
    # rm -Rf minion

    Now lets restart salt-minion

    # systemctl restart salt-minion

    And check its status.

    # systemctl status salt-minion
    salt-minion.service - The Salt Minion
    Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled)
    Active: active (running) since Wed 2015-08-19 09:59:06 EDT; 6s ago
    Main PID: 2501 (salt-minion)
    CGroup: /system.slice/salt-minion.service
    ├─2501 /usr/bin/python /usr/bin/salt-minion
    └─2729 /usr/bin/python /usr/bin/salt-minion
    Aug 19 09:59:06 osd01.lab.localdomain systemd[1]: Starting The Salt Minion...
    Aug 19 09:59:06 osd01.lab.localdomain systemd[1]: Started The Salt Minion.

    Excellent, issue resolved. Apparently other issues, such as mismatched salt rpms between the master and the minion can also cause this error, however that was not the case for me.