Skip to content

[Bug]: Regression: 3006.16 master exhausted all inodes #68351

@OrangeDog

Description

@OrangeDog

What happened?

Upgraded from 3006.14 -> 3006.16.
Within 24 hours all inodes on the partition had been used, rendering the entire Salt deployment unusable.

# du -x --inodes /var | sort -n | tail
1363    /var/cache/salt/master/jobs/86
1381    /var/cache/salt/master/jobs/7a
1399    /var/cache/salt/master/jobs/ad
1514    /var/cache/salt/master/gitfs
1675    /var/cache/salt/master/files
2465    /var/lib/dpkg/info
2535    /var/lib/dpkg
3269    /var/lib
295877  /var/cache/salt/master/jobs
300575  /var/cache/salt/master
301081  /var/cache/salt
301542  /var/cache
305204  /var
Image

Deployment has only twelve minions, which run highstate (with a custom returner) every fifteen minutes.
There is no non-default job config on the master.
Nothing is logged between startup and the first "No space left on device" error.

Forced to downgrade back to 3006.14.

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output

Salt Version:
          Salt: 3006.16

Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.17.1
      cherrypy: 18.10.0
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.9.1
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.18.2
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 24.04.3 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-83-generic
        system: Linux
       version: Ubuntu 24.04.3 noble

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugbroken, incorrect, or confusing behaviorneeds-triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions