Lmod is installed explicitly on workers; environment-modules 4.x is pulled in on the mgmt node when gcc-toolset-9 is explicitly installed.
Is GCC 9 needed? If not, then is it better to just dnf group install "Development Tools" via Ansible on the mgmt node and in the Packer recipe and explicitly install and use one of Lmod or environment-modules everywhere?