Replace ingress-nginx with supported ingress controller

[l0wl3vel]

Description

ingress-nginx is EoL.

Replace ingress-nginx with a maintained alternative for the remaining services after #309 is merged.

Requirements

• Easy to install (No istio gateway or cilium requiring complex setup, helm install with default values should do)
• TCP/UDP exposure is not required anymore
• Actively Maintained
• Preferably open source
• Usable in a metal-stack production setup

(Non-exhaustive) Options

• https://projectcontour.io/
• https://haproxy-ingress.github.io/

Definition-of-done

• Use Community SONiC 202605 #309 is merged
• mini-lab migrated to new ingress controller
• Document new Ingress Controller recommendation in https://github.com/metal-stack/website