In continuing to try to get Kubeflow v1 and v2 working, the ephemeral storage support is REALLY spotty. Lots of weird, hard to track down bugs. My proposal - we have instructions how to add a storage bucket and mount that storage bucket into kubeflow on all the major clouds, and then use that storage rather than relying on the ephemeral storage provided. I don't think this is insanely challenging, and will eliminate a large class of bugs (I suspect).
Thoughts? cc @lukemarsden @Bubblyworld