LoRA Traning Question

[SnaxxNZ]

Finally managed to get this working on Windows, but I have a question about the time required.

I have a 3600x CPU, 4070 GPU and 64GB of System Memory.

I have set up a dataset of 45 songs, but trying to pre-process the files takes about 2.5 hours per song, meaning the full dataset will take 112+ hours to pre-process.

Training is about 60-120 seconds per step as well (I used 2 songs for the test; not sure if it's longer with more data). No idea how many steps are needed, but if we stick to the 1000 default, thats going to add another day and a half.

Is it normal for it to take 6+ days to train a LoRA?

[1larity]

That's quite a lot of samples, so that's going to take a while anyway. VRAM is the key here, some 4070's have 8Gb of ram, some have 12, which is not ideal. You could have offloading to CPU (this should be evident in your console log) which will run dramatically slower than pure GPU processing. I've done some work to optimise this workflow in the tensor pre-processing, Ang got some good gains, but I have identified a couple of further areas, specifically in LoRA that could be improved.

If it's any consolation, I'm doing 10 songs on a 3070, and that's looking to take 340+ hours :) At least it's keeping my office warm.

I'm no LoRA expert, I'm assuming that 1000 epochs is appropriate, but you can go lower depending on your threshold for quality. Once you have got past your 'save every n epochs' limit, you can theoretically stop it and call it a day, also check you have got output in your LoRA output directory.

In theory you can resume training from a saved checkpoint, but currently the UI does not (yet) have the controls to let you do this.

[SnaxxNZ]

It's a 12GB 4070.

The biggest time sink is the pre-processing. It took forever to figure out how to even get it running (Windows). Pretty sure it's running on the CPU though, and using my system memory as that shoots up to about 50+GB in use.
The GPU doesn't appear to be doing any work during pre-processing.

[1larity]

You're lucky you did it after my recent optimisations, or it would take ~200x longer :)

Feel free to put in a issue: feature request to add the UI controls to invoke LoRA resume_from checkpoint path in train_from_preprocessed. :)

[Kwisss]

The trainer so far has been a farce, the user should not be able to run training settings that can't work.
Imagine how much compute already went out the window? shift on the base model?
who makes these things up?

[1larity]

It's a large project, and hasn't been been out in the wild very long, in the last week it's picked up ~10 new contributors in the last week alone, which is a 30% increase. Just looking at the commits, anyone can see the majority of work has been done quite recently basically ironing out the wrinkles from an MVP based on a handful of huge monolithic files. Its clear they have a good model architecture (in itself an expensive, time consuming and intellectually challenging bit of work) with just enough UI and wiring to make it vaguely usable. We can build on that and make it a smooth, cool user experience.

Lets give it some time and TLC and I'm sure it will become something great. Just having something available FOSS in this niche is a big win in itself.