Full GPU Rewrite, Performance Boost + misc#7
Conversation
Benchmark <0.75GB VRAM usage
Benchmark <0.75GB VRAM usage
Similarity checker
|
Hey @RobViren Not going to push this on the main just yet, wanted to get your eyes on it and hopefully you get some time to try it out. 3x FASTER totally on GPU, and tiny footprint |
|
Oh dang! You've been at work on this. I had not heard of speechbrain. Is it still avoiding over fitting and sounding like a demon? Kudos on the better GPU usage, super impressive. Gonna run tonight |
|
Haha, yeah I've noticed that you really need a voice that's already pretty close to get decent results. Resemblyzer will rate things as super similar when in reality they aren't. SB by contrast is much more critical, even rating target audios by the same speaker from the same recording as only a partial match, though technically their cos sim threshold is only 0.25, so that's why I think benchmarking a speaker against themselves is the way to go. Also, working with kokolab before I have a huge voice library and a toolbox of different voice model "surgery" methods that I think would fit nicely into a methodology here so it's like surgery->randomwalk->surgery, repeat until it's pretty close. another idea, there's porting kokoro to tpu as it's by far the biggest time sink, so it would be possible to get like another 5x or more speed boost to search the latent space faster, plus do batch evals for the best directional heading for mutations. |
# Conflicts: # utilities/fitness_scorer.py # utilities/initial_selector.py # utilities/kvoicewalk.py
|
Yeah. I really think maintaining a map of the results would really help to
guide other walks. Still believe genetic algo is the way to go. Only even
remotely feasible because of the small size. TPU would be great. It is just
monkeys on a keyboard trying to clone a voice.
…On Mon, Jun 16, 2025, 7:15 PM Robert Agee ***@***.***> wrote:
*RobertAgee* left a comment (RobViren/kvoicewalk#7)
<#7 (comment)>
Haha, yeah I've noticed that you really need a voice that's already pretty
close to get decent results. Resemblyzer will rate things as super similar
when in reality they aren't. SB by contrast is much more critical, even
rating target audios by the same speaker from the same recording as only a
partial match, though technically their cos sim threshold is only 0.25, so
that's why I think benchmarking a speaker against themselves is the way to
go.
Also, working with kokolab before I have a huge voice library and a
toolbox of different voice model "surgery" methods that I think would fit
nicely into a methodology here so it's like surgery->randomwalk->surgery,
repeat until it's pretty close.
another idea, there's porting kokoro to tpu as it's by far the biggest
time sink, so it would be possible to get like another 5x or more speed
boost to search the latent space faster, plus do batch evals for the best
directional heading for mutations.
—
Reply to this email directly, view it on GitHub
<#7 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEDLIDD7T64HWR2POUQX2DL3D5M3ZAVCNFSM6AAAAAB7OTOX3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZYGUZTEMBYGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Agreed! I have some ideas for making a 'smart' randomwalk wherein it can do 3 things (together or as separate explorative modes). A: Batch process and compare like 10-20 samples at once. Rank them in scoring and along the axis of improved score, continue randomizing into those select nodes and/or negative blend with the worst voices. Just like moving away from a voice in kokovoicelab. Continue until voice score begins to degrade then go back to scanning direction in batch comparison, rinse repeat. Like picking up on a signal but not sure where it's coming from. Just keep going until it gets fainter the reassess next direction.. B: As the voice gets closer to the target audio similarity, decrease the size of randomization allowed. When far away, move at lightspeed, when close by go to impulse thrusters. C: Feature-iterative randomwalk - target feature order of human recognition importance: 1. Pitch, 2. Prosody, etc... whatever that order might be, but maximize feature similarity for one feature at a time, then move to maximize next without retrograding the previous features. A round about approach of using one planet's gravity to slow you down without leaving the solar system, so to speak. C2: Feature-focused randomwalk - target single feature and maximize for similarity. Create a matching voice for each feature. Then cobble them together like frankenstein's monster. ====== by the way, I just restarted my wsl instance completely and on a fresh instance, it's clocking a 7-8x speed up (as opposed 3x I'd thought). The audio is a little shorter, but fingers crossed the actual performance gains are higher than expected. If you get benchmarks on your system, please share! |
|
Oh, and I should add too, taking the worst performers vs target_audio during the top_performers method, and use them to push the starting voice in the right direction strongly. So even if there's no great matching voice (e.g. no deep masculine voices in KokoroTTS) you can still use really "off" voices to your advantage. Gravel - https://voca.ro/17dMqrKJIXLR Here's a map to see where they live in Kokoro latent space (right hand side):
|
|
Why can't you do the gradient decent on the cross entropy, optimizing the voice embedding? Similarly to how we fine tune other models but optimize the weights/ |
Depends on what you're trying to do (ie voice cloning vs voice crafting), and many different hammers can functionally do the same task. |
|
Every score for comparison between starting models within this main branch seems to be the same. |
I know it's been a while for you but I've just discovered this and, congrats, this is really a beautiful and super-useful piece of software! I ran the process on 20+ target samples the past 2 days, getting some hits and some misses, with self sim results between 0.80 and 0.90 with some lucky ones above that. I then started playing around with gradient-less methods instead of Random Walk. I ended up doing a couple of tests using BOBYQA (mostly because I'd been using it in the past, but it turns out it isn't a great choice for higher-dim noisy exploration spaces like what we have here). But I eventually landed on the CMA-ES optimizer, which has been able to land me in the 0.93-0.97 Self Sim range within 3000 steps for almost all of my 26 tests of very different voice targets, and this being much less dependent on the initial starting sample, as multiple samples tend to converge to pretty similar results. There were still some hard cases that weren't able to push much past 0.8, but clearly the results are good. Just mentioning this because your idea of an adaptive random walk is good but there are already strategies that you can plug and play that can do a lot of heavy lifting. One bit of research that was interesting and allowed me to get consistently past the 0.90 threshold: Sadly, the code I've written is mostly claude-coded and my coding capabilities in python are far lower than my knowledge in ML, so I don't know to what extent you might want to integrate any of my code in your repo, but even if it never lands in the code itself (and for anyone who wants to instruct their AI agent to test something out), just know that using the CMA-ES optimizer instead of random walk is much faster at converging and converges much better than random walk on basically any test I ran. Hope this is of some use in any case! A direct comparison on the same samples :
And the runs I'd done for all the voices (mostly to see the difference of seeds on results):
Starting seeds were chosen by running similarity measure on the sample vs default sample generation, and picking the top 2 
|
5 participants
Full GPU Implementation and Optimization
Settings Configuration, Debug, Memory, Process Times logging, misc
Stuff ToDo: