target_modules can match a subset of layers with no warning

[submaroon]

Feature request

target_modules is resolved by module name suffix, and ValueError is raised only
when the whole list matched nothing. If a name matches on some layers and not on
others, the adapter is built quietly. The only visible sign is a trainable parameter
count, and people rarely hand verify that.

I would like the coverage to be visible. get_model_status() and
get_layer_status() already exist and look like the natural place: for each entry in
target_modules, how many modules it matched and on which layer indices. Then a user
can assert on it in their own code.

A warning would work too, something along the lines of target 'v_proj' matched 25 modules on layers [0-4, 6-10, ...], other targets matched 30, emitted only when
coverage is uneven across targets. I do not know how noisy that would be across the
model zoo, so the reporting version seems like the safer ask.

Motivation

This is not an edge case any more. google/gemma-4-26B-A4B has 30 text layers, and
five of them (5, 11, 17, 23, 29) are global attention layers where the model reuses
the key projection as the value projection. Gemma4TextConfig carries
attention_k_eq_v: true, and Gemma4TextAttention.__init__ reads:

self.use_alternative_attention = config.attention_k_eq_v and not self.is_sliding
self.v_proj = nn.Linear(...) if not self.use_alternative_attention else None

The checkpoint has no self_attn.v_proj.weight for those five layers. So
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"] gives v_proj on 25 layers
and the other three on 30. The adapter is asymmetric across depth and nothing says so.

Repro:

from collections import Counter
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-26B-A4B", torch_dtype="auto", device_map="auto")
model = get_peft_model(model, LoraConfig(
    task_type="CAUSAL_LM", r=32, lora_alpha=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]))

print(Counter(
    n.split(".lora_")[0].rsplit(".", 1)[-1]
    for n, p in model.named_parameters()
    if p.requires_grad and "lora_A" in n
))
# v_proj: 25, q_proj: 30, k_proj: 30, o_proj: 30

Real configs hit this. Published Gemma 4 adapters use regexes like
(mlp|self_attn)\.(up|down|gate|q|k|v|o)_proj that treat v uniformly across depth
(the axolotl config in the zerofata/G4-MeroMero-26B-A4B model card is one public
example). Several third party Gemma 4 fine-tuning guides recommend the seven name
list with no caveat about layer coverage. target_modules="all-linear" is fine,
because it enumerates what exists rather than what you named.

The consequence is worse than a smaller adapter, at least on this architecture. On
those five layers k_proj is the value matrix, so an adapter you believe is queries
and keys only is editing the value path, and an adapter you believe is values and
output only cannot reach values there at all. Anyone comparing QK against VO
adaptation on Gemma 3/4, Qwen3 or OLMo is comparing something other than what they
configured.

None of this is PEFT's fault. But PEFT is the layer that knows what matched, and it
is the only place a fix does not have to be repeated in every model card.

Contribution

Happy to open a PR against get_model_status() / get_layer_status() if the
direction is acceptable. Details and the forward pass walkthrough:
https://huggingface.co/SubMaroon/gemma4-lora-traps

[LOGO127]

Hi maintainers — I’d like to work on this issue if the direction is welcome. My proposed scope is to expose per-target coverage through the existing get_model_status() / get_layer_status() reporting path, without changing target-module matching semantics, plus a small synthetic-model regression test for uneven suffix matches. I’ll avoid opening a duplicate PR, include the relevant tests, and disclose AI assistance in any PR description. Could a maintainer confirm that this scope is acceptable before I start?

[BenjaminBossan]

Thanks for the suggestion @submaroon. This general problem has been a thorn in our side for some time, but finding the right solution is tricky. For the specific problem you mention with Gemma, I'm not sure how big of a practical issue this really is. It's not like targeting q_proj and v_proj is the best configuration, it's more the standard that was established in the original paper. I would expect that in practical terms, the difference will be small and total number of trainable parameters is more relevant.

I would like the coverage to be visible. get_model_status() and
get_layer_status() already exist and look like the natural place: for each entry in
target_modules, how many modules it matched and on which layer indices. Then a user
can assert on it in their own code.

I'm still lacking the fantasy what this would look like. Could you please provide an example of you expect to see there? Also, my impression is that these functions are not often used, even if they can be very helpful when diagnosing this type of issue.

I think a more interesting problem to solve would be the following: If I pass target_modules=["q_proj", "v_proj", "foobar_proj"], I don't get any indication that the latter didn't match anything. Getting, say, a warning there would be the better solution IMO. It still doesn't solve the issue with passing regexes, where one part of the regex might match and other part doesn't. But that just might be unsolvable.

Anyway, please don't open a PR yet, we first need to get the shape of the solution right.

@LOGO127 Please don't open a PR, the poster already offered to do that.

[submaroon]

For an ordinary fine-tune the size difference is minor, yes.

What I actually ran into sits on those five layers. Gemma 4 reuses the key projection as the value projection there, so k_proj is the value matrix.
target_modules=["q_proj", "k_proj"] ends up editing the value path on 5 of the 30 layers, and ["v_proj", "o_proj"] cannot reach values on those same 5. Those five are also the only global attention layers in the model — the other 25 are windowed at 1024 tokens.

That’s where it stopped being “just a smaller adapter” for me. I was comparing QK against OV adaptation to see which one carried style, and on those five layers neither arm was what I had configured, so I couldn’t tell what I had actually measured.

Mixed layer types show up elsewhere too — sliding + global attention, dense + sparse blocks, cross-layer KV sharing. So I expect entries in target_modules to keep being true for some layers and false for others.

On what I was picturing: layer indices were the wrong ask. PEFT has no general notion of a layer and would have to start parsing names for one. Module names already carry enough.

A count per entry on TunerModelStatus:

>>> get_model_status(model).target_module_matches
{'q_proj': 30, 'k_proj': 30, 'v_proj': 25, 'o_proj': 30}

and the entry that wrapped a given module, on TunerLayerStatus:

>>> get_layer_status(model)[0].matched_by
'q_proj'

The second is what makes the first cheap to compute, and it leaves the grouping to the user, since only they know how their model is laid out.

From my side the check would be one line:

assert get_model_status(model).target_module_matches == {
    "q_proj": 30, "k_proj": 30, "v_proj": 30, "o_proj": 30
}

which fails with 'v_proj': 25.

For regexes it degrades to a total rather than a per-part answer, which is still something:

>>> # target_modules = r'.*\.(q|k|v|o)_proj$'
>>> get_model_status(model).target_module_matches
{'.*\\.(q|k|v|o)_proj$': 115}

Someone expecting 4 × 30 sees 115. That is exactly the shape I hit in the wild — published Gemma 4 adapters use patterns like (mlp|self_attn)\.(up|down|gate|q|k|v|o)_proj, which treat v uniformly across depth.

The zero-match warning would catch the more common mistake. It would not have caught mine: v_proj matched 25 modules and never reaches zero.

On the status functions being little used — I only went looking at them after a run had already come out wrong. That’s probably when most people would.

Holding off on a PR until the shape is settled.

Thank you!

[BenjaminBossan]

@submaroon Thanks for expanding further. I think it's best to see what this change in get_layer_status and get_model_status entails based on a (draft) PR. So if you have something ready or a plan how to implement this, please go ahead. Let's leave the other issue I mentioned, part of target_modules not matching at all, out of scope for this issue.