Add Astra (Activation-Space Tail-Eigenvector LoRA) initialization

[LyoAI]

Feature request

Description

I would like to propose adding Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation) as a LoRA
initialization method to PEFT. Astra is published at ACL 2026 Findings ([arXiv:2602.19111](https://arxiv.org/abs/2602.19111)) with a reference implementation at [LyoAI/Astra](https://github.com/LyoAI/Astra).

As with PiSSA/CorDA, the model output is unchanged at the start of training, but the trainable update is constrained to a task-aware activation subspace. In the paper, Astra outperforms LoRA/PiSSA/CorDA-style baselines on 16 NLU/NLG benchmarks (GLUE-style tasks, math reasoning, code generation, commonsense reasoning) at reduced rank budgets, and all results are reproducible from the public code.

Proposed code

from peft import LoraConfig, get_peft_model
from peft.tuners.lora.astra import preprocess_astra
from peft.tuners.lora.config import AstraConfig

lora_config = LoraConfig(
    init_lora_weights="astra",
    r=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    astra_config=AstraConfig(astra_method="ipm"),  # or "kpm"
)
preprocess_astra(model, lora_config, run_model=run_model)  # collects covariance + eigendecomposition
model = get_peft_model(model, lora_config)

Implementation outline

AstraConfig sub-config (cache_file, covariance_file, astra_method, verbose,
use_float16_for_covariance, prune_temporary_fields) + LoraConfig(astra_config=...).

• preprocess_astra() in src/peft/tuners/lora/astra.py with forward hooks on module outputs.
• LoraLayer.astra_init() and an init_lora_weights="astra" branch in update_layer().
• Tests in tests/test_initialization.py (identity transform, cache/covariance reuse, hook cleanup, temporary field
  pruning, IPM/KPM) and tests/test_config.py (nested config roundtrip).
• Docs entry + examples/astra_finetuning/ example.

Motivation

PEFT already ships several data- and weight-driven initializations (PiSSA, CorDA, EVA, LoRA-GA, ...). Astra
complements them with an output-activation-eigenspace initialization that is cheap to compute (one eigendecomposition
per target layer, no SVD of the weight, no covariance inverse) and, per the paper, gives better convergence and
final quality at the same or lower rank. The integration is small and follows the existing CorDA pattern almost
one-to-one, so maintenance cost should be low.

Additional context

• Paper: https://arxiv.org/abs/2602.19111 (ACL 2026 Findings)
• Code: https://github.com/LyoAI/Astra
• The reference implementation currently pins peft==0.14.0 and uses a custom AstraLayer; the proposed integration
  reuses the stock PEFT LoRA layers and needs no custom layer class.

Your contribution

I'm one of the Astra authors and would be happy to submit the PR myself. I'll open the PR as soon as the approach is approved, and I'm also willing to help maintain this integration afterwards.