Autograd: connect the runtime link to the Fréchet-derivative model - #25
Conversation
92ab40f to
646290a
Compare
|
Thanks, Nicolas. This is a useful bridge, and the proofs compile cleanly. One wording change before we merge: could you make it explicit that the endpoint is the exact tape model instantiated over ℝ, not the native Float or CUDA execution path? Phrases such as "executable dense reverse pass" and "runtime reverse pass" currently read more broadly than the theorem states. The theorem itself looks good. |
…model
The runtime link (Runtime/Link/BackwardGraph) proves the executable dense
reverse pass agrees with the algebraic model's backpropAllCtx over any
commutative semiring; the analytic tape model (Tape/Core/FDeriv) proves
reverse-mode accumulation computes the adjoint of the Fréchet derivative
of the forward evaluation over the reals. The two developments were
stated on different graph types with no formal connection between them.
New module NN/Proofs/Autograd/Runtime/Link/FDeriv.lean supplies it:
- Algebra.{Node,Graph}.toReal specialize the algebraic model at the real
carrier and a fixed environment to the analytic model, and
{Node,Graph}.toAlgebra embed the analytic model back as the Unit-
environment slice; the round trip is the identity, so the analytic
model is exactly the environment-free real slice of the algebraic one.
Commutation lemmas cover all three semantics (eval, jvpCtx,
backpropCtx).
- TList.takeLeft plus takeLeft_backpropAllCtx (Graph and GraphData
forms) identify the input-prefix block of the full backpropagation
with the inputs-only backpropCtx.
- flattenCtx transport lemmas commute context vectorization with
cast/snoc/unsnoc/add, exhibiting evalVec/jvpVec/backpropVec as the
flattenings of the TList semantics.
- Composed endpoints backpropCtx_eq_adjoint_fderiv and
backwardDenseFrom_compileAux_adjoint_fderiv, with pointwise _at
variants for graphs containing non-smooth primitives: the executable
dense reverse pass on a compiled graph returns the full
backpropagation context, whose input prefix is (fderiv eval x)† seed.
The round-trip lemmas double as the correctness specification for a
possible follow-up that re-founds the analytic model as an abbreviation
of the algebraic one (as already done for TList).
All new declarations depend only on [propext, Classical.choice,
Quot.sound].
The endpoint theorems are statements about Tape.backwardDenseFrom instantiated at the exact carrier ℝ, not about the native Float evaluation or the CUDA execution path. Rephrase "executable dense reverse pass" / "runtime reverse pass" accordingly and add an explicit scope paragraph to the module docstring. No code changes.
646290a to
59f2920
Compare
|
Thanks, Nicolas — the revised scope is clear now. This addresses my wording concern. I’ll wait for the current CI run to finish before merging. |
|
Thanks, Nicolas! This is merged now. The exact tape connection to Fréchet derivatives is a valuable addition, and I’m doing a final local pass to tighten the documentation around the two-way correspondence. |
Why
The autograd proof stack currently has two disconnected halves:
NN/Proofs/Autograd/Runtime/Link/*proves the tape engine's dense reverse pass (Tape.backwardDenseFrom) agrees with the algebraic model'sbackpropAllCtx, over any commutative semiring carrier and non-differentiable environmentΔ.NN/Proofs/Autograd/Tape/Core/FDeriv.leanproves reverse-mode accumulation computes the adjoint of the Fréchet derivative of the forward evaluation, overℝ.They are stated on different graph types with no translation function and no lemma relating them, so neither result reaches the other's conclusion: the link theorem never mentions real derivatives, and the analytic theorem never touches
Tape.backwardDenseFrom. Anyone using the exact tape model's gradients for sensitivity analysis or verification purposes has to trust the algebraic model's hand-written VJP fields rather than a calculus-level statement.What
One new self-contained, additive-only module,
NN/Proofs/Autograd/Runtime/Link/FDeriv.lean(plus itsLink.leanaggregator import):Algebra.{Node,Graph}.toRealspecialize the algebraic model atα := ℝand a fixed environment to the analytic model;{Node,Graph}.toAlgebraembed the analytic model back as theΔ := Unitslice; the round trip is the identity, so the analytic model is exactly the environment-free real slice of the algebraic one. Commutation lemmas cover all three semantics (eval,jvpCtx,backpropCtx).TList.takeLeftandtakeLeft_backpropAllCtx(GraphandGraphDataforms) identify the input block of the full backpropagation with the inputs-onlybackpropCtx— the missing lemma relating the two backpropagation forms the codebase keeps side by side.flattenCtxcommutes withcast/snoc/unsnoc/add, exhibitingevalVec/jvpVec/backpropVecas exactly the flattenings of theTListsemantics.backpropCtx_eq_adjoint_fderivandbackwardDenseFrom_compileAux_adjoint_fderiv(with pointwise_atvariants for graphs containing non-smooth primitives): the tape model's dense reverse pass on a compiled graph, instantiated atα := ℝ, succeeds and returns the full backpropagation context, whose input prefix is exactly(fderiv ℝ eval x)† seed.Scope. Every statement is about the exact tape model —
Tape.backwardDenseFrominstantiated at the exact carrierα := ℝ. Nothing here is a claim about the nativeFloatevaluation or the CUDA execution path; relating those to the exact model is a separate (approximation) concern.All new declarations depend only on
[propext, Classical.choice, Quot.sound].Follow-up enabled
The round-trip lemmas double as a ready-made migration specification if you later want to re-found the analytic model as an abbreviation of the algebraic one at
α := ℝ,Δ := Unit(as is already done forTList). Happy to take that on as a separate PR if wanted.