Restructure the deep-networks documentation - #405
Draft
LegrandNico wants to merge 10 commits into
Draft
Conversation
LegrandNico
force-pushed
the
feat/transformer-parity
branch
from
July 8, 2026 21:51
fbc682b to
28b81b3
Compare
LegrandNico
force-pushed
the
deep-networks-docs
branch
from
July 8, 2026 21:51
af6c969 to
005fc42
Compare
LegrandNico
force-pushed
the
feat/transformer-parity
branch
from
July 9, 2026 19:20
28b81b3 to
69fb470
Compare
LegrandNico
force-pushed
the
deep-networks-docs
branch
from
July 9, 2026 19:20
005fc42 to
623c611
Compare
LegrandNico
force-pushed
the
feat/transformer-parity
branch
from
July 9, 2026 20:21
69fb470 to
366c260
Compare
LegrandNico
force-pushed
the
deep-networks-docs
branch
from
July 9, 2026 20:21
623c611 to
b67de82
Compare
LegrandNico
force-pushed
the
feat/transformer-parity
branch
from
July 9, 2026 20:28
366c260 to
30f7c3c
Compare
LegrandNico
force-pushed
the
deep-networks-docs
branch
from
July 9, 2026 20:45
b67de82 to
ffbab25
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
LegrandNico
force-pushed
the
deep-networks-docs
branch
2 times, most recently
from
July 10, 2026 10:49
4972933 to
af1c134
Compare
LegrandNico
marked this pull request as draft
July 10, 2026 12:11
LegrandNico
force-pushed
the
deep-networks-docs
branch
4 times, most recently
from
July 23, 2026 19:09
f3c5f4e to
58f6cc5
Compare
The narrative and protocol follow the default build of the vectorised backend: f32 through the pure-Rust matrix kernels, with the `f64` and `blas` cargo features as the opt-ins. JAX runs in its default float32 (the x64 flag is gone), data arrays are created in float32 so no timed call pays a dtype conversion, the parity checks use float32 accumulation bounds, and the closed-form state size counts 4 bytes per value. Executed outputs are included (nb_execution_mode = "cache"). Anticipates the optim branch; until that branch lands, the engine on this branch computes in f64, which the notebook's tolerance bounds also accommodate.
The memory narrative follows the engine: the batched state lives in a workspace reused across batch_update calls, so the Rust step increment stays small after warm-up (2.4 MiB at batch 4096, against 28 MiB of batched state) instead of growing with the batch. Outputs re-executed.
A deep network run in the framework's native adaptive-precision regime rather than the pinned one: with the volatility levels off, the input layer's precision is the network's uncertainty source, and the notebook measures what it controls. Four results, each checked in the notebook: the closed-form propagation of input uncertainty into the hidden layer's expected precision, the bitwise precision-independence of the forward means, the learning-rate-like modulation of the precision-weighted updates under SGD (with its stability boundary), and the evidence-driven accumulation of the interior precisions across batch updates (faster for trusted inputs, stalling once prediction errors vanish, unbounded under persistent input noise).
A deep linear teacher-student task with a graded singular spectrum turns one training run into a sequence of measurable saddle escapes; a small initialisation stretches the plateaus. Both learners are the same network under the same single-sweep updates: the precision_weighted rule (backpropagation node for node) against the standard rule on moved beliefs with loose interior priors (single-sweep prospective configuration). Measured: the backpropagation staircase with sequential mode learning; the interior prior precision shifting every transition earlier at a matched learning rate; the best-vs-best comparison in which predictive coding leaves the first saddle three times sooner but falls behind by the fourth mode as the growing bottom-up precision quenches the amplification; and the origin-escape time growing three times more steeply with shrinking initialisation for backpropagation than for predictive coding.
The benchmark now measures the two entry points a training loop exercises (the batched forward pass and one learning step) for the three learners, and presents them in a single figure: execution time on the left, total peak resident memory on the right, colour for the learner and line style for the entry point. The memory harness gained a forward-pass mode so both entries appear in both panels. The sequential fit timing, the per-entry bar chart, the analytical memory cross-checks, and the narrative tied to them are removed; the parity check and the protocol notes remain.
The Rust engine is measured under its blas feature (matrix products through the system BLAS), stated in the configuration note alongside the trade-off of the default pure-Rust build. The figure becomes a two-by-two grid, forward pass and learning step as columns with execution time and total peak resident memory as rows sharing each row's scale, and the learners are coloured green for backpropagation, blue for the JAX predictive coding backend, and a lighter blue for the Rust engine.
The measured unit becomes five epochs of minibatch learning steps over 4096 samples with one validation forward pass per epoch, swept over the batch size: wall clock time of the loop (median of three, after a warm-up loop that pays the one-off JAX compilation) and the peak resident memory of one cold loop in a fresh subprocess, compilation included. The figure is one row, speed on the left and memory on the right. Since the loop visits every sample once per epoch at any batch size, the arithmetic is constant along the sweep and the slopes price the per-call overhead alone, which the reading section now explains.
LegrandNico
force-pushed
the
deep-networks-docs
branch
from
July 25, 2026 11:15
58f6cc5 to
b2f6d83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the single deep-learning notebook with a graded sequence: the theory of deep predictive coding networks (0.5), the implementation walkthrough (0.6), the deep-learning tutorial (0.7, the former 0.5 notebook), and the Transformer port (0.8).