SensLI - #22
Open
ferdinandlouapre wants to merge 9 commits into
Open
Conversation
added 4 commits
June 10, 2026 12:09
TheoRudkiewicz
requested changes
Jun 18, 2026
TheoRudkiewicz
left a comment
TheoRudkiewicz
left a comment
Collaborator
There was a problem hiding this comment.
- It's missing a TLDR and possibly one or two intro sentence (including the paper motivation, goal, framework). Could be something like "SensLI grows network by inserting function preserving layers. They propose a criterion to choose where to add a layer justified by sensitivity analysis that boils down to choosing the layer with the biggest gradient.".
- We should keep uniform notations across the wiki.
- We can consider the notion of function preserving is already know and use it. Also no need to explain in details how to make a convolution an identity.
- If I understood correctly even in the ResNet case they initialize a lyer with identity. This is strange since one would expect the use of the resnet structure to allow the initialization of at least one fully random layer.
- I think the discussion about the choice of norm for convolution is worth been mentioned.
- About the experiments:
- SensLI fit better the training loss than fixed architecture or random (in the sense of "where") growing architecture but this is not translated to better test performance: SensLI seems to lead to overfitting. (Unfortunately they don't show test accuracy for the "LIother" strategy that minimize the gradient of inserted layers)
| It leads to the following initializations: | ||
|
|
||
| * Fully-connected layers: :math:`W = \text{Id}` and :math:`b=0` | ||
| * Residual blocks: :math:`W_2=0` and :math:`W_1, b` arbitrary |
Collaborator
There was a problem hiding this comment.
In practice it's not what they do they init W_1=0.8 * Id.
| | **Comparing SensLI with initial and final architecture (trained from the begining)** | ||
|
|
||
| - Adding 1 layer gives lower loss than base architecture but higher loss than training final architecture from the start | ||
| - Adding 3 layers (one at a time with fixed growing schedule) gives lower loss than final architecture trained from the start |
Collaborator
There was a problem hiding this comment.
Only train loss, test loss is unclear.
Comment on lines
+116
to
+120
| | **Comparison of where to insert a layer** | ||
| | The article compares the position of the inserted layer between the highest merit layer, the lowest merit layer and random selection | ||
|
|
||
| - For 1 added layer, the highest merit beats the two other for full batch GD but results are equivaluents for mini-batch GD (dominated by batch selection noise) | ||
| - For 3 layers added at constant interval, highest merit beats the other two for both full-batch and mini-batch GD |
Collaborator
There was a problem hiding this comment.
Again quiet unclear for test accuracies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.