A popular activation function used in neural networks. It outputs the input directly if it is positive, and zero otherwise.
[ \text{ReLU}(x) = \max(0, x) ]
- Introduces non-linearity to the network.
- Helps mitigate the vanishing gradient problem.
- Computationally efficient.
A variant of ReLU that caps the output at 6, providing numerical stability and suitability for low-precision computations.
[ \text{ReLU6}(x) = \min(\max(0, x), 6) ]
- Used in mobile-friendly architectures like MobileNet.
- Prevents exploding activations and improves quantization.
A pooling technique that reduces each feature map to a single value by taking the spatial average.
For a feature map ( X ) of size ( H \times W ) and channel ( c ): [ GAP_c = \frac{1}{H \times W} \sum_{i=1}^{H} \sum_{j=1}^{W} X_{i,j,c} ]
- Reduces dimensions without introducing additional parameters.
- Helps prevent overfitting.
- Commonly used in architectures like ResNet and MobileNet.
An efficient convolutional operation that splits a standard convolution into two steps:
- Depthwise Convolution: Applies a single filter per input channel.
- Pointwise Convolution: Combines the outputs of the depthwise convolution using a ( 1 \times 1 ) kernel.
- Significantly reduces computational cost compared to standard convolutions.
- Used in MobileNet for lightweight model design.
A problem in deep neural networks where gradients become very small during backpropagation, making it difficult for the network to learn.
- Activation functions like sigmoid or tanh squash input into small ranges, leading to diminishing gradients.
- Use activation functions like ReLU or architectures like ResNet with skip connections.
A technique introduced in ResNet to mitigate the vanishing gradient problem. It allows gradients to flow directly through the network by skipping certain layers.
For an input ( x ) to a layer: [ y = F(x) + x ] Where ( F(x) ) represents the transformation applied by the skipped layer.
- Enables training of very deep networks.
- Helps in learning identity mappings.
A hyperparameter used in MobileNet to control the number of filters in each layer.
[ \text{Number of Filters in Layer} = \alpha \times \text{Original Filters} ]
- Reduces model size and computational cost.
- Common values: ( \alpha = 0.75, 0.5 ), etc.
A hyperparameter in MobileNet that adjusts the input image resolution.
[ \text{Input Size} = \rho \times \text{Original Size} ]
- Trades accuracy for computational efficiency.
- Common values: ( \rho = 1.0, 0.5 ), etc.
A layer where every neuron is connected to every neuron in the previous layer.
For input ( x ) with weights ( W ) and bias ( b ): [ y = Wx + b ]
- Typically used for classification tasks.
- High parameter count can lead to overfitting.
An activation function commonly used in the output layer for multi-class classification. It converts logits into probabilities.
For an input vector ( z ): [ \text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}} ]
- Ensures outputs are in the range ( [0, 1] ) and sum to ( 1 ).
- Suitable for multi-class classification problems.
A regularization technique that randomly disables neurons during training to prevent overfitting.
During training, a neuron ( i ) is retained with probability ( p ): [ y_i = x_i \cdot \text{mask}_i ] Where ( \text{mask}_i \sim \text{Bernoulli}(p) ).
- Reduces overfitting by promoting redundancy in the network.
- Commonly used in AlexNet and VGG.
A technique to normalize the inputs to each layer, improving training stability and convergence speed.
For a mini-batch with mean ( \mu ) and variance ( \sigma^2 ): [ \hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} ] The normalized input is then scaled and shifted: [ y = \gamma \hat{x} + \beta ]
- Reduces internal covariate shift.
- Enables higher learning rates.