1 Neural Network

1.1 Neural Networks

1.1.1 Overview & Motivation

A neural network is a parameterized function \(f_\theta: \mathbb{R}^n \to \mathbb{R}^m\) built by composing simple, differentiable transformations called layers. Each layer applies an affine map followed by a non-linear activation, and the whole composition is trained end-to-end by gradient-based optimization.

Neural networks are powerful because of the universal approximation theorem: a single hidden layer with enough neurons can approximate any continuous function on a compact set to arbitrary accuracy. In practice, depth (many layers) is more parameter-efficient than width for learning hierarchical features.

This library provides a minimal, statically-sized neural network framework designed for embedded inference and on-device training — no heap allocation, no dynamic shapes, full compile-time dimension checking.

1.1.2 Mathematical Theory

1.1.2.1 Forward Propagation

Given \(L\) layers, the network computes:

\[a_0 = x\] \[z_\ell = W_\ell \, a_{\ell-1} + b_\ell, \quad \ell = 1, \ldots, L\] \[a_\ell = f_\ell(z_\ell)\] \[\hat{y} = a_L\]

where \(W_\ell \in \mathbb{R}^{n_\ell \times n_{\ell-1}}\) are weights, \(b_\ell \in \mathbb{R}^{n_\ell}\) are biases, and \(f_\ell\) is the activation function for layer \(\ell\).

1.1.2.2 Loss Function

Training minimizes a scalar loss \(\mathcal{L}(\hat{y}, y)\) that measures how far the prediction \(\hat{y}\) is from the target \(y\). Common choices:

1.1.2.3 Backpropagation

Backpropagation efficiently computes \(\nabla_\theta \mathcal{L}\) via the chain rule, working from the output layer backward:

\[\delta_L = \nabla_{a_L}\mathcal{L} \odot f_L'(z_L)\] \[\delta_\ell = (W_{\ell+1}^T \delta_{\ell+1}) \odot f_\ell'(z_\ell)\]

The gradients with respect to parameters are:

\[\frac{\partial \mathcal{L}}{\partial W_\ell} = \delta_\ell \, a_{\ell-1}^T, \qquad \frac{\partial \mathcal{L}}{\partial b_\ell} = \delta_\ell\]

1.1.2.4 Parameter Update

An optimizer uses the gradients to update the parameter vector \(\theta\):

\[\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}\]

where \(\eta\) is the learning rate. More sophisticated optimizers (momentum, Adam) modify this basic rule.

1.1.3 Complexity Analysis

where \(P = \sum_\ell (n_\ell \cdot n_{\ell-1} + n_\ell)\) is the total parameter count. For small embedded networks (\(P < 10{,}000\)), both passes complete in microseconds.

1.1.4 Step-by-Step Walkthrough

Network: 2 inputs → 2 hidden (ReLU) → 1 output (Sigmoid). Learning XOR.

Architecture:

graph LR
    x1((x₁)) --> h1((h₁))
    x1 --> h2((h₂))
    x2((x₂)) --> h1
    x2 --> h2
    h1 --> y((ŷ))
    h2 --> y

Epoch 0 — Forward pass with input \(x = [1, 0]^T\), target \(y = 1\):

Epoch 0 — Backward pass:

Update: \(W \leftarrow W - 0.1 \cdot \nabla W\). After ~500 epochs, the network correctly classifies all four XOR inputs.

1.1.5 Pitfalls & Edge Cases

• Vanishing gradients. Deep networks with Sigmoid or Tanh activations suffer exponential gradient decay. Prefer ReLU-family activations in hidden layers.
• Exploding gradients. Large weights amplify gradients exponentially. Use proper weight initialization (Xavier/He) and gradient clipping.
• Dead neurons. ReLU neurons that receive only negative inputs output zero forever. LeakyReLU mitigates this.
• Fixed-point saturation. In Q15/Q31, activations and gradients must stay within \([-1, 1)\). Scale inputs and learning rates accordingly.
• Learning rate sensitivity. Too high → divergence; too low → no progress. Start with \(\eta = 0.01\) and adjust.
• Overfitting. Small embedded datasets are easily memorized. Apply regularization (L2 weight decay).

1.1.6 Variants & Generalizations

1.1.7 Applications

• Function approximation — Learning arbitrary input-output mappings from data.
• Classification — Mapping inputs to discrete categories (fault detection, gesture recognition).
• Regression — Predicting continuous values (sensor calibration, system identification).
• Control — Neural network policies for model-free or model-predictive control on embedded targets.
• Signal processing — Learned filters replacing hand-designed FIR/IIR chains.

1.1.8 Connections to Other Algorithms

graph TD
    NN["Neural Network"]
    Layer["Dense Layer"]
    Act["Activation Functions"]
    Loss["Loss Functions"]
    Opt["Optimizer"]
    Reg["Regularization"]
    Model["Model"]
    LR["Linear Regression"]

    Layer --> NN
    Act --> NN
    Loss --> NN
    Opt --> NN
    Reg --> NN
    NN --> Model
    NN -.->|"single-layer, linear activation, MSE loss"| LR

1.2 Dense Layer

1.2.1 Overview & Motivation

A dense (fully-connected) layer is the most fundamental building block of a neural network. It maps an input vector \(a_{\text{in}} \in \mathbb{R}^n\) to an output vector \(a_{\text{out}} \in \mathbb{R}^m\) through a learnable affine transformation followed by a non-linear activation:

\[a_{\text{out}} = f(W \, a_{\text{in}} + b)\]

Every input neuron is connected to every output neuron — hence “fully connected.” The layer’s parameters are the weight matrix \(W\) and bias vector \(b\); training adjusts these to minimize the loss.

In this library, input size, output size, and parameter count are all compile-time constants, enabling stack allocation and dimension checking with zero runtime overhead.

1.2.2 Mathematical Theory

1.2.2.1 Forward Pass

\[z = W \, a_{\text{in}} + b, \qquad a_{\text{out}} = f(z)\]

where \(W \in \mathbb{R}^{m \times n}\), \(b \in \mathbb{R}^m\), and \(f\) is the activation function.

1.2.2.2 Backward Pass

Given the gradient of the loss with respect to the output \(\frac{\partial \mathcal{L}}{\partial a_{\text{out}}}\):

1. Pre-activation gradient: \[\delta = \frac{\partial \mathcal{L}}{\partial a_{\text{out}}} \odot f'(z)\]

2. Weight gradient: \[\frac{\partial \mathcal{L}}{\partial W} = \delta \, a_{\text{in}}^T\]

3. Bias gradient: \[\frac{\partial \mathcal{L}}{\partial b} = \delta\]

4. Input gradient (propagated to the previous layer): \[\frac{\partial \mathcal{L}}{\partial a_{\text{in}}} = W^T \delta\]

1.2.2.3 Parameter Count

\[P = m \times n + m = m(n + 1)\]

For a layer with 128 inputs and 64 outputs: \(P = 64 \times 129 = 8{,}256\) parameters.

1.2.3 Complexity Analysis

The matrix-vector product dominates both passes. For embedded networks (e.g. \(n = 32, m = 16\)), a single forward pass takes ~512 multiply-accumulate operations.

1.2.4 Step-by-Step Walkthrough

Layer: 3 inputs → 2 outputs, ReLU activation.

\[W = \begin{bmatrix} 0.5 & -0.3 & 0.8 \\ 0.1 & 0.7 & -0.2 \end{bmatrix}, \quad b = \begin{bmatrix} 0.1 \\ -0.1 \end{bmatrix}, \quad a_{\text{in}} = \begin{bmatrix} 1.0 \\ 0.5 \\ -1.0 \end{bmatrix}\]

Forward:

Backward with \(\frac{\partial \mathcal{L}}{\partial a_{\text{out}}} = [0.2,\; -0.4]^T\):

1.2.5 Pitfalls & Edge Cases

• Dimension mismatch. In a multi-layer network, the output size of layer \(\ell\) must equal the input size of layer \(\ell+1\). This library enforces this at compile time.
• Weight initialization. Zero-initialized weights cause all neurons to compute the same thing (symmetry problem). Use He initialization for ReLU: \(W_{ij} \sim \mathcal{N}(0, \sqrt{2/n})\).
• Gradient accumulation. When processing mini-batches, accumulate \(\nabla W\) across samples before updating — do not update per-sample.
• Fixed-point overflow in \(W a\). The dot product of \(n\) terms can exceed Q15/Q31 range. Use a wider accumulator (Q31 for Q15 data) or scale weights.
• Large layers exhaust stack. A 256×256 weight matrix of float consumes 256 KB. Size layers to fit the target’s stack budget.

1.2.6 Variants & Generalizations

1.2.7 Applications

• Hidden layers — One or more dense layers form the core of feed-forward networks for regression and classification.
• Output layer — A final dense layer maps to the target dimensionality (1 for regression, \(k\) for \(k\)-class classification).
• Embedding projection — Dense layers project high-dimensional sparse inputs to low-dimensional dense representations.
• Controller networks — In neural network-based control, small dense layers map state vectors to actuator commands.

1.2.8 Connections to Other Algorithms

graph TD
    Layer["Dense Layer"]
    Act["Activation Functions"]
    Model["Model"]
    Opt["Optimizer"]
    LR["Linear Regression"]

    Act --> Layer
    Layer --> Model
    Model --> Opt
    Layer -.->|"no activation, MSE loss"| LR

1.3 Activation Functions

1.3.1 Overview & Motivation

An activation function \(f\) is a non-linear, element-wise transformation applied after the affine map in each neural network layer:

\[a = f(z) = f(W x + b)\]

Without activation functions, stacking layers would collapse into a single affine transformation — the network could only represent linear mappings regardless of depth. Activation functions are what give neural networks their expressive power.

The choice of activation function controls gradient flow during back-propagation, output range, and computational cost — all critical on resource-constrained embedded targets.

1.3.2 Mathematical Theory

1.3.2.1 Forward and Backward

Every activation function exposes two operations:

The chain rule connects them during training:

\[\frac{\partial \mathcal{L}}{\partial z} = \frac{\partial \mathcal{L}}{\partial a} \cdot f'(z)\]

1.3.2.2 Catalogue

1.3.2.2.1 ReLU (Rectified Linear Unit)

\[f(z) = \max(0, z), \qquad f'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \le 0 \end{cases}\]

• Computationally cheapest (single comparison).
• Sparse activations accelerate training.
• Risk: neurons with \(z \le 0\) for all inputs are permanently dead.

1.3.2.2.2 Leaky ReLU

\[f(z) = \begin{cases} z & z > 0 \\ \alpha z & z \le 0 \end{cases}, \qquad f'(z) = \begin{cases} 1 & z > 0 \\ \alpha & z \le 0 \end{cases}\]

where \(\alpha\) is a small positive constant (typically \(0.01\)). Prevents dead neurons by allowing a small gradient for \(z < 0\).

1.3.2.2.3 Sigmoid

\[f(z) = \frac{1}{1 + e^{-z}}, \qquad f'(z) = f(z)(1 - f(z))\]

• Output range \((0, 1)\) — natural for probabilities.
• Risk: saturates for \(|z| \gg 0\), causing vanishing gradients.

1.3.2.2.4 Tanh (Hyperbolic Tangent)

\[f(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}, \qquad f'(z) = 1 - f(z)^2\]

• Output range \((-1, 1)\) — zero-centered, unlike Sigmoid.
• Same saturation problem as Sigmoid at extremes.

1.3.2.2.5 Softmax

For a vector \(\mathbf{z} \in \mathbb{R}^k\):

\[f(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}}\]

• Outputs form a probability distribution (\(\sum f(z_i) = 1\)).
• Used exclusively at the output layer for multi-class classification.
• The Jacobian is a full \(k \times k\) matrix — more expensive than element-wise activations.

1.3.3 Complexity Analysis

1.3.4 Step-by-Step Walkthrough

Scenario: Forward and backward pass through a 3-neuron hidden layer with ReLU, given pre-activations \(z = [-0.5, \; 1.2, \; 0.0]\).

Forward:

Backward with incoming gradient \(\frac{\partial \mathcal{L}}{\partial a} = [0.3, \; -0.7, \; 0.1]\):

Neuron 1 is dead — its gradient is zero and its weights will not update. If this persists across all training samples, the neuron is permanently inactive.

1.3.5 Pitfalls & Edge Cases

• Vanishing gradients with Sigmoid/Tanh. For deep networks, gradients shrink exponentially through saturated activations. Use ReLU or Leaky ReLU in hidden layers.
• Dead ReLU neurons. If a neuron’s bias drifts negative enough that no input ever produces \(z > 0\), it stops learning. Leaky ReLU or careful initialization (He init) prevents this.
• Softmax numerical instability. Computing \(e^{z_i}\) directly overflows for large \(z_i\). Subtract \(\max(\mathbf{z})\) before exponentiation: \(f(z_i) = e^{z_i - z_{\max}} / \sum e^{z_j - z_{\max}}\).
• Fixed-point range. Sigmoid outputs \((0, 1)\) and Tanh outputs \((-1, 1)\) both fit in Q15/Q31, but the exponential intermediate values do not. Compute in floating-point and convert.
• Non-differentiable points. ReLU is technically non-differentiable at \(z = 0\). In practice, assigning \(f'(0) = 0\) works fine.

1.3.6 Variants & Generalizations

1.3.7 Applications

• Hidden layers — ReLU (or Leaky ReLU) is the default choice for feed-forward and convolutional hidden layers.
• Binary classification output — Sigmoid maps the output to a probability.
• Multi-class classification output — Softmax produces a probability distribution.
• Recurrent networks — Tanh is traditionally used in LSTM/GRU gates.
• Embedded inference — Hard Sigmoid/Tanh avoid expensive exp() calls on MCUs without FPU.

1.3.8 Connections to Other Algorithms

graph TD
    Act["Activation Functions"]
    Layer["Dense Layer"]
    NN["Neural Network"]
    Loss["Loss Functions"]
    Act --> Layer
    Layer --> NN
    NN --> Loss

1.4 Loss Functions

1.4.1 Overview & Motivation

A loss function \(\mathcal{L}(\hat{y}, y)\) quantifies how far a model’s prediction \(\hat{y}\) is from the true target \(y\). Training a neural network means finding parameters \(\theta\) that minimize the expected loss over the training data:

\[\theta^* = \arg\min_\theta \; \mathbb{E}[\mathcal{L}(f_\theta(x), y)]\]

The loss function defines the entire learning objective — different losses lead to different optimal models even on the same data. It must also provide a gradient \(\nabla_{\hat{y}} \mathcal{L}\) for back-propagation.

1.4.2 Mathematical Theory

1.4.2.1 Mean Squared Error (MSE)

\[\mathcal{L}_{\text{MSE}} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}_i - y_i)^2\]

\[\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = \frac{2}{m}(\hat{y}_i - y_i)\]

• Penalizes large errors quadratically → sensitive to outliers.
• Natural choice for regression tasks.
• Optimal when the error distribution is Gaussian.

1.4.2.2 Mean Absolute Error (MAE)

\[\mathcal{L}_{\text{MAE}} = \frac{1}{m} \sum_{i=1}^{m} |\hat{y}_i - y_i|\]

\[\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = \frac{1}{m} \mathrm{sign}(\hat{y}_i - y_i)\]

• Linear penalty → robust to outliers.
• Non-differentiable at \(\hat{y}_i = y_i\) (use sub-gradient in practice).

1.4.2.3 Binary Cross-Entropy (BCE)

\[\mathcal{L}_{\text{BCE}} = -\frac{1}{m}\sum_{i=1}^{m} \left[ y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i) \right]\]

\[\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = -\frac{1}{m}\left(\frac{y_i}{\hat{y}_i} - \frac{1 - y_i}{1 - \hat{y}_i}\right)\]

• Assumes \(\hat{y}_i \in (0, 1)\) — pair with Sigmoid output activation.
• Maximum likelihood estimator for Bernoulli-distributed targets.

1.4.2.4 Categorical Cross-Entropy (CCE)

\[\mathcal{L}_{\text{CCE}} = -\sum_{i=1}^{k} y_i \log \hat{y}_i\]

\[\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = -\frac{y_i}{\hat{y}_i}\]

• Assumes \(\hat{y}\) is a probability distribution (\(\sum \hat{y}_i = 1\)) — pair with Softmax output activation.
• Standard loss for multi-class classification.

1.4.3 Complexity Analysis

All losses are \(O(m)\) where \(m\) is the output dimension. The computational cost is negligible compared to the dense layer matrix products.

1.4.4 Step-by-Step Walkthrough

Scenario: 3-class classification. Target \(y = [0, 1, 0]\) (class 2). Softmax output \(\hat{y} = [0.1, 0.7, 0.2]\).

CCE Forward:

\[\mathcal{L} = -(0 \cdot \log 0.1 + 1 \cdot \log 0.7 + 0 \cdot \log 0.2) = -\log(0.7) \approx 0.357\]

CCE Backward:

The gradient is non-zero only for the true class, and its magnitude \(1/\hat{y}_2\) grows as the prediction worsens — providing a strong corrective signal.

For comparison — MSE on the same example:

\[\mathcal{L}_{\text{MSE}} = \frac{1}{3}[(0.1)^2 + (0.7-1)^2 + (0.2)^2] = \frac{1}{3}[0.01 + 0.09 + 0.04] = 0.047\]

MSE gives a much weaker signal and does not account for the probabilistic nature of the output.

1.4.5 Pitfalls & Edge Cases

• \(\log(0)\) is \(-\infty\). Clamp predictions to \([\varepsilon, 1 - \varepsilon]\) before computing BCE or CCE. A typical \(\varepsilon = 10^{-7}\).
• Mismatched activation and loss. Sigmoid + CCE or Softmax + BCE produce incorrect gradients. Always pair: Sigmoid ↔︎ BCE, Softmax ↔︎ CCE.
• MSE for classification. MSE can train a classifier but converges slower than cross-entropy because its gradient does not account for the log-likelihood structure.
• Label encoding. CCE expects one-hot encoded targets. Integer labels must be converted first.
• Fixed-point log. The log() function needed for cross-entropy is expensive and ill-conditioned near zero in fixed-point. Evaluate in floating-point.

1.4.6 Variants & Generalizations

1.4.7 Applications

• Regression — MSE for Gaussian noise, MAE for Laplacian noise or when outlier robustness is needed.
• Binary classification — BCE for yes/no decisions (fault detection, anomaly flagging).
• Multi-class classification — CCE for mutually exclusive categories (gesture recognition, signal type identification).
• Probabilistic output — Cross-entropy losses calibrate prediction confidence, not just accuracy.

1.4.8 Connections to Other Algorithms

graph TD
    Loss["Loss Functions"]
    Act["Activation Functions"]
    Opt["Optimizer"]
    Reg["Regularization"]
    Model["Model"]
    LR["Linear Regression"]

    Act -->|"output activation must match loss"| Loss
    Loss --> Opt
    Reg -->|"added to loss"| Loss
    Loss --> Model
    Loss -.->|"MSE + normal equation"| LR

1.5 Model (Neural Network Composition)

1.5.1 Overview & Motivation

A Model composes a sequence of dense layers into a single trainable function \(f: \mathbb{R}^n \to \mathbb{R}^m\). It is the orchestrator that:

1. Chains layers so the output of each feeds into the next (forward pass).
2. Propagates gradients backward through the chain (backward pass).
3. Flattens all layer parameters into a single vector for the optimizer.
4. Verifies dimensional compatibility at compile time using variadic templates.

In this library, the Model is fully statically typed — layer dimensions, parameter counts, and memory footprints are all known at compile time, enabling zero-overhead abstraction on embedded targets.

1.5.2 Mathematical Theory

1.5.2.1 Composition

For \(L\) layers with transformations \(f_1, f_2, \ldots, f_L\):

\[\hat{y} = (f_L \circ f_{L-1} \circ \cdots \circ f_1)(x) = f_L(f_{L-1}(\ldots f_1(x) \ldots))\]

Each \(f_\ell\) is a dense layer: \(f_\ell(a) = \sigma_\ell(W_\ell a + b_\ell)\).

1.5.2.2 Parameter Vector

All weights and biases are concatenated into a single vector:

\[\theta = [\text{vec}(W_1), b_1, \text{vec}(W_2), b_2, \ldots, \text{vec}(W_L), b_L] \in \mathbb{R}^P\]

where \(P = \sum_{\ell=1}^L m_\ell(n_\ell + 1)\).

1.5.2.3 Forward Pass (Chained Evaluation)

graph LR
    X["x ∈ ℝⁿ"] --> L1["Layer 1"] --> L2["Layer 2"] --> Ldots["⋯"] --> LL["Layer L"] --> Y["ŷ ∈ ℝᵐ"]

1.5.2.4 Backward Pass (Reverse Chain Rule)

\[\frac{\partial \mathcal{L}}{\partial \theta_\ell} = \frac{\partial \mathcal{L}}{\partial a_L} \cdot \frac{\partial a_L}{\partial a_{L-1}} \cdots \frac{\partial a_{\ell+1}}{\partial a_\ell} \cdot \frac{\partial a_\ell}{\partial \theta_\ell}\]

Each layer stores its input \(a_{\ell-1}\) during the forward pass so it can compute \(\nabla W_\ell\) and \(\nabla b_\ell\) during the backward pass.

1.5.2.5 Training Loop

graph TD
    FP["Forward pass: ŷ = Model(x)"]
    LC["Loss: ℒ(ŷ, y)"]
    BP["Backward pass: ∇θ ℒ"]
    UP["Update: θ ← θ − η ∇θ ℒ"]
    FP --> LC --> BP --> UP --> FP

1.5.3 Complexity Analysis

All operations scale linearly with the total parameter count \(P\).

1.5.4 Step-by-Step Walkthrough

Model: 2 → 3 → 1 (two layers).

Compile-time verification chain:

Parameter layout (\(P = 3(2 + 1) + 1(3 + 1) = 9 + 4 = 13\)):

Forward pass with \(x = [1.0, 0.5]^T\):

1. Layer 1: \(a_1 = \text{ReLU}(W_1 x + b_1) = [0.0, 0.8, 0.3]^T\)
2. Layer 2: \(\hat{y} = \sigma(W_2 a_1 + b_2) = [0.62]\)

Backward pass with loss gradient \(\delta_{\text{out}} = [0.12]\):

1. Layer 2 backward → produces \(\nabla W_2\), \(\nabla b_2\), and \(\delta_1 = W_2^T \delta_2 \odot \text{ReLU}'(z_1)\)
2. Layer 1 backward → produces \(\nabla W_1\), \(\nabla b_1\)

Optimizer receives the full \(\nabla \theta \in \mathbb{R}^{13}\) and updates \(\theta\).

1.5.5 Pitfalls & Edge Cases

• Dimension mismatch caught at compile time. If layer \(\ell\) outputs \(m\) but layer \(\ell+1\) expects \(n \ne m\), a static_assert fires during compilation.
• Empty model. The variadic template requires at least one layer. static_assert(sizeof...(Layers) > 0).
• Parameter ordering. GetParameters() and SetParameters() must use the same concatenation order. The implementation iterates layers via std::index_sequence to guarantee consistency.
• Training with wrong optimizer size. The optimizer’s parameter count template argument must equal Model::TotalParameters. A mismatch is also caught at compile time.
• Large parameter vectors. All parameters live on the stack. A model with \(P > 10{,}000\) floats consumes 40 KB — verify this fits the target’s stack.

1.5.6 Variants & Generalizations

1.5.7 Applications

• Embedded inference — A pre-trained model’s parameters are loaded via SetParameters() and only Forward() is called at runtime.
• On-device training — The full forward → loss → backward → update loop runs on the MCU for online learning/adaptation.
• System identification — A small model (2–3 layers) learns the plant dynamics from input-output data.
• Sensor fusion — Multiple sensor inputs are mapped to a unified state estimate through a trained model.

1.5.8 Connections to Other Algorithms

graph TD
    Model["Model"]
    Layer["Dense Layer"]
    Loss["Loss Functions"]
    Opt["Optimizer"]
    Reg["Regularization"]
    NN["Neural Network"]

    Layer --> Model
    Model --> Opt
    Model --> Loss
    Reg --> Loss
    Model --> NN

2 References

• Abadi, M. et al., “TensorFlow: A system for large-scale machine learning”, OSDI, 2016.
• Bishop, C.M., Pattern Recognition and Machine Learning, Springer, 2006 — Chapter 4.3 (cross-entropy).
• Clevert, D.-A., Unterthiner, T., and Hochreiter, S., “Fast and accurate deep network learning by exponential linear units (ELUs)”, ICLR, 2016.
• Glorot, X. and Bengio, Y., “Understanding the difficulty of training deep feedforward neural networks”, AISTATS, 2010 — Xavier initialization.
• Glorot, X., Bordes, A., and Bengio, Y., “Deep sparse rectifier neural networks”, AISTATS, 2011.
• Goodfellow, I., Bengio, Y., and Courville, A., Deep Learning, MIT Press, 2016 — Chapter 6 (deep feedforward networks).
• Goodfellow, I., Bengio, Y., and Courville, A., Deep Learning, MIT Press, 2016 — Chapter 6.
• Goodfellow, I., Bengio, Y., and Courville, A., Deep Learning, MIT Press, 2016 — Chapter 6.2 (cost functions).
• Goodfellow, I., Bengio, Y., and Courville, A., Deep Learning, MIT Press, 2016 — Chapters 6–8.
• He, K. et al., “Delving deep into rectifiers”, ICCV, 2015 — He weight initialization.
• He, K. et al., “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification”, ICCV, 2015 — He initialization.
• Lin, T.-Y. et al., “Focal loss for dense object detection”, ICCV, 2017.
• Nair, V. and Hinton, G.E., “Rectified linear units improve restricted Boltzmann machines”, ICML, 2010.
• Paszke, A. et al., “PyTorch: An imperative style, high-performance deep learning library”, NeurIPS, 2019 — inspiration for the sequential/functional model API.
• Rumelhart, D.E., Hinton, G.E., and Williams, R.J., “Learning representations by back-propagating errors”, Nature, 323, 1986.