前言

自从看了Transformer以后，里面提到了layer normalization。想到之前的也看过相关的normalization资料，很可惜没有记录下来。于是在这篇文章中记录normalization相关的知识，最常见的normalization是batch normalization，后面出现了layer normalization、instance normalization和group normalization。本篇博客主要参考知乎大神Juliuszh的详解深度学习中的Normalization，BN/LN/WN。
​

Normalization

吴恩达大致说过深度学习=80%数据+20%模型。训练的到的模型算法摇号，莫过于数据分布能够“独立同分布”independent and identically distributed，简称为i.i.d.。独立同分布的数据可以简化常规机器学习模型的训练、提升机器学习模型的预测能力。因此喂入数据之前需要“白化”：

1. 去除特征的相关性，即独立
2. 使得特征具有相同的均值和方差

Why：为什么需要normalization？

normalization主要解决深度学习里面的Internal Covariate Shift。深度神经网络涉及到很多层的叠加，而每一层的参数更新会导致上层的输入数据分布发生变化，通过层层叠加，高层的输入分布变化会非常剧烈，这就使得高层需要不断去重新适应底层的参数更新。关于Internal Covariate Shift的解释如下图所示，来自知乎魏秀参在一个回答中的解释。说到底就是为了解决梯度弥散的问题，提高训练速度。值得注意的是RNN无法使用Batch Normalization，因为它是一个动态的网络结构，同一个batch中训练实例有长有短，导致每一个时间步长必须维持各自的统计量，这使得BN并不能正确的使用。在RNN中，对bn进行改进也非常的困难。

具体的Normalization方法如下：

Internal Covariate Shift现象是：将每一层的输入作为一个分布看待，由于底层的参数随着训练更新，导致相同的输入分布得到的输出分布改变了。而机器学习中有个很重要的假设：IID独立同分布假设，就是假设训练数据和测试数据是满足相同分布的，这是通过训练数据获得的模型能够在测试集获得好的效果的一个基本保障。那么，细化到神经网络的每一层间，每轮训练时分布都是不一致，那么相对的训练效果就得不到保障，所以称为层间的covariate shift。

What：有什么normalization？

normalization的标准公式是$$h=f(g\cdot \frac{x-\mu }{\sigma }+b)$$

Batch Normalization

套用标准公式，$${\mu }_i=\frac{1}{M}\sum x_i,{\sigma }_i=\sqrt{\frac{1}{M}\sum (x_i-{\mu }_i{)}^2+ϵ}$$其中$M$是mini-batch的大小。Btatch Normalization适用场景是mini batch比较大，数据分布比较接近。在训练之前需要充分的shuffle。

Layer Normalization

$${\mu }_i=\sum x_i,{\sigma }_i=\sqrt{\sum (x_i-{\mu }_i{)}^2+ϵ}$$其中$i$枚举了该层所有的神经元。layer normalization可以避免BN中受mini batch的影响，适用于小mini batch 场景，动态网络场景和RNN场景。

Instance Normalization

Instance Normalization适用于生成模型中，比如图片风格迁移。因为图片生成的结果主要依赖于某个图像实例，所以对整个Batch进行Normalization操作并不适合图像风格化的任务，在风格迁移中适用IN不仅可以加速模型收敛，并且可以保持每个图像实例之间的独立性。还适用于图像超分辨率等。计算公式如下：

Group Normalization

看前沿的图也比较容易理解，极端情况下可等价于LN和IN。

其他

normalization除了上述方法外，还有Cosine Normalization和Weight Normalization，具体可以参考：https://zhuanlan.zhihu.com/p/33173246的文章。

How：怎么实现normalization

Batch Normalization

from keras.layers import Dense, Dropout, Activation, Flatten, BatchNormalization

Layer Normalization

## Layer Normalization
```python
def GroupNorm(x, gamma, beta, eps=1e−5):# x: input features with shape [N,C,H,W]# gamma, beta: scale and offset, with shape [1,C,1,1]# G: number of groups for GNN, C, H, W = x.shapex = tf.reshape(x, [N, G, C, H, W])mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True)x = (x − mean) / tf.sqrt(var + eps)x = tf.reshape(x, [N, C, H, W])return x ∗ gamma + beta
``

Group Normalization

def GroupNorm(x, gamma, beta, G, eps=1e−5):# x: input features with shape [N,C,H,W]# gamma, beta: scale and offset, with shape [1,C,1,1]# G: number of groups for GNN, C, H, W = x.shapex = tf.reshape(x, [N, G, C // G, H, W])mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True)x = (x − mean) / tf.sqrt(var + eps)x = tf.reshape(x, [N, C, H, W])return x ∗ gamma + beta
``## Instance Normalization
```pythonfrom keras.layers import Layer, InputSpec
from keras import initializers, regularizers, constraints
from keras import backend as Kclass InstanceNormalization(Layer):"""Instance normalization layer.Normalize the activations of the previous layer at each step,i.e. applies a transformation that maintains the mean activationclose to 0 and the activation standard deviation close to 1.# Argumentsaxis: Integer, the axis that should be normalized(typically the features axis).For instance, after a `Conv2D` layer with`data_format="channels_first"`,set `axis=1` in `InstanceNormalization`.Setting `axis=None` will normalize all values in eachinstance of the batch.Axis 0 is the batch dimension. `axis` cannot be set to 0 to avoid errors.epsilon: Small float added to variance to avoid dividing by zero.center: If True, add offset of `beta` to normalized tensor.If False, `beta` is ignored.scale: If True, multiply by `gamma`.If False, `gamma` is not used.When the next layer is linear (also e.g. `nn.relu`),this can be disabled since the scalingwill be done by the next layer.beta_initializer: Initializer for the beta weight.gamma_initializer: Initializer for the gamma weight.beta_regularizer: Optional regularizer for the beta weight.gamma_regularizer: Optional regularizer for the gamma weight.beta_constraint: Optional constraint for the beta weight.gamma_constraint: Optional constraint for the gamma weight.# Input shapeArbitrary. Use the keyword argument `input_shape`(tuple of integers, does not include the samples axis)when using this layer as the first layer in a Sequential model.# Output shapeSame shape as input.# References- [Layer Normalization](https://arxiv.org/abs/1607.06450)- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://arxiv.org/abs/1607.08022)"""def __init__(self,axis=None,epsilon=1e-3,center=True,scale=True,beta_initializer='zeros',gamma_initializer='ones',beta_regularizer=None,gamma_regularizer=None,beta_constraint=None,gamma_constraint=None,**kwargs):super(InstanceNormalization, self).__init__(**kwargs)self.supports_masking = Trueself.axis = axisself.epsilon = epsilonself.center = centerself.scale = scaleself.beta_initializer = initializers.get(beta_initializer)self.gamma_initializer = initializers.get(gamma_initializer)self.beta_regularizer = regularizers.get(beta_regularizer)self.gamma_regularizer = regularizers.get(gamma_regularizer)self.beta_constraint = constraints.get(beta_constraint)self.gamma_constraint = constraints.get(gamma_constraint)def build(self, input_shape):ndim = len(input_shape)if self.axis == 0:raise ValueError('Axis cannot be zero')if (self.axis is not None) and (ndim == 2):raise ValueError('Cannot specify axis for rank 1 tensor')self.input_spec = InputSpec(ndim=ndim)if self.axis is None:shape = (1,)else:shape = (input_shape[self.axis],)if self.scale:self.gamma = self.add_weight(shape=shape,name='gamma',initializer=self.gamma_initializer,regularizer=self.gamma_regularizer,constraint=self.gamma_constraint)else:self.gamma = Noneif self.center:self.beta = self.add_weight(shape=shape,name='beta',initializer=self.beta_initializer,regularizer=self.beta_regularizer,constraint=self.beta_constraint)else:self.beta = Noneself.built = Truedef call(self, inputs, training=None):input_shape = K.int_shape(inputs)reduction_axes = list(range(0, len(input_shape)))if self.axis is not None:del reduction_axes[self.axis]del reduction_axes[0]mean = K.mean(inputs, reduction_axes, keepdims=True)stddev = K.std(inputs, reduction_axes, keepdims=True) + self.epsilonnormed = (inputs - mean) / stddevbroadcast_shape = [1] * len(input_shape)if self.axis is not None:broadcast_shape[self.axis] = input_shape[self.axis]if self.scale:broadcast_gamma = K.reshape(self.gamma, broadcast_shape)normed = normed * broadcast_gammaif self.center:broadcast_beta = K.reshape(self.beta, broadcast_shape)normed = normed + broadcast_betareturn normeddef get_config(self):config = {'axis': self.axis,'epsilon': self.epsilon,'center': self.center,'scale': self.scale,'beta_initializer': initializers.serialize(self.beta_initializer),'gamma_initializer': initializers.serialize(self.gamma_initializer),'beta_regularizer': regularizers.serialize(self.beta_regularizer),'gamma_regularizer': regularizers.serialize(self.gamma_regularizer),'beta_constraint': constraints.serialize(self.beta_constraint),'gamma_constraint': constraints.serialize(self.gamma_constraint)}base_config = super(InstanceNormalization, self).get_config()return dict(list(base_config.items()) + list(config.items()))