本文是 2020人工神经网络第一次作业 的参考答案第三部分
 

➤03 第三题参考答案

---

1.构造BP网络进行函数逼近

(1) 逼近简单函数

构建单隐层的神经网络，隐层节点个数20，传递函数为sigmoid函数，输出神经元的传递函数为线性函数。

^{▲ 神经网络结构}

直接在（0 ~ 1）之间均匀采样50个样本，使用最基本的BP算法训练上述网络。

^{▲ 逼近函数}

随着训练次数的增加，网络的输入输出关系变化如下：

^{▲ 训练过程中网络对应函数的变化}

下面是网络误差收敛曲线：

^{▲ 网络误差收敛曲线}

(2) 将数据进行预处理

• 将输入x转换到（-0.5, 0.5）之间
• 将网络输出转换到（-0.8,0.8）之间
  ^{▲ 网络训练过程对应函数变化}

^{▲ 网络训练误差收敛曲线}

可以看到对于数据进行预处理之后，将样本的输入、输出都转换到关于0对称，网络收敛的速度和精度都得到了提高。

(3) 修改网络隐层传递函数

将网络隐层传递函数修改为双曲正切函数。修改学习速率$\eta =0.25$。可以看到网络收敛速度又再一次增加了。

^{▲ 随着训练网络输入输出之间的关系}

^{▲ 网络训练误差收敛曲线}

(4) 训练四个周期的函数

学习速率$\eta =0.5$

^{▲ 逼近函数}

^{▲ 函数逼近效果}

^{▲ 网络训练误差收敛曲线}

(5) 训练六个周期的函数

使用隐层节点为20的神经网络，逼近六个周期的sin函数，单层网络训练不收敛。

^{▲ 逼近六个周期的函数}

随着训练过程增加，网络的输入输出函数稳定在一个中间值的状态。

^{▲ 网络输入输出函数变化情况}

^{▲ 网络误差收敛曲线}

#!/usr/local/bin/python
# -*- coding: gbk -*-
#============================================================
# HW13BP.PY                    -- by Dr. ZhuoQing 2020-11-17
#
# Note:
#============================================================from headm import *#------------------------------------------------------------
# Samples data constructionx_data = linspace(-0.5, 0.5, 50).reshape(-1, 1)
y_data = (sin(x_data*6*pi)*0.8).reshape(1,-1)#------------------------------------------------------------xx = linspace(-0.5,0.5, 500)
yy = sin(xx*6*pi) * 0.8
plt.plot(xx, yy, label='sin(2pix)')
plt.scatter(x_data.reshape(-1,1), y_data.reshape(-1,1), color='r')
plt.xlabel("x")
plt.ylabel("f(x)")
plt.grid(True)
plt.tight_layout()
plt.show()#------------------------------------------------------------
def shuffledata(X, Y):id = list(range(X.shape[0]))random.shuffle(id)return X[id], (Y.T[id]).T#------------------------------------------------------------
# Define and initialization NN
def initialize_parameters(n_x, n_h, n_y):random.seed(2)W1 = random.randn(n_h, n_x) * 0.5          # dot(W1,X.T)W2 = random.randn(n_y, n_h) * 0.5          # dot(W2,Z1)b1 = zeros((n_h, 1))                       # Column vectorb2 = zeros((n_y, 1))                       # Column vectorparameters = {'W1':W1,'b1':b1,'W2':W2,'b2':b2}return parameters#------------------------------------------------------------
# Forward propagattion
# X:row->sample;
# Z2:col->sample
def forward_propagate(X, parameters):W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']Z1 = dot(W1, X.T) + b1                    # X:row-->sample; Z1:col-->sample
#    A1 = 1/(1+exp(-Z1))A1 = (1-exp(-Z1))/(1+exp(-Z1))Z2 = dot(W2, A1) + b2                     # Z2:col-->sample
#    A2 = 1/(1+exp(-Z2))                       # A:col-->sampleA2 = Z2                                   # Linear outputcache = {'Z1':Z1,'A1':A1,'Z2':Z2,'A2':A2}return Z2, cache#------------------------------------------------------------
# Calculate the cost
# A2,Y: col->sample
def calculate_cost(A2, Y, parameters):err = [x1-x2 for x1,x2 in zip(A2.T, Y.T)]cost = [dot(e,e) for e in err]return mean(cost)#------------------------------------------------------------
# Backward propagattion
def backward_propagate(parameters, cache, X, Y):m = X.shape[0]                  # Number of the samplesW1 = parameters['W1']W2 = parameters['W2']A1 = cache['A1']A2 = cache['A2']dZ2 = (A2 - Y) #* (A2 * (1-A2))dW2 = dot(dZ2, A1.T) / mdb2 = sum(dZ2, axis=1, keepdims=True) / m#    dZ1 = dot(W2.T, dZ2) * (A1 * (1-A1))dZ1 = dot(W2.T, dZ2) * (1-A1**2)dW1 = dot(dZ1, X) / mdb1 = sum(dZ1, axis=1, keepdims=True) / mgrads = {'dW1':dW1,'db1':db1,'dW2':dW2,'db2':db2}return grads#------------------------------------------------------------
# Update the parameters
def update_parameters(parameters, grads, learning_rate):W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']dW1 = grads['dW1']db1 = grads['db1']dW2 = grads['dW2']db2 = grads['db2']W1 = W1 - learning_rate * dW1W2 = W2 - learning_rate * dW2b1 = b1 - learning_rate * db1b2 = b2 - learning_rate * db2parameters = {'W1':W1,'b1':b1,'W2':W2,'b2':b2}return parameters#------------------------------------------------------------
# Define the training
DISP_STEP           = 2000#------------------------------------------------------------
pltgif = PlotGIF()#------------------------------------------------------------
def train(X, Y, num_iterations, learning_rate, print_cost=False):
#    random.seed(3)n_x = 1n_y = 1n_h = 20lr = learning_rateparameters = initialize_parameters(n_x, n_h, n_y)XX,YY = shuffledata(X, Y)costdim = []x = linspace(-0.5, 0.5, 250).reshape(-1,1)for i in range(0, num_iterations):A2, cache = forward_propagate(XX, parameters)cost = calculate_cost(A2, YY, parameters)grads = backward_propagate(parameters, cache, XX, YY)parameters = update_parameters(parameters, grads, lr)if print_cost and i % DISP_STEP == 0:printf('Cost after iteration:%i: %f'%(i, cost))costdim.append(cost)plt.clf()y,cache = forward_propagate(x, parameters)plt.plot(x.reshape(-1,1), y.reshape(-1,1))plt.xlabel("x")plt.ylabel("f(x)")plt.grid(True)plt.tight_layout()plt.draw()plt.pause(.1)pltgif.append(plt)if cost < 0.0001:breakXX,YY = shuffledata(X, Y)return parameters, costdim#------------------------------------------------------------
parameter,costdim = train(x_data, y_data, 200000, 0.5, True)
pltgif.save(r'd:\temp\1.gif')#------------------------------------------------------------
plt.clf()
plt.plot(arange(len(costdim))*DISP_STEP, costdim)
plt.xlabel("Step(10)")
plt.ylabel("Cost")
plt.grid(True)
plt.tight_layout()
plt.show()#------------------------------------------------------------
#        END OF FILE : HW13BP.PY
#============================================================

2.将BP网络隐层传递函数修改成ReLU

将上述神经网络中间隐层的传递函数修改成ReLU函数。

修改程序部分：
对于 forward_propagate()中，A1的取值过程为：

A1 = Z1
A1[A1< 0]=0

对于backward_propaget()部分：

A11 = A1
A11[A11>0] = 1
A11[A11<0] = 0
dZ1 = dot(W2.T, dZ2) * A11

^{▲ 网络输入输出函数关系演变过程}

^{▲ 逼近单个sin函数式，网络训练误差变化曲线}

从前面的逼近结果来看，使用ReLU函数，在相同的隐层节点的情况下，函数误差偏大。

为了改善函数逼近的效果，将神经网络中间的神经元增加到50个，此时函数比较的效果如下：

^{▲ 隐层节点个数为50个时网络收敛情况}

^{▲ 网络误差收敛情况}

下面是处逼近四个周期的sin函数的过程。

^{▲ 逼近四个周期sin函数的过程}

^{▲ 网络训练误差收敛情况}

下面是逼近六个周期sin函数的网络收敛过程。

^{▲ 逼近六个周期sin信号网络输入输出关系变化情况}

^{▲ 逼近六个周期sin函数的网络误差收敛情况}

下面将神经网络中的隐层节点的个数提高到100个，训练样本的个数增加到250个。仍然使用传递函数ReLU 来逼近六个周期的sin函数。函数的收敛情况如下：

下面是网络输入输出之间的函数随着训练次数的增加出现的变化：

^{▲ 网络输入输出之间的函数关系的变化情况}

下面是网络训练误差随着训练次数瘦脸的情况：

^{▲ 逼近六个周期sin网络误差变化}

#!/usr/local/bin/python
# -*- coding: gbk -*-
#============================================================
# HW13BPRELU.PY                    -- by Dr. ZhuoQing 2020-11-17
#
# Note:
#============================================================from headm import *#------------------------------------------------------------
# Samples data constructionx_data = linspace(-0.5, 0.5, 250).reshape(-1, 1)
y_data = (sin(x_data*4*pi)*0.8).reshape(1,-1)#------------------------------------------------------------
'''
xx = linspace(-0.5,0.5, 500)
yy = sin(xx*6*pi) * 0.8
plt.plot(xx, yy, label='sin(2pix)')
plt.scatter(x_data.reshape(-1,1), y_data.reshape(-1,1), color='r')
plt.xlabel("x")
plt.ylabel("f(x)")
plt.grid(True)
plt.tight_layout()
plt.show()'''
#------------------------------------------------------------
def shuffledata(X, Y):id = list(range(X.shape[0]))random.shuffle(id)return X[id], (Y.T[id]).T#------------------------------------------------------------
# Define and initialization NN
def initialize_parameters(n_x, n_h, n_y):random.seed(2)W1 = random.randn(n_h, n_x) * 0.5          # dot(W1,X.T)W2 = random.randn(n_y, n_h) * 0.5          # dot(W2,Z1)b1 = zeros((n_h, 1))                       # Column vectorb2 = zeros((n_y, 1))                       # Column vectorparameters = {'W1':W1,'b1':b1,'W2':W2,'b2':b2}return parameters#------------------------------------------------------------
# Forward propagattion
# X:row->sample;
# Z2:col->sample
def forward_propagate(X, parameters):W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']Z1 = dot(W1, X.T) + b1                    # X:row-->sample; Z1:col-->sample
#    A1 = 1/(1+exp(-Z1))
#    A1 = (1-exp(-Z1))/(1+exp(-Z1))
#    A1 = array([[x if x > 0 else 0 for x in l] for l in Z1])A1 = Z1A1[A1< 0]=0Z2 = dot(W2, A1) + b2                     # Z2:col-->sample
#    A2 = 1/(1+exp(-Z2))                       # A:col-->sampleA2 = Z2                                   # Linear outputcache = {'Z1':Z1,'A1':A1,'Z2':Z2,'A2':A2}return Z2, cache#------------------------------------------------------------
# Calculate the cost
# A2,Y: col->sample
def calculate_cost(A2, Y, parameters):err = [x1-x2 for x1,x2 in zip(A2.T, Y.T)]cost = [dot(e,e) for e in err]return mean(cost)#------------------------------------------------------------
# Backward propagattion
def backward_propagate(parameters, cache, X, Y):m = X.shape[0]                  # Number of the samplesW1 = parameters['W1']W2 = parameters['W2']A1 = cache['A1']A2 = cache['A2']dZ2 = (A2 - Y) #* (A2 * (1-A2))dW2 = dot(dZ2, A1.T) / mdb2 = sum(dZ2, axis=1, keepdims=True) / m#    dZ1 = dot(W2.T, dZ2) * (A1 * (1-A1))
#    dZ1 = dot(W2.T, dZ2) * (1-A1**2)#    A11 = array([[1 if x >= 0 else 0 for x in l] for l in A1])A11 = A1A11[A11>0] = 1A11[A11<0] = 0dZ1 = dot(W2.T, dZ2) * A11dW1 = dot(dZ1, X) / mdb1 = sum(dZ1, axis=1, keepdims=True) / mgrads = {'dW1':dW1,'db1':db1,'dW2':dW2,'db2':db2}return grads#------------------------------------------------------------
# Update the parameters
def update_parameters(parameters, grads, learning_rate):W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']dW1 = grads['dW1']db1 = grads['db1']dW2 = grads['dW2']db2 = grads['db2']W1 = W1 - learning_rate * dW1W2 = W2 - learning_rate * dW2b1 = b1 - learning_rate * db1b2 = b2 - learning_rate * db2parameters = {'W1':W1,'b1':b1,'W2':W2,'b2':b2}return parameters#------------------------------------------------------------
# Define the training
DISP_STEP           = 1000#------------------------------------------------------------
pltgif = PlotGIF()#------------------------------------------------------------
def train(X, Y, num_iterations, learning_rate, print_cost=False):
#    random.seed(3)n_x = 1n_y = 1n_h = 50lr = learning_rateparameters = initialize_parameters(n_x, n_h, n_y)XX,YY = shuffledata(X, Y)costdim = []x = linspace(-0.5, 0.5, 250).reshape(-1,1)for i in range(0, num_iterations):A2, cache = forward_propagate(XX, parameters)cost = calculate_cost(A2, YY, parameters)grads = backward_propagate(parameters, cache, XX, YY)parameters = update_parameters(parameters, grads, lr)if print_cost and i % DISP_STEP == 0:printf('Cost after iteration:%i: %f'%(i, cost))costdim.append(cost)plt.clf()y,cache = forward_propagate(x, parameters)plt.plot(x.reshape(-1,1), y.reshape(-1,1))plt.xlabel("x")plt.ylabel("f(x)")plt.grid(True)plt.tight_layout()plt.draw()plt.pause(.1)pltgif.append(plt)if cost < 0.0001:breakXX,YY = shuffledata(X, Y)return parameters, costdim#------------------------------------------------------------
parameter,costdim = train(x_data, y_data, 100000, 0.5, True)
pltgif.save(r'd:\temp\1.gif')#------------------------------------------------------------
plt.clf()
plt.plot(arange(len(costdim))*DISP_STEP, costdim)
plt.xlabel("Step(10)")
plt.ylabel("Cost")
plt.grid(True)
plt.tight_layout()
plt.show()#------------------------------------------------------------
#        END OF FILE : HW13BPRELU.PY
#============================================================