动态图机制-DyGraph

PaddlePaddle的DyGraph模式是一种动态的图执行机制,可以立即执行结果,无需构建整个图。同时,和以往静态的执行计算图不同,DyGraph模式下您的所有操作可以立即获得执行结果,而不必等待所构建的计算图全部执行完成,这样可以让您更加直观地构建PaddlePaddle下的深度学习任务,以及进行模型的调试,同时还减少了大量用于构建静态计算图的代码,使得您编写、调试网络的过程变得更加便捷。

PaddlePaddle DyGraph是一个更加灵活易用的模式,可提供:

  • 更加灵活便捷的代码组织结构:使用python的执行控制流程和面向对象的模型设计
  • 更加便捷的调试功能:直接使用python的打印方法即时打印所需要的结果,从而检查正在运行的模型结果便于测试更改
  • 和静态执行图通用的模型代码:同样的模型代码可以使用更加便捷的DyGraph调试,执行,同时也支持使用原有的静态图模式执行

有关的动态图机制更多的实际模型示例请参考Paddle/models/dygraph

设置和基本用法

  • 升级到最新的PaddlePaddle 1.6.0:
  1. pip install -q --upgrade paddlepaddle==1.6.0
  • 使用fluid.dygraph.guard(place=None) 上下文:
  1. import paddle.fluid as fluid
  2. with fluid.dygraph.guard():
  3. # write your executable dygraph code here

现在您就可以在fluid.dygraph.guard()上下文环境中使用DyGraph的模式运行网络了,DyGraph将改变以往PaddlePaddle的执行方式: 现在他们将会立即执行,并且将计算结果返回给Python。

Dygraph将非常适合和Numpy一起使用,使用fluid.dygraph.to_variable(x)将会将ndarray转换为fluid.Variable,而使用fluid.Variable.numpy()将可以把任意时刻获取到的计算结果转换为Numpyndarray

  1. x = np.ones([2, 2], np.float32)
  2. with fluid.dygraph.guard():
  3. inputs = []
  4. for _ in range(10):
  5. inputs.append(fluid.dygraph.to_variable(x))
  6. ret = fluid.layers.sums(inputs)
  7. print(ret.numpy())

得到输出:

  1. [[10. 10.]
  2. [10. 10.]]
这里创建了一系列ndarray的输入,执行了一个sum操作之后,我们可以直接将运行的结果打印出来

然后通过调用reduce_sum后使用Variable.backward()方法执行反向,使用Variable.gradient()方法即可获得反向网络执行完成后的梯度值的ndarray形式:

  1. loss = fluid.layers.reduce_sum(ret)
  2. loss.backward()
  3. print(loss.gradient())

得到输出 :

  1. [1.]

基于DyGraph构建网络

  • 编写一段用于DyGraph执行的Object-Oriented-Designed, PaddlePaddle模型代码主要由以下两部分组成: 请注意,如果您设计的这一层结构是包含参数的,则必须要使用继承自fluid.dygraph.Layer的Object-Oriented-Designed的类来描述该层的行为。

    • 建立一个可以在DyGraph模式中执行的,Object-Oriented的网络,需要继承自fluid.dygraph.Layer,其中需要调用基类的__init__方法,并且实现带有参数name_scope(用来标识本层的名字)的__init__构造函数,在构造函数中,我们通常会执行一些例如参数初始化,子网络初始化的操作,执行这些操作时不依赖于输入的动态信息:
  1. class MyLayer(fluid.dygraph.Layer):
  2. def __init__(self, name_scope):
  3. super(MyLayer, self).__init__(name_scope)
  • 实现一个forward(self, *inputs)的执行函数,该函数将负责执行实际运行时网络的执行逻辑, 该函数将会在每一轮训练/预测中被调用,这里我们将执行一个简单的relu -> elementwise add -> reduce sum
  1. def forward(self, inputs):
  2. x = fluid.layers.relu(inputs)
  3. self._x_for_debug = x
  4. x = fluid.layers.elementwise_mul(x, x)
  5. x = fluid.layers.reduce_sum(x)
  6. return [x]
  • fluid.dygraph.guard()中执行:

    • 使用Numpy构建输入:
  1. np_inp = np.array([1.0, 2.0, -1.0], dtype=np.float32)
  • 转换输入的ndarrayVariable, 并执行前向网络获取返回值: 使用fluid.dygraph.to_variable(np_inp)转换Numpy输入为DyGraph接收的输入,然后使用my_layer(var_inp)[0]调用callable object并且获取了x作为返回值,利用x.numpy()方法直接获取了执行得到的xndarray返回值。
  1. with fluid.dygraph.guard():
  2. var_inp = fluid.dygraph.to_variable(np_inp)
  3. my_layer = MyLayer("my_layer")
  4. x = my_layer(var_inp)[0]
  5. dy_out = x.numpy()
  • 计算梯度:自动微分对于实现机器学习算法(例如用于训练神经网络的反向传播)来说很有用, 使用x.backward()方法可以从某个fluid.Varaible开始执行反向网络,同时利用my_layer._x_for_debug.gradient()获取了网络中x梯度的ndarray 返回值:
  1. x.backward()
  2. dy_grad = my_layer._x_for_debug.gradient()

完整代码如下:

  1. import paddle.fluid as fluid
  2. import numpy as np
  3.  
  4. class MyLayer(fluid.dygraph.Layer):
  5. def __init__(self, name_scope):
  6. super(MyLayer, self).__init__(name_scope)
  7. self.fc = fluid.dygraph.nn.FC(self.full_name(), size=12)
  8.  
  9. def forward(self, inputs):
  10. x = self.fc(inputs)
  11. x = fluid.layers.relu(x)
  12. self._x_for_debug = x
  13. x = fluid.layers.elementwise_mul(x, x)
  14. x = fluid.layers.reduce_sum(x)
  15. return [x]
  16.  
  17. if __name__ == '__main__':
  18. np_inp = np.array([[1.0, 2.0, -1.0]], dtype=np.float32)
  19. with fluid.dygraph.guard():
  20. var_inp = fluid.dygraph.to_variable(np_inp)
  21. my_layer = MyLayer("my_layer")
  22. x = my_layer(var_inp)[0]
  23. dy_out = x.numpy()
  24. x.backward()
  25. dy_grad = my_layer._x_for_debug.gradient()
  26. my_layer.clear_gradients() # 将参数梯度清零以保证下一轮训练的正确性

关于自动剪枝

每个 Variable 都有一个 stop_gradient 属性,可以用于细粒度地在反向梯度计算时排除部分子图,以提高效率。

如果OP只要有一个输入需要梯度,那么该OP的输出也需要梯度。 相反,只有当OP的所有输入都不需要梯度时,该OP的输出也不需要梯度。 在所有的 Variable 都不需要梯度的子图中,反向计算就不会进行计算了。

在动态图模式下,除参数以外的所有 Variablestop_gradient 属性默认值都为 True,而参数的 stop_gradient 属性默认值为 False。 该属性用于自动剪枝,避免不必要的反向运算。

例如:

  1. import paddle.fluid as fluid
  2. import numpy as np
  3.  
  4. with fluid.dygraph.guard():
  5. x = fluid.dygraph.to_variable(np.random.randn(5, 5)) # 默认stop_gradient=True
  6. y = fluid.dygraph.to_variable(np.random.randn(5, 5)) # 默认stop_gradient=True
  7. z = fluid.dygraph.to_variable(np.random.randn(5, 5))
  8. z.stop_gradient = False
  9. a = x + y
  10. a.stop_gradient # True
  11. b = a + z
  12. b.stop_gradient # False

当你想冻结你的模型的一部分,或者你事先知道你不会使用某些参数的梯度的时候,这个功能是非常有用的。

例如:

  1. import paddle.fluid as fluid
  2. import numpy as np
  3.  
  4. with fluid.dygraph.guard():
  5. value0 = np.arange(26).reshape(2, 13).astype("float32")
  6. value1 = np.arange(6).reshape(2, 3).astype("float32")
  7. value2 = np.arange(10).reshape(2, 5).astype("float32")
  8. fc = fluid.FC("fc1", size=5, dtype="float32")
  9. fc2 = fluid.FC("fc2", size=3, dtype="float32")
  10. a = fluid.dygraph.to_variable(value0)
  11. b = fluid.dygraph.to_variable(value1)
  12. c = fluid.dygraph.to_variable(value2)
  13. out1 = fc(a)
  14. out2 = fc2(b)
  15. out1.stop_gradient = True # 将不会对out1这部分子图做反向计算
  16. out = fluid.layers.concat(input=[out1, out2, c], axis=1)
  17. out.backward()
  18. # 可以发现这里fc参数的梯度都为0
  19. assert (fc._w.gradient() == 0).all()
  20. assert (out1.gradient() == 0).all()

使用DyGraph训练模型

接下来我们将以“手写数字识别”这个最基础的模型为例,展示如何利用DyGraph模式搭建并训练一个模型:

有关手写数字识别的相关理论知识请参考PaddleBook中的内容,我们在这里默认您已经了解了该模型所需的深度学习理论知识。

  • 准备数据,我们使用paddle.dataset.mnist作为训练所需要的数据集:
  1. train_reader = paddle.batch(
  2. paddle.dataset.mnist.train(), batch_size=BATCH_SIZE, drop_last=True)
  • 构建网络,虽然您可以根据之前的介绍自己定义所有的网络结构,但是您也可以直接使用fluid.dygraph.Layer当中我们为您定制好的一些基础网络结构,这里我们利用fluid.dygraph.Conv2D以及fluid.dygraph.Pool2d构建了基础的SimpleImgConvPool
  1. class SimpleImgConvPool(fluid.dygraph.Layer):
  2. def __init__(self,
  3. name_scope,
  4. num_filters,
  5. filter_size,
  6. pool_size,
  7. pool_stride,
  8. pool_padding=0,
  9. pool_type='max',
  10. global_pooling=False,
  11. conv_stride=1,
  12. conv_padding=0,
  13. conv_dilation=1,
  14. conv_groups=1,
  15. act=None,
  16. use_cudnn=False,
  17. param_attr=None,
  18. bias_attr=None):
  19. super(SimpleImgConvPool, self).__init__(name_scope)
  20.  
  21. self._conv2d = fluid.dygraph.Conv2D(
  22. self.full_name(),
  23. num_filters=num_filters,
  24. filter_size=filter_size,
  25. stride=conv_stride,
  26. padding=conv_padding,
  27. dilation=conv_dilation,
  28. groups=conv_groups,
  29. param_attr=param_attr,
  30. bias_attr=bias_attr,
  31. act=act,
  32. use_cudnn=use_cudnn)
  33.  
  34. self._pool2d = fluid.dygraph.Pool2D(
  35. self.full_name(),
  36. pool_size=pool_size,
  37. pool_type=pool_type,
  38. pool_stride=pool_stride,
  39. pool_padding=pool_padding,
  40. global_pooling=global_pooling,
  41. use_cudnn=use_cudnn)
  42.  
  43. def forward(self, inputs):
  44. x = self._conv2d(inputs)
  45. x = self._pool2d(x)
  46. return x

注意: 构建网络时子网络的定义和使用请在__init__中进行, 而子网络的执行则在forward函数中进行

  • 利用已经构建好的SimpleImgConvPool组成最终的MNIST网络:
  1. class MNIST(fluid.dygraph.Layer):
  2. def __init__(self, name_scope):
  3. super(MNIST, self).__init__(name_scope)
  4.  
  5. self._simple_img_conv_pool_1 = SimpleImgConvPool(
  6. self.full_name(), 20, 5, 2, 2, act="relu")
  7.  
  8. self._simple_img_conv_pool_2 = SimpleImgConvPool(
  9. self.full_name(), 50, 5, 2, 2, act="relu")
  10.  
  11. pool_2_shape = 50 * 4 * 4
  12. SIZE = 10
  13. scale = (2.0 / (pool_2_shape**2 * SIZE))**0.5
  14. self._fc = fluid.dygraph.FC(self.full_name(),
  15. 10,
  16. param_attr=fluid.param_attr.ParamAttr(
  17. initializer=fluid.initializer.NormalInitializer(
  18. loc=0.0, scale=scale)),
  19. act="softmax")
  20.  
  21. def forward(self, inputs, label=None):
  22. x = self._simple_img_conv_pool_1(inputs)
  23. x = self._simple_img_conv_pool_2(x)
  24. x = self._fc(x)
  25. if label is not None:
  26. acc = fluid.layers.accuracy(input=x, label=label)
  27. return x, acc
  28. else:
  29. return x
  • fluid.dygraph.guard()中定义配置好的MNIST网络结构,此时即使没有训练也可以在fluid.dygraph.guard()中调用模型并且检查输出:
  1. with fluid.dygraph.guard():
  2. mnist = MNIST("mnist")
  3. id, data = list(enumerate(train_reader()))[0]
  4. dy_x_data = np.array(
  5. [x[0].reshape(1, 28, 28)
  6. for x in data]).astype('float32')
  7. img = fluid.dygraph.to_variable(dy_x_data)
  8. print("result is: {}".format(mnist(img).numpy()))

输出:

  1. result is: [[0.10135901 0.1051138 0.1027941 ... 0.0972859 0.10221873 0.10165327]
  2. [0.09735426 0.09970362 0.10198303 ... 0.10134517 0.10179105 0.10025002]
  3. [0.09539858 0.10213123 0.09543551 ... 0.10613529 0.10535969 0.097991 ]
  4. ...
  5. [0.10120598 0.0996111 0.10512722 ... 0.10067689 0.10088114 0.10071224]
  6. [0.09889644 0.10033772 0.10151272 ... 0.10245881 0.09878646 0.101483 ]
  7. [0.09097178 0.10078511 0.10198414 ... 0.10317434 0.10087223 0.09816764]]
  • 构建训练循环,在每一轮参数更新完成后我们调用mnist.clear_gradients()来重置梯度:
  1. with fluid.dygraph.guard():
  2. epoch_num = 5
  3. BATCH_SIZE = 64
  4. train_reader = paddle.batch(
  5. paddle.dataset.mnist.train(), batch_size=32, drop_last=True)
  6. mnist = MNIST("mnist")
  7. adam = fluid.optimizer.AdamOptimizer(learning_rate=0.001)
  8. for epoch in range(epoch_num):
  9. for batch_id, data in enumerate(train_reader()):
  10. dy_x_data = np.array([x[0].reshape(1, 28, 28)
  11. for x in data]).astype('float32')
  12. y_data = np.array(
  13. [x[1] for x in data]).astype('int64').reshape(-1, 1)
  14.  
  15. img = fluid.dygraph.to_variable(dy_x_data)
  16. label = fluid.dygraph.to_variable(y_data)
  17.  
  18. cost = mnist(img)
  19.  
  20. loss = fluid.layers.cross_entropy(cost, label)
  21. avg_loss = fluid.layers.mean(loss)
  22.  
  23. if batch_id % 100 == 0 and batch_id is not 0:
  24. print("epoch: {}, batch_id: {}, loss is: {}".format(epoch, batch_id, avg_loss.numpy()))
  25. avg_loss.backward()
  26. adam.minimize(avg_loss)
  27. mnist.clear_gradients()
  • 变量及优化器

模型的参数或者任何您希望检测的值可以作为变量封装在类中,然后通过对象获取并使用numpy()方法获取其ndarray的输出, 在训练过程中您可以使用mnist.parameters()来获取到网络中所有的参数,也可以指定某一个Layer的某个参数或者parameters()来获取该层的所有参数,使用numpy()方法随时查看参数的值

反向运行后调用之前定义的Adam优化器对象的minimize方法进行参数更新:

  1. with fluid.dygraph.guard():
  2. epoch_num = 5
  3. BATCH_SIZE = 64
  4.  
  5. mnist = MNIST("mnist")
  6. adam = fluid.optimizer.AdamOptimizer(learning_rate=0.001)
  7. train_reader = paddle.batch(
  8. paddle.dataset.mnist.train(), batch_size= BATCH_SIZE, drop_last=True)
  9.  
  10. np.set_printoptions(precision=3, suppress=True)
  11. for epoch in range(epoch_num):
  12. for batch_id, data in enumerate(train_reader()):
  13. dy_x_data = np.array(
  14. [x[0].reshape(1, 28, 28)
  15. for x in data]).astype('float32')
  16. y_data = np.array(
  17. [x[1] for x in data]).astype('int64').reshape(BATCH_SIZE, 1)
  18.  
  19. img = fluid.dygraph.to_variable(dy_x_data)
  20. label = fluid.dygraph.to_variable(y_data)
  21. label.stop_gradient = True
  22.  
  23. cost = mnist(img)
  24. loss = fluid.layers.cross_entropy(cost, label)
  25. avg_loss = fluid.layers.mean(loss)
  26.  
  27. dy_out = avg_loss.numpy()
  28.  
  29. avg_loss.backward()
  30. adam.minimize(avg_loss)
  31. mnist.clear_gradients()
  32.  
  33. dy_param_value = {}
  34. for param in mnist.parameters():
  35. dy_param_value[param.name] = param.numpy()
  36.  
  37. if batch_id % 20 == 0:
  38. print("Loss at step {}: {}".format(batch_id, avg_loss.numpy()))
  39. print("Final loss: {}".format(avg_loss.numpy()))
  40. print("_simple_img_conv_pool_1_conv2d W's mean is: {}".format(mnist._simple_img_conv_pool_1._conv2d._filter_param.numpy().mean()))
  41. print("_simple_img_conv_pool_1_conv2d Bias's mean is: {}".format(mnist._simple_img_conv_pool_1._conv2d._bias_param.numpy().mean()))

输出:

  1. ```
  2. Loss at step 0: [2.302]
  3. Loss at step 20: [1.616]
  4. Loss at step 40: [1.244]
  5. Loss at step 60: [1.142]
  6. Loss at step 80: [0.911]
  7. Loss at step 100: [0.824]
  8. Loss at step 120: [0.774]
  9. Loss at step 140: [0.626]
  10. Loss at step 160: [0.609]
  11. Loss at step 180: [0.627]
  12. Loss at step 200: [0.466]
  13. Loss at step 220: [0.499]
  14. Loss at step 240: [0.614]
  15. Loss at step 260: [0.585]
  16. Loss at step 280: [0.503]
  17. Loss at step 300: [0.423]
  18. Loss at step 320: [0.509]
  19. Loss at step 340: [0.348]
  20. Loss at step 360: [0.452]
  21. Loss at step 380: [0.397]
  22. Loss at step 400: [0.54]
  23. Loss at step 420: [0.341]
  24. Loss at step 440: [0.337]
  25. Loss at step 460: [0.155]
  26. Final loss: [0.164]
  27. _simple_img_conv_pool_1_conv2d W's mean is: 0.00606656912714
  28. _simple_img_conv_pool_1_conv2d Bias's mean is: -3.4576318285e-05
  29. ```
  • 性能

在使用fluid.dygraph.guard()时可以通过传入fluid.CUDAPlace(0)或者fluid.CPUPlace()来选择执行DyGraph的设备,通常如果不做任何处理将会自动适配您的设备。

使用多卡训练模型

目前PaddlePaddle支持通过多进程方式进行多卡训练,即每个进程对应一张卡。训练过程中,在第一次执行前向操作时,如果该操作需要参数,则会将0号卡的参数Broadcast到其他卡上,确保各个卡上的参数一致;在计算完反向操作之后,将产生的参数梯度在所有卡之间进行聚合;最后在各个GPU卡上分别进行参数更新。

  1. place = fluid.CUDAPlace(fluid.dygraph.parallel.Env().dev_id)
  2. with fluid.dygraph.guard(place):
  3.  
  4. strategy = fluid.dygraph.parallel.prepare_context()
  5. mnist = MNIST("mnist")
  6. adam = AdamOptimizer(learning_rate=0.001)
  7. mnist = fluid.dygraph.parallel.DataParallel(mnist, strategy)
  8.  
  9. train_reader = paddle.batch(
  10. paddle.dataset.mnist.train(), batch_size=BATCH_SIZE, drop_last=True)
  11. train_reader = fluid.contrib.reader.distributed_batch_reader(
  12. train_reader)
  13.  
  14. for epoch in range(epoch_num):
  15. for batch_id, data in enumerate(train_reader()):
  16. dy_x_data = np.array([x[0].reshape(1, 28, 28)
  17. for x in data]).astype('float32')
  18. y_data = np.array(
  19. [x[1] for x in data]).astype('int64').reshape(-1, 1)
  20.  
  21. img = to_variable(dy_x_data)
  22. label = to_variable(y_data)
  23. label.stop_gradient = True
  24.  
  25. cost, acc = mnist(img, label)
  26.  
  27. loss = fluid.layers.cross_entropy(cost, label)
  28. avg_loss = fluid.layers.mean(loss)
  29.  
  30. avg_loss = mnist.scale_loss(avg_loss)
  31. avg_loss.backward()
  32. mnist.apply_collective_grads()
  33.  
  34. adam.minimize(avg_loss)
  35. mnist.clear_gradients()
  36. if batch_id % 100 == 0 and batch_id is not 0:
  37. print("epoch: {}, batch_id: {}, loss is: {}".format(epoch, batch_id, avg_loss.numpy()))

动态图单卡训练转多卡训练需要修改的地方主要有四处:

  • 需要从环境变量获取设备的ID,即:
  1. place = fluid.CUDAPlace(fluid.dygraph.parallel.Env().dev_id)
  • 需要对原模型做一些预处理,即:
  1. strategy = fluid.dygraph.parallel.prepare_context()
  2. mnist = MNIST("mnist")
  3. adam = AdamOptimizer(learning_rate=0.001)
  4. mnist = fluid.dygraph.parallel.DataParallel(mnist, strategy)
  • 数据读取,必须确保每个进程读取的数据是不同的,即所有进程读取数据的交集为空,所有进程读取数据的并集是完整的数据集:
  1. train_reader = paddle.batch(
  2. paddle.dataset.mnist.train(), batch_size=BATCH_SIZE, drop_last=True)
  3. train_reader = fluid.contrib.reader.distributed_batch_reader(
  4. train_reader)
  • 需要对loss进行调整,以及对参数的梯度进行聚合,即:
  1. avg_loss = mnist.scale_loss(avg_loss)
  2. avg_loss.backward()
  3. mnist.apply_collective_grads()

Paddle动态图多进程多卡模型训练启动时需要指定使用的GPU,即如果使用0,1,2,3卡,启动方式如下:

  1. python -m paddle.distributed.launch --selected_gpus=0,1,2,3 --log_dir ./mylog train.py

输出结果为:

  1. ----------- Configuration Arguments -----------
  2. cluster_node_ips: 127.0.0.1
  3. log_dir: ./mylog
  4. node_ip: 127.0.0.1
  5. print_config: True
  6. selected_gpus: 0,1,2,3
  7. started_port: 6170
  8. training_script: train.py
  9. training_script_args: ['--use_data_parallel', '1']
  10. use_paddlecloud: True
  11. ------------------------------------------------
  12. trainers_endpoints: 127.0.0.1:6170,127.0.0.1:6171,127.0.0.1:6172,127.0.0.1:6173 , node_id: 0 , current_node_ip: 127.0.0.1 , num_nodes: 1 , node_ips: ['127.0.0.1'] , nranks: 4

此时,程序会将每个进程的输出log导出到./mylog路径下:

  1. .
  2. ├── mylog
  3. ├── workerlog.0
  4. ├── workerlog.1
  5. ├── workerlog.2
  6. └── workerlog.3
  7. └── train.py

如果不指定--log_dir,程序会将打印出所有进程的输出,即:

  1. ----------- Configuration Arguments -----------
  2. cluster_node_ips: 127.0.0.1
  3. log_dir: None
  4. node_ip: 127.0.0.1
  5. print_config: True
  6. selected_gpus: 0,1,2,3
  7. started_port: 6170
  8. training_script: train.py
  9. training_script_args: ['--use_data_parallel', '1']
  10. use_paddlecloud: True
  11. ------------------------------------------------
  12. trainers_endpoints: 127.0.0.1:6170,127.0.0.1:6171,127.0.0.1:6172,127.0.0.1:6173 , node_id: 0 , current_node_ip: 127.0.0.1 , num_nodes: 1 , node_ips: ['127.0.0.1'] , nranks: 4
  13. grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
  14. grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
  15. grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
  16. grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
  17. I0923 09:32:36.423513 56410 nccl_context.cc:120] init nccl context nranks: 4 local rank: 1 gpu id: 1
  18. I0923 09:32:36.425287 56411 nccl_context.cc:120] init nccl context nranks: 4 local rank: 2 gpu id: 2
  19. I0923 09:32:36.429337 56409 nccl_context.cc:120] init nccl context nranks: 4 local rank: 0 gpu id: 0
  20. I0923 09:32:36.429440 56412 nccl_context.cc:120] init nccl context nranks: 4 local rank: 3 gpu id: 3
  21. W0923 09:32:42.594097 56412 device_context.cc:198] Please NOTE: device: 3, CUDA Capability: 70, Driver API Version: 9.0, Runtime API Version: 9.0
  22. W0923 09:32:42.605836 56412 device_context.cc:206] device: 3, cuDNN Version: 7.5.
  23. W0923 09:32:42.632463 56410 device_context.cc:198] Please NOTE: device: 1, CUDA Capability: 70, Driver API Version: 9.0, Runtime API Version: 9.0
  24. W0923 09:32:42.637948 56410 device_context.cc:206] device: 1, cuDNN Version: 7.5.
  25. W0923 09:32:42.648674 56411 device_context.cc:198] Please NOTE: device: 2, CUDA Capability: 70, Driver API Version: 9.0, Runtime API Version: 9.0
  26. W0923 09:32:42.654021 56411 device_context.cc:206] device: 2, cuDNN Version: 7.5.
  27. W0923 09:32:43.048696 56409 device_context.cc:198] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.0, Runtime API Version: 9.0
  28. W0923 09:32:43.053236 56409 device_context.cc:206] device: 0, cuDNN Version: 7.5.
  29. start data reader (trainers_num: 4, trainer_id: 2)
  30. start data reader (trainers_num: 4, trainer_id: 3)
  31. start data reader (trainers_num: 4, trainer_id: 1)
  32. start data reader (trainers_num: 4, trainer_id: 0)
  33. Loss at epoch 0 step 0: [0.57390565]
  34. Loss at epoch 0 step 0: [0.57523954]
  35. Loss at epoch 0 step 0: [0.575606]
  36. Loss at epoch 0 step 0: [0.5767452]

模型参数的保存

动态图由于模型和优化器在不同的对象中存储,模型参数和优化器信息要分别存储。

在模型训练中可以使用 paddle.fluid.dygraph.save_dygraph(state_dict, model_path) 来保存模型参数的dict或优化器信息的dict。

同样可以使用 paddle.fluid.dygraph.load_dygraph(model_path) 获取保存的模型参数的dict和优化器信息的dict。

再使用your_modle_object.set_dict(para_dict)接口来恢复保存的模型参数从而达到继续训练的目的。

以及使用your_optimizer_object.set_dict(opti_dict)接口来恢复保存的优化器中的learning rate decay值。

下面的代码展示了如何在“手写数字识别”任务中保存参数并且读取已经保存的参数来继续训练。

  1. import paddle.fluid as fluid
  2.  
  3. with fluid.dygraph.guard():
  4. epoch_num = 5
  5. BATCH_SIZE = 64
  6.  
  7. mnist = MNIST("mnist")
  8. adam = fluid.optimizer.Adam(learning_rate=0.001)
  9. train_reader = paddle.batch(
  10. paddle.dataset.mnist.train(), batch_size= BATCH_SIZE, drop_last=True)
  11.  
  12. np.set_printoptions(precision=3, suppress=True)
  13. dy_param_init_value={}
  14. for epoch in range(epoch_num):
  15. for batch_id, data in enumerate(train_reader()):
  16. dy_x_data = np.array(
  17. [x[0].reshape(1, 28, 28)
  18. for x in data]).astype('float32')
  19. y_data = np.array(
  20. [x[1] for x in data]).astype('int64').reshape(BATCH_SIZE, 1)
  21.  
  22. img = fluid.dygraph.to_variable(dy_x_data)
  23. label = fluid.dygraph.to_variable(y_data)
  24. label.stop_gradient = True
  25.  
  26. cost = mnist(img)
  27. loss = fluid.layers.cross_entropy(cost, label)
  28. avg_loss = fluid.layers.mean(loss)
  29.  
  30. dy_out = avg_loss.numpy()
  31.  
  32. avg_loss.backward()
  33. adam.minimize(avg_loss)
  34. if batch_id == 20:
  35. fluid.dygraph.save_dygraph(mnist.state_dict(), "paddle_dy")
  36. mnist.clear_gradients()
  37.  
  38. if batch_id == 20:
  39. for param in mnist.parameters():
  40. dy_param_init_value[param.name] = param.numpy()
  41. model, _ = fluid.dygraph.load_dygraph("paddle_dy")
  42. mnist.set_dict(model)
  43. break
  44. if epoch == 0:
  45. break
  46. restore = mnist.parameters()
  47. # check save and load
  48.  
  49. success = True
  50. for value in restore:
  51. if (not np.array_equal(value.numpy(), dy_param_init_value[value.name])) or (not np.isfinite(value.numpy().all())) or (np.isnan(value.numpy().any())):
  52. success = False
  53. print("model save and load success? {}".format(success))

需要注意的是,如果采用多卡训练,只需要一个进程对模型参数进行保存,因此在保存模型参数时,需要进行指定保存哪个进程的参数,比如

  1. if fluid.dygraph.parallel.Env().local_rank == 0:
  2. fluid.dygraph.save_dygraph(mnist.state_dict(), "paddle_dy")

模型评估

当我们需要在DyGraph模式下利用搭建的模型进行预测任务,可以使用YourModel.eval()接口,在之前的手写数字识别模型中我们使用mnist.eval()来启动预测模式(我们默认在fluid.dygraph.guard()上下文中是训练模式),在预测的模式下,DyGraph将只会执行前向的预测网络,而不会进行自动求导并执行反向网络:

下面的代码展示了如何使用DyGraph模式训练一个用于执行“手写数字识别”任务的模型并保存,并且利用已经保存好的模型进行预测。

我们在fluid.dygraph.guard()上下文中进行了模型的保存和训练,值得注意的是,当我们需要在训练的过程中进行预测时需要使用YourModel.eval()切换到预测模式,并且在预测完成后使用YourModel.train()切换回训练模式继续训练。

我们在inference_mnist 中启用另一个fluid.dygraph.guard(),并在其上下文中load之前保存的checkpoint进行预测,同样的在执行预测前需要使用YourModel.eval()来切换到预测模式。

  1. def test_mnist(reader, model, batch_size):
  2. acc_set = []
  3. avg_loss_set = []
  4. for batch_id, data in enumerate(reader()):
  5. dy_x_data = np.array([x[0].reshape(1, 28, 28)
  6. for x in data]).astype('float32')
  7. y_data = np.array(
  8. [x[1] for x in data]).astype('int64').reshape(batch_size, 1)
  9.  
  10. img = fluid.dygraph.to_variable(dy_x_data)
  11. label = fluid.dygraph.to_variable(y_data)
  12. label.stop_gradient = True
  13. prediction, acc = model(img, label)
  14. loss = fluid.layers.cross_entropy(input=prediction, label=label)
  15. avg_loss = fluid.layers.mean(loss)
  16. acc_set.append(float(acc.numpy()))
  17. avg_loss_set.append(float(avg_loss.numpy()))
  18.  
  19. # get test acc and loss
  20. acc_val_mean = np.array(acc_set).mean()
  21. avg_loss_val_mean = np.array(avg_loss_set).mean()
  22.  
  23. return avg_loss_val_mean, acc_val_mean
  24.  
  25. def inference_mnist():
  26. with fluid.dygraph.guard():
  27. mnist_infer = MNIST("mnist")
  28. # load checkpoint
  29. model_dict, _ = fluid.dygraph.load_dygraph("paddle_dy")
  30. mnist_infer.load_dict(model_dict)
  31. print("checkpoint loaded")
  32.  
  33. # start evaluate mode
  34. mnist_infer.eval()
  35.  
  36. def load_image(file):
  37. im = Image.open(file).convert('L')
  38. im = im.resize((28, 28), Image.ANTIALIAS)
  39. im = np.array(im).reshape(1, 1, 28, 28).astype(np.float32)
  40. im = im / 255.0 * 2.0 - 1.0
  41. return im
  42.  
  43. cur_dir = os.path.dirname(os.path.realpath(__file__))
  44. tensor_img = load_image(cur_dir + '/image/infer_3.png')
  45.  
  46. results = mnist_infer(fluid.dygraph.to_variable(tensor_img))
  47. lab = np.argsort(results.numpy())
  48. print("Inference result of image/infer_3.png is: %d" % lab[0][-1])
  49.  
  50. with fluid.dygraph.guard():
  51. epoch_num = 1
  52. BATCH_SIZE = 64
  53. mnist = MNIST("mnist")
  54. adam = fluid.optimizer.AdamOptimizer(learning_rate=0.001)
  55. test_reader = paddle.batch(
  56. paddle.dataset.mnist.test(), batch_size=BATCH_SIZE, drop_last=True)
  57.  
  58. train_reader = paddle.batch(
  59. paddle.dataset.mnist.train(),
  60. batch_size=BATCH_SIZE,
  61. drop_last=True)
  62.  
  63. for epoch in range(epoch_num):
  64. for batch_id, data in enumerate(train_reader()):
  65. dy_x_data = np.array([x[0].reshape(1, 28, 28)
  66. for x in data]).astype('float32')
  67. y_data = np.array(
  68. [x[1] for x in data]).astype('int64').reshape(-1, 1)
  69.  
  70. img = fluid.dygraph.to_variable(dy_x_data)
  71. label = fluid.dygraph.to_variable(y_data)
  72. label.stop_gradient = True
  73.  
  74. cost, acc = mnist(img, label)
  75.  
  76. loss = fluid.layers.cross_entropy(cost, label)
  77. avg_loss = fluid.layers.mean(loss)
  78.  
  79. avg_loss.backward()
  80.  
  81. adam.minimize(avg_loss)
  82. # save checkpoint
  83. mnist.clear_gradients()
  84. if batch_id % 100 == 0:
  85. print("Loss at epoch {} step {}: {:}".format(
  86. epoch, batch_id, avg_loss.numpy()))
  87.  
  88. mnist.eval()
  89. test_cost, test_acc = test_mnist(test_reader, mnist, BATCH_SIZE)
  90. mnist.train()
  91. print("Loss at epoch {} , Test avg_loss is: {}, acc is: {}".format(
  92. epoch, test_cost, test_acc))
  93.  
  94. fluid.dygraph.save_dygraph(mnist.state_dict(), "paddle_dy")
  95. print("checkpoint saved")
  96.  
  97. inference_mnist()

输出:

  1. Loss at epoch 0 step 0: [2.2991252]
  2. Loss at epoch 0 step 100: [0.15491392]
  3. Loss at epoch 0 step 200: [0.13315125]
  4. Loss at epoch 0 step 300: [0.10253005]
  5. Loss at epoch 0 step 400: [0.04266362]
  6. Loss at epoch 0 step 500: [0.08894891]
  7. Loss at epoch 0 step 600: [0.08999012]
  8. Loss at epoch 0 step 700: [0.12975612]
  9. Loss at epoch 0 step 800: [0.15257305]
  10. Loss at epoch 0 step 900: [0.07429226]
  11. Loss at epoch 0 , Test avg_loss is: 0.05995981965082674, acc is: 0.9794671474358975
  12. checkpoint saved
  13. No optimizer loaded. If you didn't save optimizer, please ignore this. The program can still work with new optimizer.
  14. checkpoint loaded
  15. Inference result of image/infer_3.png is: 3

编写兼容的模型

以上一步中手写数字识别的例子为例,动态图的模型代码可以直接用于静态图中作为模型代码,执行时,直接使用PaddlePaddle静态图执行方式即可,这里以静态图中的executor为例, 模型代码可以直接使用之前的模型代码,执行时使用Executor执行即可

  1. epoch_num = 1
  2. BATCH_SIZE = 64
  3. exe = fluid.Executor(fluid.CPUPlace())
  4.  
  5. mnist = MNIST("mnist")
  6. sgd = fluid.optimizer.SGDOptimizer(learning_rate=1e-3)
  7. train_reader = paddle.batch(
  8. paddle.dataset.mnist.train(), batch_size=BATCH_SIZE, drop_last=True)
  9. img = fluid.layers.data(
  10. name='pixel', shape=[1, 28, 28], dtype='float32')
  11. label = fluid.layers.data(name='label', shape=[1], dtype='int64')
  12. cost = mnist(img)
  13. loss = fluid.layers.cross_entropy(cost, label)
  14. avg_loss = fluid.layers.mean(loss)
  15. sgd.minimize(avg_loss)
  16.  
  17. out = exe.run(fluid.default_startup_program())
  18.  
  19. for epoch in range(epoch_num):
  20. for batch_id, data in enumerate(train_reader()):
  21. static_x_data = np.array(
  22. [x[0].reshape(1, 28, 28)
  23. for x in data]).astype('float32')
  24. y_data = np.array(
  25. [x[1] for x in data]).astype('int64').reshape([BATCH_SIZE, 1])
  26.  
  27. fetch_list = [avg_loss.name]
  28. out = exe.run(
  29. fluid.default_main_program(),
  30. feed={"pixel": static_x_data,
  31. "label": y_data},
  32. fetch_list=fetch_list)
  33.  
  34. static_out = out[0]
  35.  
  36. if batch_id % 100 == 0 and batch_id is not 0:
  37. print("epoch: {}, batch_id: {}, loss: {}".format(epoch, batch_id, static_out))