noam_decay

  • paddle.fluid.layers.noam_decay(d_model, warmup_steps)[源代码]

Noam衰减方法

noam衰减的numpy实现如下:

  1. import paddle.fluid as fluid
  2. import numpy as np
  3. # 设置超参数
  4. d_model = 2
  5. current_steps = 20
  6. warmup_steps = 200
  7. # 计算
  8. lr_value = np.power(d_model, -0.5) * np.min([
  9. np.power(current_steps, -0.5),
  10. np.power(warmup_steps, -1.5) * current_steps])

请参照 attention is all you need

  • 参数:
    • d_model (Variable|int) - 模型的输入、输出向量特征维度。类型可设置为标量Tensor,或int值。
    • warmup_steps (Variable|int) - 预热步数,类型可设置为标量Tensor,或int值。

返回:衰减的学习率

返回类型: Variable

代码示例

  1. import paddle.fluid as fluid
  2. warmup_steps = 100
  3. learning_rate = 0.01
  4. lr = fluid.layers.learning_rate_scheduler.noam_decay(
  5. 1/(warmup_steps *(learning_rate ** 2)),
  6. warmup_steps)