noam_decay
paddle.fluid.layers.
noam_decay
(d_model, warmup_steps)[源代码]
Noam衰减方法
noam衰减的numpy实现如下:
- import paddle.fluid as fluid
- import numpy as np
- # 设置超参数
- d_model = 2
- current_steps = 20
- warmup_steps = 200
- # 计算
- lr_value = np.power(d_model, -0.5) * np.min([
- np.power(current_steps, -0.5),
- np.power(warmup_steps, -1.5) * current_steps])
- 参数:
- d_model (Variable|int) - 模型的输入、输出向量特征维度。类型可设置为标量Tensor,或int值。
- warmup_steps (Variable|int) - 预热步数,类型可设置为标量Tensor,或int值。
返回:衰减的学习率
返回类型: Variable
代码示例:
- import paddle.fluid as fluid
- warmup_steps = 100
- learning_rate = 0.01
- lr = fluid.layers.learning_rate_scheduler.noam_decay(
- 1/(warmup_steps *(learning_rate ** 2)),
- warmup_steps)