PNN

1. 算法介绍

FNN(Product-Based Neural Networks)算法是在Embedding的基础上, 对Embedding
的结果进行两两内积或外积, 然后将内/外积结果与原始的Embedding结果拼接起来输入DNN进一步提取高阶特特交叉. 值得注意的是, PNN并没有放弃一阶特征, 最后将一阶特征与高阶特征组合起来进行预测, 其构架如下:

PNN

注: 目前Angel只实现了内积形式的PNN.

1.1 BiInnerCross层的说明

在实现中, 用Embedding的方式存储Product Neural Network(PNN) - 图2, 调用Embedding的calOutput后, 将Product Neural Network(PNN) - 图3计算后一起输出, 所以一个样本的Embedding output结果为:

model=(\bold{u}_1,\bold{u}_2,\bold{u}_3,\cdots,\bold{u}_k))

对Embedding特征两两做内积有:

model)

以上即是BiInnerCross的前向计算方式, 用Scala代码实现为:

  1. (0 until batchSize).foreach { row =>
  2.     val partitions = mat.getRow(row).getPartitions
  3.     var opIdx = 0
  4.     partitions.zipWithIndex.foreach { case (vector_outter, cidx_outter) =>
  5.     if (cidx_outter != partitions.length - 1) {
  6.         ((cidx_outter + 1) until partitions.length).foreach { cidx_inner =>
  7.         data(row * outputDim + opIdx) = vector_outter.dot(partitions(cidx_inner))
  8.         opIdx += 1
  9.         }
  10.     }
  11.     }
  12. }

BiInnerCross与BiInnerSumCross的区别在于后者将两两内积的结果加和起来输出为一个标量, 前者没有加和起来, 输出是一个向量. 对于BiInnerCross, 输出的维数为Product Neural Network(PNN) - 图6为field的个数, 与Embedding向量的维数无关.

1.2 其它层说明

  • SparseInputLayer: 稀疏数据输入层, 对稀疏高维数据做了特别优化, 本质上是一个FCLayer
  • Embedding: 隐式嵌入层, 如果特征非one-hot, 则乘以特征值
  • FCLayer: DNN中最常见的层, 线性变换后接传递函数
  • SumPooling: 将多个输入的数据做element-wise的加和, 要求输入具本相同的shape
  • SimpleLossLayer: 损失层, 可以指定不同的损失函数

1.3 网络构建

  1.   override def buildNetwork(): Unit = {
  2.     val wide = new SparseInputLayer("input", 1, new Identity(),
  3.       JsonUtils.getOptimizerByLayerType(jsonAst, "SparseInputLayer"))
  5.     val embeddingParams = JsonUtils.getLayerParamsByLayerType(jsonAst, "Embedding")
  6.       .asInstanceOf[EmbeddingParams]
  7.     val embedding = new Embedding("embedding", embeddingParams.outputDim, embeddingParams.numFactors,
  8.       embeddingParams.optimizer.build()
  9.     )
  11.     val crossOutputDim = numFields * (numFields - 1) / 2
  12.     val innerCross = new BiInnerCross("innerPooling", crossOutputDim, embedding)
  14.     val concatOutputDim = embeddingParams.outputDim + crossOutputDim
  15.     val concatLayer = new ConcatLayer("concatMatrix", concatOutputDim, Array[Layer](embedding, innerCross))
  17.     val hiddenLayers = JsonUtils.getFCLayer(jsonAst, concatLayer)
  19.     val join = new SumPooling("sumPooling", 1, Array[Layer](wide, hiddenLayers))
  21.     new SimpleLossLayer("simpleLossLayer", join, lossFunc)
  22.   }

2. 运行与性能

2.1 Json配置文件说明

PNN的参数较多, 需要用Json配置文件的方式指定(关于Json配置文件的完整说明请参考Json说明), 一个典型的例子如下:

  1. {
  2.   "data": {
  3.     "format": "dummy",
  4.     "indexrange": 148,
  5.     "numfield": 13,
  6.     "validateratio": 0.1
  7.   },
  8.   "model": {
  9.     "modeltype": "T_FLOAT_SPARSE_LONGKEY",
  10.     "modelsize": 148
  11.   },
  12.   "train": {
  13.     "epoch": 10,
  14.     "numupdateperepoch": 10,
  15.     "lr": 0.01,
  16.     "decay": 0.1
  17.   },
  18.   "default_optimizer": "Momentum",
  19.   "layers": [
  20.     {
  21.       "name": "wide",
  22.       "type": "sparseinputlayer",
  23.       "outputdim": 1,
  24.       "transfunc": "identity"
  25.     },
  26.     {
  27.       "name": "embedding",
  28.       "type": "embedding",
  29.       "numfactors": 8,
  30.       "outputdim": 104,
  31.       "optimizer": {
  32.         "type": "momentum",
  33.         "momentum": 0.9,
  34.         "reg2": 0.01
  35.       }
  36.     },
  37.     {
  38.       "name": "biInnerCross",
  39.       "type": "BiInnerCross",
  40.       "outputdim": 78,
  41.       "inputlayer": "embedding"
  42.     },
  43.     {
  44.       "name": "concatlayer",
  45.       "type": "ConcatLayer",
  46.       "outputdim": 182,
  47.       "inputlayers": [
  48.         "embedding",
  49.         "biInnerCross"
  50.       ]
  51.     },
  52.     {
  53.       "name": "fclayer",
  54.       "type": "FCLayer",
  55.       "outputdims": [
  56.         200,
  57.         200,
  58.         1
  59.       ],
  60.       "transfuncs": [
  61.         "relu",
  62.         "relu",
  63.         "identity"
  64.       ],
  65.       "inputlayer": "concatlayer"
  66.     },
  67.     {
  68.       "name": "sumPooling",
  69.       "type": "SumPooling",
  70.       "outputdim": 1,
  71.       "inputlayers": [
  72.         "wide",
  73.         "fclayer"
  74.       ]
  75.     },
  76.     {
  77.       "name": "simplelosslayer",
  78.       "type": "simplelosslayer",
  79.       "lossfunc": "logloss",
  80.       "inputlayer": "sumPooling"
  81.     }
  82.   ]
  83. }

2.2 提交脚本说明

  1. runner="com.tencent.angel.ml.core.graphsubmit.GraphRunner"
  2. modelClass="com.tencent.angel.ml.classification.ProductNeuralNetwork"
  4. $ANGEL_HOME/bin/angel-submit \
  5.     --angel.job.name DeepFM \
  6.     --action.type train \
  7.     --angel.app.submit.class $runner \
  8.     --ml.model.class.name $modelClass \
  9.     --angel.train.data.path $input_path \
  10.     --angel.workergroup.number $workerNumber \
  11.     --angel.worker.memory.gb $workerMemory  \
  12.     --angel.ps.number $PSNumber \
  13.     --angel.ps.memory.gb $PSMemory \  
  14.     --angel.task.data.storage.level $storageLevel \
  15.     --angel.task.memorystorage.max.gb $taskMemory

对深度学习模型, 其数据, 训练和网络的配置请优先使用Json文件指定.