PyAngel快速入门

环境

  • Linux任意发行版本,CentOS，Ubuntu等均可
  • Angel >= 1.3
  • Python >= 2.7 / 3.6（PyAngel1.4支持python2，python1.4放弃对python2的支持，只支持python3）

编写和编译

  1. 编写: 推荐使用Atom或者pyCharm，高手请自备Vim或者Emacs
  2. 编译：参考Angel编译指南，注意Python的版本

提交任务

PyAngel支持交互式和脚本式两种提交任务的模式，而每种提交任务的模式，都支持2种运行模式：local & Yarn。Yarn模式依赖Hadoop，需要在提交机器上将Hadoop安装好，并且保证HADOOP_HOME设置正确，具体可以参考Yarn运行模式

  • 交互式

    • Local模式

      1. bin/pyangel local

    • Yarn模式

      1. bin/pyangel

  • 脚本式

    • Local模式

      1.  bin/angel-local-submit --angel.pyangel.pyfile ${ANGEL_HOME}/python/examples/gbdt_local_example/py

    • Yarn模式

      1.  bin/angel-submit --angel.pyangel.pyfile ${ANGEL_HOME}/python/examples/gbdt_example.py

样例命令

  • Local模式提交

    1. bin/angel-local-submit \
    2.   --angel.pyangel.pyfile ${ANGEL_HOME}/python/examples/gbdt_local_example.py \
    3.   --angel.train.data.path "file:///${ANGEL_HOME}/data/exampledata/GBDTLocalExampleData/agaricus.txt.train" \
    4.   --angel.log.path "file:///${ANGEL_HOME}/data/log" \
    5.   --angel.save.model.path "file:///${ANGEL_HOME}/data/output"

Example Code

  • PyAngel版本的GBDT

可以通过运行bin/pyangel local命令启动PyAngel本地交互式命令行，然后在命令行中输入下面的代码，运行GBDTRunner，注意：需要将input_path中的${YOUR_ANGERL_HOME}修改为你自己的angel绝对安装路径 

  2. from pyangel.ml.gbdt.runner import GBDTRunner
  4. # Trainning data input path
  5. input_path = "file:///${YOUR_ANGEL_HOME}/data/exampledata/GBDTLocalExampleData/agaricus.txt.train"
  7. # Algo param
  8. feature_num = 127
  9. feature_nzz = 25
  10. tree_num = 2
  11. tree_depth = 2
  12. split_num = 10
  13. sample_ratio = 1.0
  15. # Data format
  16. data_fmt = "libsvm"
  18. # Learning rate
  19. learn_rate = 0.01
  21. # Set GBDT training data path
  22. conf[AngelConf.ANGEL_TRAIN_DATA_PATH] = input_path
  24. # Set GBDT algorithm parameters
  25. conf[MLConf.ML_FEATURE_NUM] = str(feature_num)
  26. conf[MLConf.ML_FEATURE_NNZ] = str(feature_nzz)
  27. conf[MLConf.ML_GBDT_TREE_NUM] = str(tree_num)
  28. conf[MLConf.ML_GBDT_TREE_DEPTH] = str(tree_depth)
  29. conf[MLConf.ML_GBDT_SPLIT_NUM] = str(split_num)
  30. conf[MLConf.ML_GBDT_SAMPLE_RATIO] = str(sample_ratio)
  31. conf[MLConf.ML_LEARN_RATE] = str(learn_rate)
  33. runner = GBDTRunner()
  34. runner.train(conf)

或者也可以通过创建一个字典的方式将参数传入:

  1. cate_feat = "0:2,1:2,2:2,3:2,4:2,5:2,6:2,7:2,8:2,9:2,10:2,11:2,12:2,13:2,14:2,15:2,16:2,17:2,18:2,19:2,20:2," \
  2.     "21:2,22:2,23:2,24:2,25:2,26:2,27:2,28:2,29:2,30:2,31:2,32:2,33:2,34:2,35:2,36:2,37:2,38:2,39:2,40:2," \
  3.     "41:2,42:2,43:2,44:2,45:2,46:2,47:2,48:2,49:2,50:2,51:2,52:2,53:2,54:2,55:2,56:2,57:2,58:2,59:2,60:2," \
  4.     "61:2,62:2,63:2,64:2,65:2,66:2,67:2,68:2,69:2,70:2,71:2,72:2,73:2,74:2,75:2,76:2,77:2,78:2,79:2,80:2," \
  5.     "81:2,82:2,83:2,84:2,85:2,86:2,87:2,88:2,89:2,90:2,91:2,92:2,93:2,94:2,95:2,96:2,97:2,98:2,99:2,100:2," \
  6.     "101:2,102:2,103:2,104:2,105:2,106:2,107:2,108:2,109:2,110:2,111:2,112:2,113:2,114:2,115:2,116:2,117:2," \
  7.     "118:2,119:2,120:2,121:2,122:2,123:2,124:2,125:2,126:2"
  8. params = {
  9.     AngelConf.ANGEL_DEPLOY_MODE: 'LOCAL',
  10.     'mapred.mapper.new-api': True,
  11.     AngelConf.ANGEL_INPUTFORMAT_CLASS: 'org.apache.hadoop.mapreduce.lib.input.CombineTextInputFormat',
  12.     AngelConf.ANGEL_JOB_OUTPUT_PATH_DELETEONEXIST: True,
  13.     AngelConf.ANGEL_WORKERGROUP_NUMBER: 1,
  14.     AngelConf.ANGEL_WORKER_TASK_NUMBER: 1,
  15.     AngelConf.ANGEL_PS_NUMBER: 1,
  16.     MLConf.ML_DATA_FORMAT: 'libsvm',
  17.     MLConf.ML_FEATURE_NUM: 127,
  18.     MLConf.ML_FEATURE_NNZ: 25,
  19.     MLConf.ML_GBDT_TREE_NUM: 2,
  20.     MLConf.ML_GBDT_TREE_DEPTH: 2,
  21.     MLConf.ML_GBDT_SPLIT_NUM: 10,
  22.     MLConf.ML_GBDT_SAMPLE_RATIO: 1.0,
  23.     MLConf.ML_LEARN_RATE: 0.01,
  24.     MLConf.ML_GBDT_CATE_FEAT: cate_feat
  25. }
  26. self.conf.update(params)
  28. runner = GBDTRunner()
  29. runner.train(conf)

新版本

支持自定义Model，Task等操作，以及和Spark配合的相关功能正在开发中，如有疑问以及需求，欢迎提Issue和PR，或者联系Angel8号