Introduction to Pipeline

In Pipcook, we use Pipeline to represent the training process of a model, so in general, what kind of pipeline is needed to train a model? The developer can use a JSON to describe pipeline of modeling from sample collection, model definition, training to model evaluation:

  1. {
  2.   "plugins": {
  3.     "dataCollect": {
  4.       "package": "@pipcook/plugins-csv-data-collect",
  5.       "params": {
  6.         "url": "http://foobar"
  7.       }
  8.     },
  9.     "dataAccess": {
  10.       "package": "@pipcook/plugins-csv-data-access",
  11.       "params": {
  12.         "labelColumn": "output"
  13.       }
  14.     },
  15.     "modelDefine": {
  16.       "package": "@pipcook/plugins-bayesian-model-define"
  17.     },
  18.     "modelTrain": {
  19.       "package": "@pipcook/plugins-bayesian-model-train"
  20.     },
  21.     "modelEvaluate": {
  22.       "package": "@pipcook/plugins-bayesian-model-evaluate"
  23.     }
  24.   }
  25. }

As shown above, a pipeline is composed of different plugins, and we add the field params to each plugin to pass given parameters. Then the pipeline interpreter will perform the corresponding operation(s) by its plugin type and parameters.

See Introduction to Plugin for more details about plugin.

Next, when we have defined such a pipeline, we can run it through Pipcook.

Preparation

Follow the Pipcook Tools Initlization to get the Pipcook ready.

Run Pipeline

Save the above JSON of your pipeline in anywhere, and run:

  1. $ pipcook run /path/to/your/pipeline-config.json

The trained model will generate an output directory under cwd(3):

  1. 📂output
  2.    ┣ 📂logs
  3.    ┣ 📂model
  4.    ┣ 📜package.json
  5.    ┣ 📜metadata.json
  6.    ┗ 📜index.js

To get started with your trained model, follow the below steps:

  1. $ npm install

It will install dependencies which contain the plugins and Python packages. Pipcook provides a way to use tuna mirror when it downloads Python and packages:

  1. $ BOA_TUNA=1 npm install

Once the output is initialized, just import it as the following:

  1. import * as predict from './output';
  2. predict('your input data');