Scikit Transformers Examples

Here we outline some more complicated transformers and transformers that require additional processing to work nicely with Pipelines and Feature Unions.

Label Encoder

The LabelEncoder is synonymous with StringIndexer in Spark, however there are a couple of unique features of the scikit transformer that we need to account for:

  1. LabelEncoder only opperates on a single feature at a time
  2. The output of the LabelEncoder is a numpy array of shape (1,n) instead of (n,1), which is required for further processing like One-Hot-Encoding

Here is what an example Pipeline looks like for a LabelEncoder

  1. # Create a dataframe with some a categorical and a continuous feature
  2. df = pd.DataFrame(np.array([ ['Alice', 32], ['Jack', 18], ['Bob',34]]), columns=['name', 'age'])
  4. # Define our feature extractor
  5. feature_extractor_tf = FeatureExtractor(input_scalars=['name'], 
  6.                                         output_vector='name_continuous_feature', 
  7.                                         output_vector_items=['name_label_encoded'])
  9. # Label Encoder for x1 Label 
  10. label_encoder_tf = LabelEncoder()
  11. label_encoder_tf.mlinit(input_features = feature_extractor_tf.output_vector_items, output_features='name_label_le')
  13. # Reshape the output of the LabelEncoder to N-by-1 array
  14. reshape_le_tf = ReshapeArrayToN1()
  16. # Create our pipeline object and initialize MLeap Serialization
  17. le_pipeline = Pipeline([(feature_extractor_tf.name, feature_extractor_tf),
  18.                         (label_encoder_tf.name, label_encoder_tf),
  19.                         (reshape_le_tf.name, reshape_le_tf)
  20.                         ])
  21. le_pipeline.mlinit()
  23. # Transform our DataFrame
  24. le_pipeline.fit_transform(df)
  26. # output
  27. array([[0],
  28.        [2],
  29.        [1]])

Next step is to combine the label indexer with a OneHotEncoder

Scikit OneHotEncoder

We’ll continue the example above to demonstrate how the out-of-the-box Scikit OneHotEncoder works.

  1. ## Vector Assembler for x1 One Hot Encoder
  2. one_hot_encoder_tf = OneHotEncoder(sparse=False) # Make sure to set sparse=False
  3. one_hot_encoder_tf.mlinit(prior_tf=label_encoder_tf, output_features = '{}_one_hot_encoded'.format(label_encoder_tf.output_features))
  4. #
  6. # Construct our pipeline
  7. one_hot_encoder_pipeline_x0 = Pipeline([
  8.                                          (feature_extractor_tf.name, feature_extractor_tf),
  9.                                          (label_encoder_tf.name, label_encoder_tf),
  10.                                          (reshape_le_tf.name, reshape_le_tf),
  11.                                          (one_hot_encoder_tf.name, one_hot_encoder_tf)
  12.                                         ])
  14. one_hot_encoder_pipeline_x0.mlinit()
  16. # Execute our LabelEncoder + OneHotEncoder pipeline on our dataframe
  17. one_hot_encoder_pipeline_x0.fit_transform(df)
  18. matrix([[ 1.,  0.,  0.],
  19.         [ 0.,  0.,  1.],
  20.         [ 0.,  1.,  0.]])

One of the short-comings of Scikit’s OneHotEncoder is that it’s missing a drop_last functionality that’s required in ML pipelines.MLeap comes with it’s own OneHotEncoder that enables that function

MLeap OneHotEncoder

Very similar to the Scikit OneHotEncoder, except we set an additional drop_last attribute.

  1. from mleap.sklearn.extensions.data import OneHotEncoder
  3. ## Vector Assembler for x1 One Hot Encoder
  4. one_hot_encoder_tf = OneHotEncoder(sparse=False, drop_last=True) # Make sure to set sparse=False
  5. one_hot_encoder_tf.mlinit(prior_tf=label_encoder_tf, output_features = '{}_one_hot_encoded'.format(label_encoder_tf.output_features))
  6. #
  8. # Construct our pipeline
  9. one_hot_encoder_pipeline_x0 = Pipeline([
  10.                                          (feature_extractor_tf.name, feature_extractor_tf),
  11.                                          (label_encoder_tf.name, label_encoder_tf),
  12.                                          (reshape_le_tf.name, reshape_le_tf),
  13.                                          (one_hot_encoder_tf.name, one_hot_encoder_tf)
  14.                                         ])
  16. one_hot_encoder_pipeline_x0.mlinit()
  18. # Execute our LabelEncoder + OneHotEncoder pipeline on our dataframe
  19. one_hot_encoder_pipeline_x0.fit_transform(df)
  20. matrix([[ 1.,  0.],
  21.         [ 0.,  0.],
  22.         [ 0.,  1.]])