Define Models for SQLFlow

SQLFlow enables SQL programs to call deep learning models defined in Python. This document is about how to define models for SQLFlow.

Keras v.s. Estimator

Many deep learners define models using Keras API or as an Estimator derived class. We prefer Keras over Estimator for some reasons:

  1. TensorFlow Dev Summit 2019 announced that TensorFlow 2.x will closely integrate with Keras.

  2. We found more documents about Keras than Estimator.

  3. We found more models defined using Keras than Estimator.

Keras APIs

Keras provides three approaches to define models.

1. Subclassing tf.keras.Model

  1.   class DNNClassifier(tf.keras.Model):
  2.       def __init__(self, feature_columns, hidden_units, n_classes):
  3.           super(DNNClassifier, self).__init__()
  4.           self.feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
  5.           self.hidden_layers = []
  6.           for hidden_unit in hidden_units:
  7.               self.hidden_layers.append(tf.keras.layers.Dense(hidden_unit))
  8.           self.prediction_layer = tf.keras.layers.Dense(n_classes, activation='softmax')
  10.       def call(self, inputs):
  11.           x = self.feature_layer(inputs)
  12.           for hidden_layer in self.hidden_layers:
  13.               x = hidden_layer(x)
  14.           return self.prediction_layer(x)
  16.   model = DNNClassifier(feature_columns, hidden_units, n_classes)

Please be aware that tf.keras.Model has methods save_weights and load_weights, which save/load model parameters but no the topology, as explained in this guidance and this example list.

2. Functional API

  1.   x = tf.feature_column.input_layer(shape=(5,))
  2.   for n in hidden_units:
  3.       x = tf.keras.layers.Dense(n, activation='relu')(x)
  4.   pred = tf.keras.layers.Dense(n_classes, activation='softmax')(x)
  5.   model = tf.keras.models.Model(inputs=feature_columns, outputs=pred)

The functional API can work with feature column API only by assigning tf.keras.Input to each original feature column. See thislink for an example.

3. keras.Sequential

  1.   model = tf.keras.Sequential()
  2.   model.add(tf.keras.layers.DenseFeatures(feature_columns))
  3.   for n in hidden_units:
  4.     model.add(tf.keras.layers.Dense(n, activation='relu'))
  5.   model.add(tf.keras.layers.Dense(n_classes, activation='softmax'))

Please be aware that tf.keras.Sequential() only covers a small variety of models. It doesn’t cover many well-known models including ResNet, Transforms, and WideAndDeep.

The Choice

We chose the approach of subclassing tf.keras.Model according to the following table.

Keras APIsWork with feature column APISave/load modelsModel coverage
tf.keras.Model☑️weights-only, no topologyHigh
Functional API☑️☑️High
Sequential Model☑️☑️Low

A Subclass model Example

Here is an example DNNClassifier of multiple hidden layers as a Python class derived from tf.keras.Model. To run it, please use TensorFlow 2.0 alpha or newer versions.

  1. class DNNClassifier(tf.keras.Model):
  2.     def __init__(self, feature_columns, hidden_units, n_classes):
  3.         """DNNClassifier
  4.         :param feature_columns: feature columns.
  5.         :type feature_columns: list[tf.feature_column].
  6.         :param hidden_units: number of hidden units.
  7.         :type hidden_units: list[int].
  8.         :param n_classes: List of hidden units per layer.
  9.         :type n_classes: int.
  10.         """
  11.         super(DNNClassifier, self).__init__()
  13.         # combines all the data as a dense tensor
  14.         self.feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
  15.         self.hidden_layers = []
  16.         for hidden_unit in hidden_units:
  17.             self.hidden_layers.append(tf.keras.layers.Dense(hidden_unit))
  18.         self.prediction_layer = tf.keras.layers.Dense(n_classes, activation='softmax')
  20.     def call(self, inputs):
  21.         x = self.feature_layer(inputs)
  22.         for hidden_layer in self.hidden_layers:
  23.             x = hidden_layer(x)
  24.         return self.prediction_layer(x)
  26.     def default_optimizer(self):
  27.         """Default optimizer name. Used in model.compile."""
  28.         return 'adam'
  30.     def default_loss(self):
  31.         """Default loss function. Used in model.compile."""
  32.         return 'categorical_crossentropy'
  34.     def default_training_epochs(self):
  35.         """Default training epochs. Used in model.fit."""
  36.         return 5
  38.     def prepare_prediction_column(self, prediction):
  39.         """Return the class label of highest probability."""
  40.         return prediction.argmax(axis=-1)

A Functional API model Example

  1. def MyExampleModel(feature_columns, field_metas, learning_rate=0.01):
  2.     feature_layer_inputs = dict()
  3.     for fm in field_metas:
  4.         feature_layer_inputs[fm.name] = tf.keras.Input(shape=(fm.shape), name=fm.name, dtype=fm.dtype)
  5.     feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
  6.     feature_layer_outputs = feature_layer(feature_layer_inputs)
  8.     x = layers.Dense(128, activation='relu')(feature_layer_outputs)
  9.     x = layers.Dense(64, activation='relu')(x)
  10.     pred = layers.Dense(1, activation='sigmoid')(x)
  11.     return keras.Model(inputs=[v for v in feature_layer_inputs.values()], outputs=pred)
  13. def loss(output, labels):
  14.     labels = tf.reshape(labels, [-1])
  15.     return tf.reduce_mean(
  16.         input_tensor=tf.nn.sparse_softmax_cross_entropy_with_logits(
  17.             logits=output, labels=labels
  18.         )
  19.     )
  21. def optimizer(lr=0.1):
  22.     return tf.optimizers.SGD(lr)
  24. def prepare_prediction_column(self, prediction):
  25.     """Return the class label of highest probability."""
  26.     return prediction.argmax(axis=-1)

Further Reading

We read the following Keras source code files: models.py, network.py, and training.py.