6-1 Three Ways of Modeling

There are three ways of modeling: using Sequential to construct model with the order of layers, using functional APIs to construct model with arbitrary structure, using child class inheriting from the base class Model.

For the models with sequenced structure, Sequential method should be given the highest priority.

For the models with nonsequenced structures such as multiple input/output, shared weights, or residual connections, modeling with functional API is recommended.

Modeling through child class of Model should be AVOIDED unless with special requirements. This method is flexible, but also fallible.

Here is are the examples of modeling using the three above-mentioned methods to classify IMDB movie reviews.

  1. import numpy as np 
  2. import pandas as pd 
  3. import tensorflow as tf
  4. from tqdm import tqdm 
  5. from tensorflow.keras import *
  8. train_token_path = "../data/imdb/train_token.csv"
  9. test_token_path = "../data/imdb/test_token.csv"
  11. MAX_WORDS = 10000  # We will only consider the top 10,000 words in the dataset
  12. MAX_LEN = 200  # We will cut reviews after 200 words
  13. BATCH_SIZE = 20 
  15. # Constructing data pipeline
  16. def parse_line(line):
  17.     t = tf.strings.split(line,"\t")
  18.     label = tf.reshape(tf.cast(tf.strings.to_number(t[0]),tf.int32),(-1,))
  19.     features = tf.cast(tf.strings.to_number(tf.strings.split(t[1]," ")),tf.int32)
  20.     return (features,label)
  22. ds_train=  tf.data.TextLineDataset(filenames = [train_token_path]) \
  23.    .map(parse_line,num_parallel_calls = tf.data.experimental.AUTOTUNE) \
  24.    .shuffle(buffer_size = 1000).batch(BATCH_SIZE) \
  25.    .prefetch(tf.data.experimental.AUTOTUNE)
  27. ds_test=  tf.data.TextLineDataset(filenames = [test_token_path]) \
  28.    .map(parse_line,num_parallel_calls = tf.data.experimental.AUTOTUNE) \
  29.    .shuffle(buffer_size = 1000).batch(BATCH_SIZE) \
  30.    .prefetch(tf.data.experimental.AUTOTUNE)

1. Modeling Using Sequential

  1. tf.keras.backend.clear_session()
  3. model = models.Sequential()
  5. model.add(layers.Embedding(MAX_WORDS,7,input_length=MAX_LEN))
  6. model.add(layers.Conv1D(filters = 64,kernel_size = 5,activation = "relu"))
  7. model.add(layers.MaxPool1D(2))
  8. model.add(layers.Conv1D(filters = 32,kernel_size = 3,activation = "relu"))
  9. model.add(layers.MaxPool1D(2))
  10. model.add(layers.Flatten())
  11. model.add(layers.Dense(1,activation = "sigmoid"))
  13. model.compile(optimizer='Nadam',
  14.             loss='binary_crossentropy',
  15.             metrics=['accuracy',"AUC"])
  17. model.summary()

6-1 Three Ways of Modeling - 图1

  1. import datetime
  2. baselogger = callbacks.BaseLogger(stateful_metrics=["AUC"])
  3. logdir = "../data/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
  4. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
  5. history = model.fit(ds_train,validation_data = ds_test,
  6.         epochs = 6,callbacks=[baselogger,tensorboard_callback])
  1. %matplotlib inline
  2. %config InlineBackend.figure_format = 'svg'
  4. import matplotlib.pyplot as plt
  6. def plot_metric(history, metric):
  7.     train_metrics = history.history[metric]
  8.     val_metrics = history.history['val_'+metric]
  9.     epochs = range(1, len(train_metrics) + 1)
  10.     plt.plot(epochs, train_metrics, 'bo--')
  11.     plt.plot(epochs, val_metrics, 'ro-')
  12.     plt.title('Training and validation '+ metric)
  13.     plt.xlabel("Epochs")
  14.     plt.ylabel(metric)
  15.     plt.legend(["train_"+metric, 'val_'+metric])
  16.     plt.show()
  1. plot_metric(history,"AUC")

6-1 Three Ways of Modeling - 图2

2. Modeling Using Functional API

  1. tf.keras.backend.clear_session()
  3. inputs = layers.Input(shape=[MAX_LEN])
  4. x  = layers.Embedding(MAX_WORDS,7)(inputs)
  6. branch1 = layers.SeparableConv1D(64,3,activation="relu")(x)
  7. branch1 = layers.MaxPool1D(3)(branch1)
  8. branch1 = layers.SeparableConv1D(32,3,activation="relu")(branch1)
  9. branch1 = layers.GlobalMaxPool1D()(branch1)
  11. branch2 = layers.SeparableConv1D(64,5,activation="relu")(x)
  12. branch2 = layers.MaxPool1D(5)(branch2)
  13. branch2 = layers.SeparableConv1D(32,5,activation="relu")(branch2)
  14. branch2 = layers.GlobalMaxPool1D()(branch2)
  16. branch3 = layers.SeparableConv1D(64,7,activation="relu")(x)
  17. branch3 = layers.MaxPool1D(7)(branch3)
  18. branch3 = layers.SeparableConv1D(32,7,activation="relu")(branch3)
  19. branch3 = layers.GlobalMaxPool1D()(branch3)
  21. concat = layers.Concatenate()([branch1,branch2,branch3])
  22. outputs = layers.Dense(1,activation = "sigmoid")(concat)
  24. model = models.Model(inputs = inputs,outputs = outputs)
  26. model.compile(optimizer='Nadam',
  27.             loss='binary_crossentropy',
  28.             metrics=['accuracy',"AUC"])
  30. model.summary()
  1. Model: "model"
  2. __________________________________________________________________________________________________
  3. Layer (type)                    Output Shape         Param #     Connected to                     
  4. ==================================================================================================
  5. input_1 (InputLayer)            [(None, 200)]        0                                            
  6. __________________________________________________________________________________________________
  7. embedding (Embedding)           (None, 200, 7)       70000       input_1[0][0]                    
  8. __________________________________________________________________________________________________
  9. separable_conv1d (SeparableConv (None, 198, 64)      533         embedding[0][0]                  
  10. __________________________________________________________________________________________________
  11. separable_conv1d_2 (SeparableCo (None, 196, 64)      547         embedding[0][0]                  
  12. __________________________________________________________________________________________________
  13. separable_conv1d_4 (SeparableCo (None, 194, 64)      561         embedding[0][0]                  
  14. __________________________________________________________________________________________________
  15. max_pooling1d (MaxPooling1D)    (None, 66, 64)       0           separable_conv1d[0][0]           
  16. __________________________________________________________________________________________________
  17. max_pooling1d_1 (MaxPooling1D)  (None, 39, 64)       0           separable_conv1d_2[0][0]         
  18. __________________________________________________________________________________________________
  19. max_pooling1d_2 (MaxPooling1D)  (None, 27, 64)       0           separable_conv1d_4[0][0]         
  20. __________________________________________________________________________________________________
  21. separable_conv1d_1 (SeparableCo (None, 64, 32)       2272        max_pooling1d[0][0]              
  22. __________________________________________________________________________________________________
  23. separable_conv1d_3 (SeparableCo (None, 35, 32)       2400        max_pooling1d_1[0][0]            
  24. __________________________________________________________________________________________________
  25. separable_conv1d_5 (SeparableCo (None, 21, 32)       2528        max_pooling1d_2[0][0]            
  26. __________________________________________________________________________________________________
  27. global_max_pooling1d (GlobalMax (None, 32)           0           separable_conv1d_1[0][0]         
  28. __________________________________________________________________________________________________
  29. global_max_pooling1d_1 (GlobalM (None, 32)           0           separable_conv1d_3[0][0]         
  30. __________________________________________________________________________________________________
  31. global_max_pooling1d_2 (GlobalM (None, 32)           0           separable_conv1d_5[0][0]         
  32. __________________________________________________________________________________________________
  33. concatenate (Concatenate)       (None, 96)           0           global_max_pooling1d[0][0]       
  34.                                                                  global_max_pooling1d_1[0][0]     
  35.                                                                  global_max_pooling1d_2[0][0]     
  36. __________________________________________________________________________________________________
  37. dense (Dense)                   (None, 1)            97          concatenate[0][0]                
  38. ==================================================================================================
  39. Total params: 78,938
  40. Trainable params: 78,938
  41. Non-trainable params: 0
  42. __________________________________________________________________________________________________

6-1 Three Ways of Modeling - 图3

  1. import datetime
  2. logdir = "../data/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
  3. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
  4. history = model.fit(ds_train,validation_data = ds_test,epochs = 6,callbacks=[tensorboard_callback])
  1. Epoch 1/6
  2. 1000/1000 [==============================] - 32s 32ms/step - loss: 0.5527 - accuracy: 0.6758 - AUC: 0.7731 - val_loss: 0.3646 - val_accuracy: 0.8426 - val_AUC: 0.9192
  3. Epoch 2/6
  4. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.3024 - accuracy: 0.8737 - AUC: 0.9444 - val_loss: 0.3281 - val_accuracy: 0.8644 - val_AUC: 0.9350
  5. Epoch 3/6
  6. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.2158 - accuracy: 0.9159 - AUC: 0.9715 - val_loss: 0.3461 - val_accuracy: 0.8666 - val_AUC: 0.9363
  7. Epoch 4/6
  8. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.1492 - accuracy: 0.9464 - AUC: 0.9859 - val_loss: 0.4017 - val_accuracy: 0.8568 - val_AUC: 0.9311
  9. Epoch 5/6
  10. 1000/1000 [==============================] - 24s 24ms/step - loss: 0.0944 - accuracy: 0.9696 - AUC: 0.9939 - val_loss: 0.4998 - val_accuracy: 0.8550 - val_AUC: 0.9233
  11. Epoch 6/6
  12. 1000/1000 [==============================] - 26s 26ms/step - loss: 0.0526 - accuracy: 0.9865 - AUC: 0.9977 - val_loss: 0.6463 - val_accuracy: 0.8462 - val_AUC: 0.9138
  1. plot_metric(history,"AUC")

6-1 Three Ways of Modeling - 图4

3. Customized Modeling Using Child Class of Model

  1. # Define a customized residual module as Layer
  3. class ResBlock(layers.Layer):
  4.     def __init__(self, kernel_size, **kwargs):
  5.         super(ResBlock, self).__init__(**kwargs)
  6.         self.kernel_size = kernel_size
  8.     def build(self,input_shape):
  9.         self.conv1 = layers.Conv1D(filters=64,kernel_size=self.kernel_size,
  10.                                    activation = "relu",padding="same")
  11.         self.conv2 = layers.Conv1D(filters=32,kernel_size=self.kernel_size,
  12.                                    activation = "relu",padding="same")
  13.         self.conv3 = layers.Conv1D(filters=input_shape[-1],
  14.                                    kernel_size=self.kernel_size,activation = "relu",padding="same")
  15.         self.maxpool = layers.MaxPool1D(2)
  16.         super(ResBlock,self).build(input_shape) # Identical to self.built = True
  18.     def call(self, inputs):
  19.         x = self.conv1(inputs)
  20.         x = self.conv2(x)
  21.         x = self.conv3(x)
  22.         x = layers.Add()([inputs,x])
  23.         x = self.maxpool(x)
  24.         return x
  26.     # Need to define get_config method in order to sequentialize the model constructed from the customized Layer by Functional API.
  27.     def get_config(self):  
  28.         config = super(ResBlock, self).get_config()
  29.         config.update({'kernel_size': self.kernel_size})
  30.         return config
  1. # Test ResBlock
  2. resblock = ResBlock(kernel_size = 3)
  3. resblock.build(input_shape = (None,200,7))
  4. resblock.compute_output_shape(input_shape=(None,200,7))
  1. TensorShape([None, 100, 7])
  1. # Customized model, which could also be implemented by Sequential or Functional API
  3. class ImdbModel(models.Model):
  4.     def __init__(self):
  5.         super(ImdbModel, self).__init__()
  7.     def build(self,input_shape):
  8.         self.embedding = layers.Embedding(MAX_WORDS,7)
  9.         self.block1 = ResBlock(7)
  10.         self.block2 = ResBlock(5)
  11.         self.dense = layers.Dense(1,activation = "sigmoid")
  12.         super(ImdbModel,self).build(input_shape)
  14.     def call(self, x):
  15.         x = self.embedding(x)
  16.         x = self.block1(x)
  17.         x = self.block2(x)
  18.         x = layers.Flatten()(x)
  19.         x = self.dense(x)
  20.         return(x)
  1. tf.keras.backend.clear_session()
  3. model = ImdbModel()
  4. model.build(input_shape =(None,200))
  5. model.summary()
  7. model.compile(optimizer='Nadam',
  8.             loss='binary_crossentropy',
  9.             metrics=['accuracy',"AUC"])
  1. Model: "imdb_model"
  2. _________________________________________________________________
  3. Layer (type)                 Output Shape              Param #   
  4. =================================================================
  5. embedding (Embedding)        multiple                  70000     
  6. _________________________________________________________________
  7. res_block (ResBlock)         multiple                  19143     
  8. _________________________________________________________________
  9. res_block_1 (ResBlock)       multiple                  13703     
  10. _________________________________________________________________
  11. dense (Dense)                multiple                  351       
  12. =================================================================
  13. Total params: 103,197
  14. Trainable params: 103,197
  15. Non-trainable params: 0
  16. _________________________________________________________________

6-1 Three Ways of Modeling - 图5

  1. import datetime
  3. logdir = "../tflogs/keras_model/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
  4. tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
  5. history = model.fit(ds_train,validation_data = ds_test,
  6.                     epochs = 6,callbacks=[tensorboard_callback])
  1. Epoch 1/6
  2. 1000/1000 [==============================] - 47s 47ms/step - loss: 0.5629 - accuracy: 0.6618 - AUC: 0.7548 - val_loss: 0.3422 - val_accuracy: 0.8510 - val_AUC: 0.9286
  3. Epoch 2/6
  4. 1000/1000 [==============================] - 43s 43ms/step - loss: 0.2648 - accuracy: 0.8903 - AUC: 0.9576 - val_loss: 0.3276 - val_accuracy: 0.8650 - val_AUC: 0.9410
  5. Epoch 3/6
  6. 1000/1000 [==============================] - 42s 42ms/step - loss: 0.1573 - accuracy: 0.9439 - AUC: 0.9846 - val_loss: 0.3861 - val_accuracy: 0.8682 - val_AUC: 0.9390
  7. Epoch 4/6
  8. 1000/1000 [==============================] - 42s 42ms/step - loss: 0.0849 - accuracy: 0.9706 - AUC: 0.9950 - val_loss: 0.5324 - val_accuracy: 0.8616 - val_AUC: 0.9292
  9. Epoch 5/6
  10. 1000/1000 [==============================] - 43s 43ms/step - loss: 0.0393 - accuracy: 0.9876 - AUC: 0.9986 - val_loss: 0.7693 - val_accuracy: 0.8566 - val_AUC: 0.9132
  11. Epoch 6/6
  12. 1000/1000 [==============================] - 44s 44ms/step - loss: 0.0222 - accuracy: 0.9926 - AUC: 0.9994 - val_loss: 0.9328 - val_accuracy: 0.8584 - val_AUC: 0.9052
  1. plot_metric(history,"AUC")

6-1 Three Ways of Modeling - 图6

Please leave comments in the WeChat official account “Python与算法之美” (Elegance of Python and Algorithms) if you want to communicate with the author about the content. The author will try best to reply given the limited time available.

You are also welcomed to join the group chat with the other readers through replying 加群 (join group) in the WeChat official account.

image.png