Sequence to sequence example in Keras (character-level).

This script demonstrates how to implement a basic character-levelsequence-to-sequence model. We apply it to translatingshort English sentences into short French sentences,character-by-character. Note that it is fairly unusual todo character-level machine translation, as word-levelmodels are more common in this domain.

Summary of the algorithm

  • We start with input sequences from a domain (e.g. English sentences) and corresponding target sequences from another domain (e.g. French sentences).
  • An encoder LSTM turns input sequences to 2 state vectors (we keep the last LSTM state and discard the outputs).
  • A decoder LSTM is trained to turn the target sequences into the same sequence but offset by one timestep in the future, a training process called "teacher forcing" in this context. It uses as initial state the state vectors from the encoder. Effectively, the decoder learns to generate targets[t+1…] given targets[…t], conditioned on the input sequence.
  • In inference mode, when we want to decode unknown input sequences, we:
    • Encode the input sequence into state vectors
    • Start with a target sequence of size 1 (just the start-of-sequence character)
    • Feed the state vectors and 1-char target sequence to the decoder to produce predictions for the next character
    • Sample the next character using these predictions (we simply use argmax).
    • Append the sampled character to the target sequence
    • Repeat until we generate the end-of-sequence character or we hit the character limit.

Data download

English to French sentence pairs.

Lots of neat sentence pairs datasets.

References

  1. from __future__ import print_function
  3. from keras.models import Model
  4. from keras.layers import Input, LSTM, Dense
  5. import numpy as np
  7. batch_size = 64  # Batch size for training.
  8. epochs = 100  # Number of epochs to train for.
  9. latent_dim = 256  # Latent dimensionality of the encoding space.
  10. num_samples = 10000  # Number of samples to train on.
  11. # Path to the data txt file on disk.
  12. data_path = 'fra-eng/fra.txt'
  14. # Vectorize the data.
  15. input_texts = []
  16. target_texts = []
  17. input_characters = set()
  18. target_characters = set()
  19. with open(data_path, 'r', encoding='utf-8') as f:
  20.     lines = f.read().split('\n')
  21. for line in lines[: min(num_samples, len(lines) - 1)]:
  22.     input_text, target_text = line.split('\t')
  23.     # We use "tab" as the "start sequence" character
  24.     # for the targets, and "\n" as "end sequence" character.
  25.     target_text = '\t' + target_text + '\n'
  26.     input_texts.append(input_text)
  27.     target_texts.append(target_text)
  28.     for char in input_text:
  29.         if char not in input_characters:
  30.             input_characters.add(char)
  31.     for char in target_text:
  32.         if char not in target_characters:
  33.             target_characters.add(char)
  35. input_characters = sorted(list(input_characters))
  36. target_characters = sorted(list(target_characters))
  37. num_encoder_tokens = len(input_characters)
  38. num_decoder_tokens = len(target_characters)
  39. max_encoder_seq_length = max([len(txt) for txt in input_texts])
  40. max_decoder_seq_length = max([len(txt) for txt in target_texts])
  42. print('Number of samples:', len(input_texts))
  43. print('Number of unique input tokens:', num_encoder_tokens)
  44. print('Number of unique output tokens:', num_decoder_tokens)
  45. print('Max sequence length for inputs:', max_encoder_seq_length)
  46. print('Max sequence length for outputs:', max_decoder_seq_length)
  48. input_token_index = dict(
  49.     [(char, i) for i, char in enumerate(input_characters)])
  50. target_token_index = dict(
  51.     [(char, i) for i, char in enumerate(target_characters)])
  53. encoder_input_data = np.zeros(
  54.     (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
  55.     dtype='float32')
  56. decoder_input_data = np.zeros(
  57.     (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
  58.     dtype='float32')
  59. decoder_target_data = np.zeros(
  60.     (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
  61.     dtype='float32')
  63. for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
  64.     for t, char in enumerate(input_text):
  65.         encoder_input_data[i, t, input_token_index[char]] = 1.
  66.     encoder_input_data[i, t + 1:, input_token_index[' ']] = 1.
  67.     for t, char in enumerate(target_text):
  68.         # decoder_target_data is ahead of decoder_input_data by one timestep
  69.         decoder_input_data[i, t, target_token_index[char]] = 1.
  70.         if t > 0:
  71.             # decoder_target_data will be ahead by one timestep
  72.             # and will not include the start character.
  73.             decoder_target_data[i, t - 1, target_token_index[char]] = 1.
  74.     decoder_input_data[i, t + 1:, target_token_index[' ']] = 1.
  75.     decoder_target_data[i, t:, target_token_index[' ']] = 1.
  76. # Define an input sequence and process it.
  77. encoder_inputs = Input(shape=(None, num_encoder_tokens))
  78. encoder = LSTM(latent_dim, return_state=True)
  79. encoder_outputs, state_h, state_c = encoder(encoder_inputs)
  80. # We discard `encoder_outputs` and only keep the states.
  81. encoder_states = [state_h, state_c]
  83. # Set up the decoder, using `encoder_states` as initial state.
  84. decoder_inputs = Input(shape=(None, num_decoder_tokens))
  85. # We set up our decoder to return full output sequences,
  86. # and to return internal states as well. We don't use the
  87. # return states in the training model, but we will use them in inference.
  88. decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
  89. decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
  90.                                      initial_state=encoder_states)
  91. decoder_dense = Dense(num_decoder_tokens, activation='softmax')
  92. decoder_outputs = decoder_dense(decoder_outputs)
  94. # Define the model that will turn
  95. # `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
  96. model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
  98. # Run training
  99. model.compile(optimizer='rmsprop', loss='categorical_crossentropy',
  100.               metrics=['accuracy'])
  101. model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
  102.           batch_size=batch_size,
  103.           epochs=epochs,
  104.           validation_split=0.2)
  105. # Save model
  106. model.save('s2s.h5')
  108. # Next: inference mode (sampling).
  109. # Here's the drill:
  110. # 1) encode input and retrieve initial decoder state
  111. # 2) run one step of decoder with this initial state
  112. # and a "start of sequence" token as target.
  113. # Output will be the next target token
  114. # 3) Repeat with the current target token and current states
  116. # Define sampling models
  117. encoder_model = Model(encoder_inputs, encoder_states)
  119. decoder_state_input_h = Input(shape=(latent_dim,))
  120. decoder_state_input_c = Input(shape=(latent_dim,))
  121. decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
  122. decoder_outputs, state_h, state_c = decoder_lstm(
  123.     decoder_inputs, initial_state=decoder_states_inputs)
  124. decoder_states = [state_h, state_c]
  125. decoder_outputs = decoder_dense(decoder_outputs)
  126. decoder_model = Model(
  127.     [decoder_inputs] + decoder_states_inputs,
  128.     [decoder_outputs] + decoder_states)
  130. # Reverse-lookup token index to decode sequences back to
  131. # something readable.
  132. reverse_input_char_index = dict(
  133.     (i, char) for char, i in input_token_index.items())
  134. reverse_target_char_index = dict(
  135.     (i, char) for char, i in target_token_index.items())
  138. def decode_sequence(input_seq):
  139.     # Encode the input as state vectors.
  140.     states_value = encoder_model.predict(input_seq)
  142.     # Generate empty target sequence of length 1.
  143.     target_seq = np.zeros((1, 1, num_decoder_tokens))
  144.     # Populate the first character of target sequence with the start character.
  145.     target_seq[0, 0, target_token_index['\t']] = 1.
  147.     # Sampling loop for a batch of sequences
  148.     # (to simplify, here we assume a batch of size 1).
  149.     stop_condition = False
  150.     decoded_sentence = ''
  151.     while not stop_condition:
  152.         output_tokens, h, c = decoder_model.predict(
  153.             [target_seq] + states_value)
  155.         # Sample a token
  156.         sampled_token_index = np.argmax(output_tokens[0, -1, :])
  157.         sampled_char = reverse_target_char_index[sampled_token_index]
  158.         decoded_sentence += sampled_char
  160.         # Exit condition: either hit max length
  161.         # or find stop character.
  162.         if (sampled_char == '\n' or
  163.            len(decoded_sentence) > max_decoder_seq_length):
  164.             stop_condition = True
  166.         # Update the target sequence (of length 1).
  167.         target_seq = np.zeros((1, 1, num_decoder_tokens))
  168.         target_seq[0, 0, sampled_token_index] = 1.
  170.         # Update states
  171.         states_value = [h, c]
  173.     return decoded_sentence
  176. for seq_index in range(100):
  177.     # Take one sequence (part of the training set)
  178.     # for trying out decoding.
  179.     input_seq = encoder_input_data[seq_index: seq_index + 1]
  180.     decoded_sentence = decode_sequence(input_seq)
  181.     print('-')
  182.     print('Input sentence:', input_texts[seq_index])
  183.     print('Decoded sentence:', decoded_sentence)