Pfthb

I've trained an LSTM model (built with Keras and TF) on multiple batches of 7 samples with 3 features each, with a shape the like below sample (numbers below are just placeholders for the purpose of explanation), each batch is labeled 0 or 1:

Data:

[
 [[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3]]
 [[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3]]
 [[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3],[1,2,3]]
 ...
]

i.e: batches of m sequences, each of length 7, whose elements are 3-dimensional vectors (so batch has shape (m*7*3))

Target:

[
 [1]
 [0]
 [1]
 ...
]

On my production environment data is a stream of samples with 3 features ([1,2,3],[1,2,3]...). I would like to stream each sample as it arrives to my model and get the intermediate probability without waiting for the entire batch (7) - see the animation below.

One of my thoughts was padding the batch with 0 for the missing samples,
[[0,0,0],[0,0,0],[0,0,0],[0,0,0],[0,0,0],[0,0,0],[1,2,3]] but that seems to be inefficient.

Will appreciate any help that will point me in the right direction of both saving the LSTM intermediate state in a persistent way, while waiting for the next sample and predicting on a model trained on a specific batch size with partial data.

Update, including model code:

opt = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=10e-8, decay=0.001)
model = Sequential()

num_features = data.shape[2]
num_samples = data.shape[1]

first_lstm = LSTM(32, batch_input_shape=(None, num_samples, num_features), return_sequences=True, activation='tanh')
model.add(
 first_lstm)
model.add(LeakyReLU())
model.add(Dropout(0.2))
model.add(LSTM(16, return_sequences=True, activation='tanh'))
model.add(Dropout(0.2))
model.add(LeakyReLU())
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer=opt,
 metrics=['accuracy', keras_metrics.precision(), keras_metrics.recall(), f1])

Model Summary:

_________________________________________________________________
Layer (type) Output Shape Param # 
=================================================================
lstm_1 (LSTM) (None, 100, 32) 6272 
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU) (None, 100, 32) 0 
_________________________________________________________________
dropout_1 (Dropout) (None, 100, 32) 0 
_________________________________________________________________
lstm_2 (LSTM) (None, 100, 16) 3136 
_________________________________________________________________
dropout_2 (Dropout) (None, 100, 16) 0 
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU) (None, 100, 16) 0 
_________________________________________________________________
flatten_1 (Flatten) (None, 1600) 0 
_________________________________________________________________
dense_1 (Dense) (None, 1) 1601 
=================================================================
Total params: 11,009
Trainable params: 11,009
Non-trainable params: 0
_________________________________________________________________

I think there might be an easier solution.

If your model does not have convolutional layers or any other layers that act upon the length/steps dimension, you can simply mark it as stateful=True

Warning: your model has layers that act on the length dimension !!

The Flatten layer transforms the length dimension into a feature dimension. This will completely prevent you from achieving your goal. If the Flatten layer is expecting 7 steps, you will always need 7 steps.

So, before applying my answer below, fix your model to not use the Flatten layer. Instead, it can just remove the return_sequences=True for the last LSTM layer.

The following code fixed that and also prepares a few things to be used with the answer below:

def createModel(forTraining):

 #model for training, stateful=False, any batch size 
 if forTraining == True:
 batchSize = None
 stateful = False

 #model for predicting, stateful=True, fixed batch size
 else:
 batchSize = 1
 stateful = True

 model = Sequential()

 first_lstm = LSTM(32, 
 batch_input_shape=(batchSize, num_samples, num_features), 
 return_sequences=True, activation='tanh', 
 stateful=stateful) 

 model.add(first_lstm)
 model.add(LeakyReLU())
 model.add(Dropout(0.2))

 #this is the last LSTM layer, use return_sequences=False
 model.add(LSTM(16, return_sequences=False, stateful=stateful, activation='tanh'))

 model.add(Dropout(0.2))
 model.add(LeakyReLU())

 #don't add a Flatten!!!
 #model.add(Flatten())

 model.add(Dense(1, activation='sigmoid'))

 if forTraining == True:
 compileThisModel(model)

With this, you will be able to train with 7 steps and predict with one step. Otherwise it will not be possible.

The usage of a stateful model as a solution for your question

First, train this new model again, because it has no Flatten layer:

trainingModel = createModel(forTraining=True)
trainThisModel(trainingModel)

Now, with this trained model, you can simply create a new model exactly the same way you created the trained model, but marking stateful=True in all its LSTM layers. And we should copy the weights from the trained model.

Since these new layers will need a fixed batch size (Keras' rules), I assumed it would be 1 (one single stream is coming, not m streams) and added it to the model creation above.

predictingModel = createModel(forTraining=False)
predictingModel.set_weights(trainingModel.get_weights())

And voilà. Just predict the outputs of the model with a single step:

pseudo for loop as samples arrive to your model:
 prob = predictingModel.predict_on_batch(sample)

 #where sample.shape == (1, 1, 3)

When you decide that you reached the end of what you consider a continuous sequence, call predictingModel.reset_states() so you can safely start a new sequence without the model thinking it should be mended at the end of the previous one.

Saving and loading states

Just get and set them, saving with h5py:

def saveStates(model, saveName):

 f = h5py.File(saveName,'w')

 for l, lay in enumerate(model.layers):
 #if you have nested models, 
 #consider making this recurrent testing for layers in layers
 if isinstance(lay,RNN):
 for s, stat in enumerate(lay.states):
 f.create_dataset('states_' + str(l) + '_' + str(s),
 data=K.eval(stat), 
 dtype=K.dtype(stat))

 f.close()


def loadStates(model, saveName):

 f = h5py.File(saveName, 'r')
 allStates = list(f.keys())

 for stateKey in allStates:
 name, layer, state = stateKey.split('_')
 layer = int(layer)
 state = int(state)

 K.set_value(model.layers[layer].states[state], f.get(stateKey))

 f.close()

Working test for saving/loading states

import h5py, numpy as np
from keras.layers import RNN, LSTM, Dense, Input
from keras.models import Model
import keras.backend as K




def createModel():
 inp = Input(batch_shape=(1,None,3))
 out = LSTM(5,return_sequences=True, stateful=True)(inp)
 out = LSTM(2, stateful=True)(out)
 out = Dense(1)(out)
 model = Model(inp,out)
 return model


def saveStates(model, saveName):

 f = h5py.File(saveName,'w')

 for l, lay in enumerate(model.layers):
 #if you have nested models, consider making this recurrent testing for layers in layers
 if isinstance(lay,RNN):
 for s, stat in enumerate(lay.states):
 f.create_dataset('states_' + str(l) + '_' + str(s), data=K.eval(stat), dtype=K.dtype(stat))

 f.close()


def loadStates(model, saveName):

 f = h5py.File(saveName, 'r')
 allStates = list(f.keys())

 for stateKey in allStates:
 name, layer, state = stateKey.split('_')
 layer = int(layer)
 state = int(state)

 K.set_value(model.layers[layer].states[state], f.get(stateKey))

 f.close()

def printStates(model):

 for l in model.layers:
 #if you have nested models, consider making this recurrent testing for layers in layers
 if isinstance(l,RNN):
 for s in l.states:
 print(K.eval(s)) 

model1 = createModel()
model2 = createModel()
model1.predict_on_batch(np.ones((1,5,3))) #changes model 1 states

print('model1')
printStates(model1)
print('model2')
printStates(model2)

saveStates(model1,'testStates5')
loadStates(model2,'testStates5')

print('model1')
printStates(model1)
print('model2')
printStates(model2)

Considerations on the aspects of the data

In your first model (if it is stateful=False), it considers that each sequence in m is individual and not connected to the others. It also considers that each batch contains unique sequences.

If this is not the case, you might want to train the stateful model instead (considering that each sequence is actually connected to the previous sequence). And then you would need m batches of 1 sequence. -> m x (1, 7 or None, 3).

If I understood correctly, you have batches of m sequences, each of length 7, whose elements are 3-dimensional vectors (so batch has shape (m*7*3)).
In any Keras RNN you can set the
return_sequences flag to True to become the intermediate states, i.e., for every batch, instead of the definitive prediction, you will get the corresponding 7 outputs, where output i represents the prediction at stage i given all inputs from 0 to i.

But you would be getting all at once at the end. As far as I know, Keras doesn't provide a direct interface for retrieving the throughput whilst the batch is being processed. This may be even more constrained if you are using any of the CUDNN-optimized variants. What you can do is basically to regard your batch as 7 succesive batches of shape (m*1*3), and feed them progressively to your LSTM, recording the hidden state and prediction at each step. For that, you can either set return_state to True and do it manually, or you can simply set statefulto True and let the object keep track of it.

The following Python2+Keras example should exactly represent what you want. Specifically:

• allowing to save the whole LSTM intermediate state in a persistent way


• while waiting for the next sample


• and predicting on a model trained on a specific batch size that may be arbitrary and unknown.

For that, it includes an example of stateful=True for easiest training, and return_state=True for most precise inference, so you get a flavor of both approaches. It also assumes that you get a model that has been serialized and from which you don't know much about. The structure is closely related to the one in Andrew Ng's course, who is definitely more authoritative than me in the topic. Since you don't specify how the model has been trained, I assumed a many-to-one training setup, but this could be easily adapted.

from __future__ import print_function
from keras.layers import Input, LSTM, Dense
from keras.models import Model, load_model
from keras.optimizers import Adam
import numpy as np

# globals
SEQ_LEN = 7
HID_DIMS = 32
OUTPUT_DIMS = 3 # outputs are assumed to be scalars


##############################################################################
# define the model to be trained on a fixed batch size:
# assume many-to-one training setup (otherwise set return_sequences=True)
TRAIN_BATCH_SIZE = 20

x_in = Input(batch_shape=[TRAIN_BATCH_SIZE, SEQ_LEN, 3])
lstm = LSTM(HID_DIMS, activation="tanh", return_sequences=False, stateful=True)
dense = Dense(OUTPUT_DIMS, activation='linear')
m_train = Model(inputs=x_in, outputs=dense(lstm(x_in)))
m_train.summary()

# a dummy batch of training data of shape (TRAIN_BATCH_SIZE, SEQ_LEN, 3), with targets of shape (TRAIN_BATCH_SIZE, 3):
batch123 = np.repeat([[1, 2, 3]], SEQ_LEN, axis=0).reshape(1, SEQ_LEN, 3).repeat(TRAIN_BATCH_SIZE, axis=0)
targets = np.repeat([[123,234,345]], TRAIN_BATCH_SIZE, axis=0) # dummy [[1,2,3],,,]-> [123,234,345] mapping to be learned


# train the model on a fixed batch size and save it
print(">> INFERECE BEFORE TRAINING MODEL:", m_train.predict(batch123, batch_size=TRAIN_BATCH_SIZE, verbose=0))
m_train.compile(optimizer=Adam(lr=0.5), loss='mean_squared_error', metrics=['mae'])
m_train.fit(batch123, targets, epochs=100, batch_size=TRAIN_BATCH_SIZE)
m_train.save("trained_lstm.h5")
print(">> INFERECE AFTER TRAINING MODEL:", m_train.predict(batch123, batch_size=TRAIN_BATCH_SIZE, verbose=0))


##############################################################################
# Now, although we aren't training anymore, we want to do step-wise predictions
# that do alter the inner state of the model, and keep track of that.


m_trained = load_model("trained_lstm.h5")
print(">> INFERECE AFTER RELOADING TRAINED MODEL:", m_trained.predict(batch123, batch_size=TRAIN_BATCH_SIZE, verbose=0))

# now define an analogous model that allows a flexible batch size for inference:
x_in = Input(shape=[SEQ_LEN, 3])
h_in = Input(shape=[HID_DIMS])
c_in = Input(shape=[HID_DIMS])
pred_lstm = LSTM(HID_DIMS, activation="tanh", return_sequences=False, return_state=True, name="lstm_infer")
h, cc, c = pred_lstm(x_in, initial_state=[h_in, c_in])
prediction = Dense(OUTPUT_DIMS, activation='linear', name="dense_infer")(h)
m_inference = Model(inputs=[x_in, h_in, c_in], outputs=[prediction, h,cc,c])

# Let's confirm that this model is able to load the trained parameters:
# first, check that the performance from scratch is not good:
print(">> INFERENCE BEFORE SWAPPING MODEL:")
predictions, hs, zs, cs = m_inference.predict([batch123,
 np.zeros((TRAIN_BATCH_SIZE, HID_DIMS)),
 np.zeros((TRAIN_BATCH_SIZE, HID_DIMS))],
 batch_size=1)
print(predictions)


# import state from the trained model state and check that it works:
print(">> INFERENCE AFTER SWAPPING MODEL:")
for layer in m_trained.layers:
 if "lstm" in layer.name:
 m_inference.get_layer("lstm_infer").set_weights(layer.get_weights())
 elif "dense" in layer.name:
 m_inference.get_layer("dense_infer").set_weights(layer.get_weights())

predictions, _, _, _ = m_inference.predict([batch123,
 np.zeros((TRAIN_BATCH_SIZE, HID_DIMS)),
 np.zeros((TRAIN_BATCH_SIZE, HID_DIMS))],
 batch_size=1)
print(predictions)


# finally perform granular predictions while keeping the recurrent activations. Starting the sequence with zeros is a common practice, but depending on how you trained, you might have an <END_OF_SEQUENCE> character that you might want to propagate instead:
h, c = np.zeros((TRAIN_BATCH_SIZE, HID_DIMS)), np.zeros((TRAIN_BATCH_SIZE, HID_DIMS))
for i in range(len(batch123)):
 # about output shape: https://keras.io/layers/recurrent/#rnn
 # h,z,c hold the network's throughput: h is the proper LSTM output, c is the accumulator and cc is (probably) the candidate
 current_input = batch123[i:i+1] # the length of this feed is arbitrary, doesn't have to be 1
 pred, h, cc, c = m_inference.predict([current_input, h, c])
 print("input:", current_input)
 print("output:", pred)
 print(h.shape, cc.shape, c.shape)
 raw_input("do something with your prediction and hidden state and press any key to continue")

Additional information:

Since we have two forms of state persistency:

1. The saved/trained parameters of the model that are the same for each sequence

2. The a, c states that evolve throughout the sequences and may be "restarted"

It is interesting to take a look at the guts of the LSTM object. In the Python example that I provide, the a and c weights are explicitly handled, but the trained parameters aren't, and it may not be obvious how they are internally implemented or what do they mean. They can be inspected as follows:

for w in lstm.weights:
 print(w.name, w.shape)

In our case (32 hidden states) returns the following:

lstm_1/kernel:0 (3, 128)
lstm_1/recurrent_kernel:0 (32, 128)
lstm_1/bias:0 (128,)

We observe a dimensionality of 128. Why is that? this link describes the Keras LSTM implementation as follows:

The g is the recurrent activation, p is the activation, Ws are the kernels, Us are the recurrent kernels, h is the hidden variable which is the output too and the notation * is an element-wise multiplication.

Which explains the 128=32*4 being the parameters for the affine transformation happening inside each one of the 4 gates, concatenated:

• The matrix of shape (3, 128) (named kernel) handles the input for a given sequence element


• The matrix of shape (32, 128) (named recurrent_kernel) handles the input for the last recurrent state h.


• The vector of shape (128,) (named bias), as usual in any other NN setup.