Pfthb

I am trying to implement a pretrained embedding layer into my generative model using GloVe.

Into the model I feed sequences of 50 (X) items pulled from a text, and it is to predict the 51. word (y) in the text.

I reach an accuracy of 0.99 already when the model only has trained for 1/100 iterations. What can be the issue?

# create a weight matrix for words in training docs
embedding_matrix = zeros((vocab_size, 100))
for word, i in tokenizer.word_index.items():
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
 embedding_matrix[i] = embedding_vector

 # define model
 model = Sequential() #assigning the sequential function to a model
model.add(Embedding(vocab_size, 100, weights=[embedding_matrix], input_length=seq_length, trainable = False)) #defining embedding layer size
model.add(LSTM(100, return_sequences=True)) #adding layer of nodes
model.add(LSTM(100)) #adding layer of nodes
model.add(Dense(100, activation='relu')) #specifying the structure of the hidden layer, recu is an argument of a rectified linear unit. 
model.add(Dense(vocab_size, activation='softmax')) #using the softmax function to creating probabilities
print(model.summary())
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# fit the model
model.fit(X, y, batch_size=128, epochs=100, verbose=1)

Link to github: https://github.com/KiriKoppelgaard/Generative_model
commit from Nov 14, 2018