LSTM Followed by Mean Pooling

I just attempted to implement the same model as the original poster, and I'm using Keras 2.0.3. The mean pooling after LSTM worked when I used GlobalAveragePooling1D, just make sure return_sequences=True in the LSTM layer. Give it a try!

Adding TimeDistributed(Dense(1)) helped:

sequence = Input(shape=(max_sent_len,), dtype='int32')
embedded = Embedding(vocab_size, word_embedding_size)(sequence)
lstm = LSTM(hidden_state_size, activation='sigmoid', inner_activation='hard_sigmoid', return_sequences=True)(embedded)
distributed = TimeDistributed(Dense(1))(lstm)
pool = AveragePooling1D()(distributed)
output = Dense(1, activation='sigmoid')(pool)