Callbackfunction modelcheckpoint causes error in keras

If you look at the amount of data Keras is trying to save under layer_names attribute (inside the output HDF5 file being create), you will find that it takes more than 64kB.

np.asarray([layer.name.encode('utf8') for layer in model.layers]).nbytes
>> 77100

I quote from https://support.hdfgroup.org/HDF5/faq/limits.html:

Is there an object header limit and how does that affect HDF5 ?

There is a limit (in HDF5-1.8) of the object header, which is 64 KB. The datatype for a dataset is stored in the object header, so there is therefore a limit on the size of the datatype that you can have. (See HDFFV-1089)

The code above was (almost entirely) copied from the traceback:

File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2746, in save_weights_to_hdf5_group
f.attrs['layer_names'] = [layer.name.encode('utf8') for layer in layers]

I am using numpy asarray method to get the figure fast but h5py gets similar figure (I guess), see https://github.com/h5py/h5py/blob/master/h5py/_hl/attrs.py#L102 if you want to find exact figure.

Anyway, either you will need to implement your own methods for saving/loading of the weights (or use existing workarounds), or you need to give a really short name to ALL the layers inside your model :), something like this:

list_of_input = [Input(shape=(8,3), name=('i%x' % i)) for i in range(splits*total_frames_with_deltas)]
conv = Conv1D(filters = 100, kernel_size = 8, name='cv%x' % steps) 
conv.append(Add(name='add%x' % section)(temp))
output_conc = Concatenate(name='ct')(conv)
output_conv = Reshape((splits, -1), name='rs1')(output_conc)
pooled = MaxPooling1D(pool_size = 6, strides = 2, name='pl')(output_conv)
reshape = Reshape((1,-1), name='rs2')(pooled) 
dense1 = Dense(units = 1024, activation = 'relu', name = "d1")(reshape) 
dense2 = Dense(units
= 1024, activation = 'relu', name = "d2")(dense1) 
dense3 = Dense(units = 1024, activation = 'relu', name = "d3")(dense1) 
final = Dense(units = 5, activation = 'relu', name = "fl")(dense3)

You mustn't forget to name all the layers because the (numpy) string array into which the layer names are converted is using the size of the longest string for each individual string in it when it is saved!

After renaming the layers as proposed above (which takes almost 26kB) the model is saved successfully. Hope this elaborate answer helps someone.

Update: I have just made a PR to Keras which should fix the issue without implementing any custom loading/saving methods, see 7508

A simple solution, albeit possibly not the most elegant, could be to run a while loop with epochs = 1.

1. Get the weights at the end of every epoch together with the accuracy and the loss
2. Save the weights to file 1 with model.get_weight
3. if accuracy is greater than at the previous epoch (i.e. loop), store the weights to a different file (file 2)
4. Run the loop again loading the weights from file 1
5. Break the loops setting a manual early stopping so that it breaks if the loss does not improve for a certain number of loops

See follow-up at https://github.com/fchollet/keras/issues/6766 and https://github.com/farizrahman4u/keras-contrib/pull/90.

I saw the YAML and the root cause is probably that you have so many Inputs. A few Inputs with many dimensions is preferred to many Inputs, especially if you can use scanning and batch operations to do everything efficiently.

Now, ignoring that entirely, here is how you can save and load your model if it has too much stuff to save as JSON efficiently:

You can pass save_weights_only=True. That won't save optimizer weights, so isn't a great solution.

Just put together a PR for saving model weights and optimizer weights but not configuration. When you want to load, first instantiate and compile the model as you did when you were going to train it, then use load_all_weights to load the model and optimizer weights into that model. I'll try to merge it soon so you can use it from the master branch.

You could use it something like this:

from keras.callbacks import LambdaCallback
from keras_contrib.utils.save_load_utils import save_all_weights, load_all_weights
# do some stuff to create and compile model
# use `save_all_weights` as a callback to checkpoint your model and optimizer weights
model.fit(..., callbacks=[LambdaCallback(on_epoch_end=lambda epoch, logs: save_all_weights(model, "checkpoint-{:05d}.h5".format(epoch))])
# use `load_all_weights` to load model and optimizer weights into an existing model
# if not compiled (no `model.optimizer`), this will just load model weights
load_all_weights(model, 'checkpoint-1337.h5')

So I don't endorse the model, but if you want to get it to save and load anyways this should probably work for you.

As a side note, if you want to save weights in a different format, something like this would work.

pickle.dump([K.get_value(w) for w in model.weights], open( "save.p", "wb" ) )

Cheers

You can use get_weights() together with numpy.save.

It's not the best solution, because it will save several files, but it actually works.

The problem is that you won't have the "optimizer" saved with the current states. But you can perhaps work around that by using smaller learning rates after loading.

Custom callback using numpy.save:

def myCallback(epoch,logs):
    global storedLoss
    #do your comparisons here using the "logs" var.
    print(logs)


    if (logs['loss'] < storedLoss):

        storedLoss = logs['loss']
        for i in range(len(model.layers)):

            WandB = model.layers[i].get_weights()

            if len (WandB) > 0: #necessary because some layers have no weights

                np.save("W" + "-" + str(i), WandB[0],False) 
                np.save("B" + "-" + str(i), WandB[1],False)


    #remember that get and set weights use a list: [weights,biases]   
    #it may happen (not sure) that there is no bias, and thus you may have to check it (len(WandB)==1).   

The logs var brings a dictionary with named metrics, such as "loss", and "accuracy", if you used it.

You can store the losses within the callback in a global var, and compare if each loss is better or worse than the last.

When fitting, use the lambda callback:

from keras.callbacks import LambdaCallback
model.fit(...,callbacks=[LambdaCallback(on_epoch_end=myCallback)])   

In the example above, I used the LambdaCallback, which has more possibilities than just on_epoch_end.

For loading, do a similar loop:

#you have to create the model first and then set the layers
def loadModel(model):
    for i in range(len(model.layers)):
        WandBForCheck = model.layers[i].get_weights() 

        if len (WandBForCheck) > 0: #necessary because some layers have no weights
            W = np.load(Wfile + str(i))   
            B = np.load(Bfile + str(i))
            model.layers[i].set_weights([W,B])