Adding regularizer to an existing layer of a trained model without resetting weights?
You need to do 2 things:
Add regularizers in the following way:
model.get_layer('Dense_1').kernel_regularizer = l2(0.01)
Recompile the model:
model.compile(...)
For tensorflow 2.2 you just need to do that:
l2 = tf.keras.regularizers.l2(1e-4)
for layer in model.layers:
# if hasattr(layer, 'kernel'):
# or
# If you want to apply just on Conv
if isinstance(layer, tf.keras.layers.Conv2D):
model.add_loss(lambda layer=layer: l2(layer.kernel))
Hope it will help
The solution from Marcin hasn't worked for me. As apatsekin mentioned, if you print layer.losses
after adding the regularizers as Marcin proposed, you will get an empty list.
I found a workaround that I do not like at all, but I am posting here so someone more capable can find a way to do this in an easier way.
I believe it works for most keras.application
networks. I copied the .py
file of a specific architecture from keras-application in Github (for example, InceptionResNetV2) to a local file regularizedNetwork.py
in my machine. I had to edit it to fix some relative imports such as:
#old version
from . import imagenet_utils
from .imagenet_utils import decode_predictions
from .imagenet_utils import _obtain_input_shape
backend = None
layers = None
models = None
keras_utils = None
to:
#new version
from keras import backend
from keras import layers
from keras import models
from keras import utils as keras_utils
from keras.applications import imagenet_utils
from keras.applications.imagenet_utils import decode_predictions
from keras.applications.imagenet_utils import _obtain_input_shape
Once the relative paths and import issues were solved, I added the regularizers in each desired layer, just as you would do when defining a new untrained network. Usually, after defining the architecture, the models from keras.application
load the pre-trained weights.
Now, in your main code/notebook, just import the new regularizedNetwork.py
and call the main method to instantiate the network.
#main code
from regularizedNetwork import InceptionResNetV2
The regularizers should be all set and you can fine-tune the regularized model normally.
I am certain there is a less gimmicky way of doing this, so, please, if someone finds it, write a new answer and/or comment in this answer.
Just for the record, I also tried instantiating the model from keras.application
, getting the its architecture with regModel = model.get_config()
, adding the regularizers as Marcin suggested and then loading the weights with regModel.set_weights(model.get_weights())
, but it still didn't work.
Edit: spelling errors.