Is there some way to save best model only with tensorflow.estimator.train_and_evaluate()?
I have been using https://github.com/bluecamel/best_checkpoint_copier which works well for me.
Example:
best_copier = BestCheckpointCopier(
name='best', # directory within model directory to copy checkpoints to
checkpoints_to_keep=10, # number of checkpoints to keep
score_metric='metrics/total_loss', # metric to use to determine "best"
compare_fn=lambda x,y: x.score < y.score, # comparison function used to determine "best" checkpoint (x is the current checkpoint; y is the previously copied checkpoint with the highest/worst score)
sort_key_fn=lambda x: x.score,
sort_reverse=False) # sort order when discarding excess checkpoints
pass it to your eval_spec:
eval_spec = tf.estimator.EvalSpec(
...
exporters=best_copier,
...)
You can try using BestExporter
. As far as I know, it's the only option for what you're trying to do.
exporter = tf.estimator.BestExporter(
compare_fn=_loss_smaller,
exports_to_keep=5)
eval_spec = tf.estimator.EvalSpec(
input_fn,
steps,
exporters)
https://www.tensorflow.org/api_docs/python/tf/estimator/BestExporter