How to "reset" tensorboard data after killing tensorflow instance
Ok, for some reason it didn't work before but now it did:
You must erase Tensorboard's log files AND kill its process
After killing the process run fuser 6006/tcp -k
to free port 6006 (if you're in linux) and fire tensorboard again.
Note: The solution you've posted (erase TensorBoard's log files and kill the process) will work, but it isn't preferred, because it destroys historical information about your training.
Instead, you can have each new training job write to a new subdirectory (of your top-level log directory). Then, TensorBoard will consider each job a new "run" and will create a nice comparison view so you can see how the training differed between iterations of your model.
In the following an example from https://www.tensorflow.org/tensorboard/get_started:
model = create_model()
...
model.compile(...)
log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
model.fit(..., callbacks=[tensorboard_callback])
Yes, I believe ultimately this aspect is positive.
As an example, in my script I automate new run logs via datetime
:
from datetime import datetime
now = datetime.now()
logdir = "tf_logs/.../" + now.strftime("%Y%m%d-%H%M%S") + "/"
Then when running TensorBoard you can click the different runs on and off provided you ran TensorBoard in the parent directory.
If you know you don't care about a previous run and want it out of your life, then yes, you need to remove the event files and restart TensorBoard.
I had a similar problem, but with a duplication of computational graphs: they've just added in tensorboard when I called
writer.add_graph(graph=sess.graph)
In my case, it wasn't about log files but about Jupyter Notebook context.
I figured out that after multiple runs of a Jupyter cell with a Graph definition the graph hasn't been reset but appeared in the context as a duplicate, so I added
tf.reset_default_graph()
before the start of building a computational graph.
Hope it will help.