Bits of Experience
In this page I'll share some tips and tricks regarding various, e.g., tools, that I use daily.
Tensorboard
While there are other tools for experiment tracking (e.g., Weights & Biases, MLFlow, Neptune, ...), I still use Tensorboard for tracking my ML experiments. It's great for tracking the evolution of your models over steps/epochs. Unfortunately, there are some caveats:
I can't see every logged image/audio/sample
This can be fixed by using the --samples_per_plugin option.
tensorboard --logdir={$DIRECTORY} --samples_per_plugin='images=200, audio=200, scalars=10000'Some hyperparameters are not visible in the HPARAMS table
It can happen especially if you're using a single Tensorboard server to show experiments with varying number of hyperparameters. See the open issue. Unfortunately this requires a fix via Tensorboard. Until then, you should be mindful not mixing experiment logs of models with a huge difference in their logged hyperparameters.
I can't access Tensorboard from another computer
Without the --bind_all option, requests coming outside of the allowed IP range (default should be localhost) will be ignored. You would also need to open a port default: 6006 in the firewall.