Hello,
I am running MONAI Label using the radiology app. However, I would like to ask if someone knows here how to visualize the results of the training I am doing using MONAI with TensorBoard.
Thank you for your time
Spiros
You know that you can access the status of your running MonaiLabel server by calling http://127.0.0.1:8000/ in a browser on that server system and that is not enough information?
Hello
Thank you for your immediate response
I ran a training task, went into a tab and copied your address , but the page cannot be reached.
To be more precise, I use the radiology app to segment the fat tissue out of 3D volumes. I have plugged in the MONAI app in Slicer and using the default Hyperparameters for the training.
Also, I am running the MONAI in a Docker.
What I want to visualize are the training curves such as training loss ,validation loss graphs.
You seem to be starting a revolution in adipose tissue.
So do you start your MONAILabel server with something like (in a shell):
monailabel start_server --app MONAILabel/sample-apps/radiology --studies c:/Data/Task06_Lung/imagesTr --conf models segmentation
?
Then the monailabel output should tell you which port it uses, use that port for your localhost:xxxx URL
My GPT-4 consultation revealed:
https://chat.openai.com/share/d4a4b141-af01-450e-ad0a-4095deed8b09
Does this help?
Maybe @diazandr3s can comment too.
1 Like
Yes, exactly like this .
Very helpful answer, thank you .
I saw your GTP consultation and is helpful too. I suppose, the modifications must be made on the radiology / lib/ trainers / segmentation.py file.
Thank you again for your immediate responses
Hello again,
Ok, I start my MONAILabel server with a command line like you described :
monailabel start_server --app /workspace/radiology --studies /workspace/Training/fat --conf models segmentationfat --conf multi_gpu true
Then by putting the correct port via web browser I have this :
From this web interface I cannot take the training curves (image below) , and optimise the number of epochs (early stopping)
This is my goal in short. So far, for the fat delineation I use 50 number of epochs (default by MONAI). I want to avoid overfitting.
Regarding the ChatGPT conversation :
These types of modifications
I do not know , in which script I have to do these modifications. I suppose in the .py file which is contained in the ‘’/radiology/lib/configs/segmentationfat.py’’ path?
Sorry for the long answer, kind of newbie here, so that is why I want to explain everything.
Thank you
Hi Spiros,
MONAI Label server automatically splits your dataset into a train and validation set - the default split is at 20% of the dataset, but you can manually change that split percentage or you can even define your own fixed validation set.
Overfitting is avoided during training by only storing model weights if the validation Dice improves upon the previous iteration.
You can still choose to visualize a Tensorboard on the current training run. Just start a Tensorboard and point it to the directory where your model is. In my radiology-app this looks something like this:
/path/to/apps/radiology/model/deepedit_dynunet/train_01
There, you can see the Tensorboard folders events_* for each training run.
1 Like
Hello and thank you for your answer and time.
Yes, I know where the tensorboard files with a similar path like yours 
Okay, so before training do I type in the terminal a command like this :
‘’ tensorboard --logdir=path/to/log_directory ‘’ where the path points to the tensorboard files ?
And later type in the web : http://localhost:6006/ to monitor the process?
Thank you again,
Spiros
Hi, yes, exactly, you can start the Tensorboard before or during training, pointing the logdir to the model folder, and it’ll be in the browser on port 6006 (either localhost or remote IP address). Let us know whether this works!
Ok, I understood the concept. Since to run MONAI I am connecting to a server I did the following:
Installed the tensorflow to my server and when I run the command
tensorboard --logdir=/path/to/project/radiology/model/segmentationfatex/train_01
I get the following message : tensorboard: command not found
Which means that I have to find the path of tensorboard to run it. Correct?
However, when I execute the command : pip show tensorboard in the command prompt I get the location of tensorboard like this:
/home/MRuser/.local/lib/python3.8/site-packages
Then I copy paste the path and run again the command . However, the output is :
-bash: /home/MRuser/.local/lib/python3.8/site-packages/tensorboard: Is a directory
As I understand I will have to find the path of the executable file to run it. But the location was given, why does it not run?
Good question. How did you install monailabel? Via pip into a local python env?
If so, you may have to still install tensorboard? Instructions can be found here: How to use TensorBoard with PyTorch — PyTorch Tutorials 2.0.1+cu117 documentation
This would be weird though, because if you pip install monailabel, it installs monai-core along with tensorboard: MONAILabel/requirements.txt at main · Project-MONAI/MONAILabel · GitHub
Perhaps, you then need to find the binary and add its location to your PATH variable?
By the way, to avoid such environment and installation issues, I usually recommend everyone to run monailabel from a docker container, which can be started from one of the public image tags:
https://hub.docker.com/r/projectmonai/monailabel/tags
These images come with Tensorboard pre-installed, it can be run from console, and generally, the entire setup is very “clean”. It is also much easier to update or switch to a different monailabel version.
In Windows, if “monailabel” command is not found from a powershell, this kind of thing solves by adding
$Env:PATH += ";C:\Users\yourname\MONAILabel\monailabel\scripts"
You could add a conda environment in your Linux and conda install tensorboard.
Okay, i run the MONAI using docker (I start a MONAI Container). I connect to this server via the command prompt in my local machine.
Okay, after chatting with GPT I think I know the issue :
After, going to the directory where the tensorboard , I searched for the files and I found the main.py file which might be the one I am looking for. Thus, I entered the command line :
~/.local/lib/python3.8/site-packages/tensorboard$ python3 main.py --logdir=logdir=/ptojectdata/path /to/radiology/model/segmentationfat/train_01
and I get as an output this :
/home/MRuser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /home/MRUser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server) /home/MRuser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34’ not found (required by /home/MRuser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server)
/home/MRuser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32’ not found (required by /home/mriuser/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server)
Then I use this : ldd --version | head -n to see the version and the answer is :
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
which is not the required GLIBC version as I understand.
Lastly, the binary path in your case, where is it?
Thank you again
Maybe a solution would be to run the tensorboard to my local machine and copy the tensorboard log files from the server to my local machine instead?
1 Like
Please keep us updated and post the complete solution here.
1 Like
Okay, I finally managed to launch Tensorboard using my local machine.
More specifically, I installed tensorflow in my laptop and downloaded the log file from the server to my laptop.
Then I used the command line:
tensorboard --logdir=C:\Users\path\to\events_20230620_192239
And it worked.
However, in tensorboard I get only the train accuracy, train loss and validation accuracy. But not the validation loss.
Thus, I suspect by default MONAI do not calculate this. Correct?
2 Likes
There is no validation loss because MONAI Label doesn’t use the validation set for training.
Hope this makes sense,
Hello after a long time and thank you in advance for your helpful reply.
However, I got some results using MONAI Label for automatic body fat segmentation in axial MR slices.
So far, the performance model (displayed from Slicer interface) is 89% which is basically the dice similarity for the validation data.
However, I used new testing data (not seen by the model) to evaluate the Dice Similarity for this group of data. I used 6 new patients and the testing Dice Similarity was 98% which is a lot higher than the corresponding validation metric.
Thus, I believe I will have to regroup my data to avoid bias selection.
My question is on how the MONAI selects the validation data ? I know the default settings is the 20% of the training data used for this calculation. But how can I change which data is used for this?
Thank you in advance for your time.