Commit 69e26835e would burn my discrete GPU!

chir.set · January 14, 2018, 8:23pm

Since commit 69e26835e, Slicer freezes as soon as I move a Volume Rendering model when I use the discrete GPU of my laptop(DRI_PRIME=1). Temperature increases to near 100°C until the system itself hangs with a noisy fan spinning. kill, killall, pkill and xkill don’t stop Slicer. I have no other choice than powering off by long pressing the power button. I have to wait 10-15 minutes before getting back a usable system, I even thought some internal device was burnt.

This does not happen with commit 69e26835e if I just use the iGPU, but rendering is not sharp as it is less powerful than the dGPU. It does not happen on my desktop PC with a single GPU PCI-E card neither.

No such behaviour happens with preceding commit 486c6ee7a. I can offload to the dGPU and voilà.

Commit 69e26835e introduces many changes, there are probably things to fix.

I build Slicer on Arch Linux, configured as follows :

cmake -DSlicer_VTK_VERSION_MAJOR:INT=9 -DQt5_DIR:PATH=/usr/lib/cmake/Qt5 -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=0 -DBUILD_TESTING:BOOL=0 -DCMAKE_BUILD_TYPE:STRING=Release …/Slicer

with gcc 7.2.1 20171224, cmake version 3.10.1 and Qt 5.10.0.

For the record, my laptop has an AMD A10-7300 APU, with an AMD Kaveri iGPU, and an AMD Topaz dGPU. I am using the amdgpu kernel module and the open source xf86-video-amdgpu driver.

Secondly, I need your advice for another issue.

(question moved to How can we avoid removing all VTK directories when updating slicer)

lassoan · January 15, 2018, 3:17pm

The is certainly below Slicer’s level, but we can help in troubleshooting. Overheating is a low-level (hardware, driver, configuration) issue, but if you only encounter this with Slicer and other VTK-based applications then it is worth investigating how VTK uses your GPU differently compared to other applications.

Can you reproduce this with a simple VTK GPU volume rendering example? You can try this example as it is, and then replace vtkFixedPointVolumeRayCastMapper by vtkGPUVolumeRayCastMapper.

chir.set · January 15, 2018, 7:55pm

I built VTK9 on my laptop, I dont know if it’s the official one from Kitware or the fork at github/Slicer.

SimpleRayCast did not hang when the model is rotated, neither with vtkFixedPointVolumeRayCastMapper nor with vtkGPUVolumeRayCastMapper. That’s for both DRI_PRIME={0,1}. Example application GPURenderDemo in the VTK tree runs without issue also.

I am downloading the VTK fork at gitub/Slicer to test, I suppose it’s the one Slicer uses.

lassoan · January 15, 2018, 8:03pm

You can find the details of VTK repository and git branch/tag in External_VTKv9.cmake:

github.com

Slicer/Slicer/blob/b748b88cefa57bc126f3a5c64a4fb59c28e4bb9b/SuperBuild/External_VTKv9.cmake#L103-L121


ExternalProject_SetIfNotDefined(
  ${CMAKE_PROJECT_NAME}_${proj}_GIT_REPOSITORY
  "${git_protocol}://github.com/Slicer/VTK.git"
  QUIET
  )


set(_git_tag)
if("${Slicer_VTK_VERSION_MAJOR}" STREQUAL "7")
set(_git_tag "43f6ee36f6e28c8347768bd97df4d767da6b4ce7")
elseif("${Slicer_VTK_VERSION_MAJOR}" STREQUAL "9")
set(_git_tag "db38b04f8d90204a5e2dccbdacfefc4ded0d3ee0")
else()
message(FATAL_ERROR "error: Unsupported Slicer_VTK_VERSION_MAJOR: ${Slicer_VTK_VERSION_MAJOR}")
endif()
ExternalProject_SetIfNotDefined(
  ${CMAKE_PROJECT_NAME}_${proj}_GIT_TAG
  ${_git_tag}
  QUIET
  )

chir.set · January 15, 2018, 9:29pm

Ah, VTK is downloaded from github while building Slicer, so I just tested two similar trees, with same results.

Anyway, my main use case is not my laptop. So I’ll still be able to keep pace with Slicer development, until the nightly builds on your website come with VTK9. I need that for Volume Rendering as it just works without the GPU proprietary driver, which is not the case for VTK7 currently in the nightly builds.

chir.set · January 16, 2018, 11:24am

To my dismay, I just discovered that Slicer freezes when rotating a Volume Rendering model on my work desktop PC also. It has a single AMD RX480 GPU. Using therefore a build from a previous commit.

lassoan · January 16, 2018, 1:48pm

What is the size of your volume?
How much GPU memory usage do you allow in volume rendering settings?

Could you try if you have similar issues with volume rendering in ParaView?

Sankhesh_Jhaveri · January 16, 2018, 2:03pm

Going over past conversations in this thread, were you able to run SimpleRayCast on the desktop with AMD GPU. Did you that have issues too?

If you have a VTK build, you could also run the volume rendering tests

$ ctest -V -R vtkRenderingVolume*

chir.set · January 16, 2018, 5:04pm

I’m using CTA-cardio.nrrd from your repo.

On my work machine, 8 GB, corresponding to the VRAM of my GPU.
On my laptop, 3GB : 1 GB for the iGPU and 2 GB for the dGPU.

Downloaded PataView, but can’t find time now to learn how to create a Volume Rendering model there.

chir.set · January 16, 2018, 5:08pm

SimpleRayCast results on my work desktop PC with AMD RX480 GPU :

vtkFixedPointVolumeRayCastMapper : normal behaviour
vtkGPUVolumeRayCastMapper : immediate UI freeze leading to hard reboot on first run, immediate UI freeze with only mouse active but useless on second run.

I’ll try again at home on my laptop when possible.

lassoan · January 16, 2018, 5:09pm

How much GPU memory do you specify in Slicer (Volume rendering module, Advanced section, Techniques tab)?

To test in ParaView: You can only load uncompressed volumes, so re-save CTA-cardio with compression disabled. Then, load the volume and in Properties/Display/Representation, choose “Volume”.

chir.set · January 16, 2018, 5:11pm

ctest -V -R vtkRenderingVolume*
UpdateCTestConfiguration from :/home/user/src/SimpleRayCast/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/user/src/SimpleRayCast/DartConfiguration.tcl
Test project /home/user/src/SimpleRayCast
Constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph…
Checking test dependency graph end
No tests were found!!!

Can’t understand what it means.

lassoan · January 16, 2018, 5:14pm

By default, Slicer does not build test applications when VTK is built. In CMake’s VTK project, set BUILD_TESTING variable to ON, then configure and build VTK. After that, volume rendering tests will be found.

jcfr · January 16, 2018, 5:35pm

@Sankhesh_Jhaveri @lassoan Look like the example allows to reproduce the problem.

chir.set · January 16, 2018, 5:48pm

After accepting the dialog about time consuming process in ParaView, it starts computing and the UI freezes after about 5 secs. No model is displayed yet.

ihnorton · January 16, 2018, 5:50pm

Can you try the Radeon driver instead, as a differential? (or maybe you did already?)

https://wiki.archlinux.org/index.php/AMD_Catalyst

chir.set · January 16, 2018, 5:52pm

ctest -V -R vtkRenderingVolume* &> ~/tmp/ctest.log

It shows a few windows for less than 1 sec each, until the UI freezes. Mouse cursor is till available but useless. Here is what could be logged :

UpdateCTestConfiguration from :/home/user/src/VTK-build/DartConfiguration.tcl
Parse Config file:/home/user/src/VTK-build/DartConfiguration.tcl
Add coverage exclude regular expressions.
Add coverage exclude: vtk[^.]+(Java|Python).cxx
Add coverage exclude: .vtkOpenGLState.
Add coverage exclude: .Testing.Cxx.cxx
Add coverage exclude: .Testing.Cxx.h
Add coverage exclude: .moc_.cxx
Add coverage exclude: ./Rendering/OpenGL/vtkgl.
Add coverage exclude: ./Utilities/.
Add coverage exclude: ./ThirdParty/.
Add coverage exclude: .vtkOpenGLPolyDataMapper.
SetCTestConfiguration:CMakeCommand:/usr/bin/cmake
UpdateCTestConfiguration from :/home/user/src/VTK-build/DartConfiguration.tcl
Parse Config file:/home/user/src/VTK-build/DartConfiguration.tcl
Test project /home/user/src/VTK-build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph…
Checking test dependency graph end
test 41
Start 41: vtkRenderingVolume-HeaderTest

41: Test command: /bin/python2 “/home/user/src/VTK/Testing/Core/HeaderTesting.py” “/home/user/src/VTK/Rendering/Volume” “VTKRENDERINGVOLUME_EXPORT”
41: Test timeout computed to be: 3600
41: Use export macro: VTKRENDERINGVOLUME_EXPORT
1/89 Test #41: vtkRenderingVolume-HeaderTest … Passed 0.81 sec
test 353
Start 353: vtkRenderingVolumeOpenGL2-HeaderTest

353: Test command: /bin/python2 “/home/user/src/VTK/Testing/Core/HeaderTesting.py” “/home/user/src/VTK/Rendering/VolumeOpenGL2” “VTKRENDERINGVOLUMEOPENGL2_EXPORT”
353: Test timeout computed to be: 3600
353: Use export macro: VTKRENDERINGVOLUMEOPENGL2_EXPORT
2/89 Test #353: vtkRenderingVolumeOpenGL2-HeaderTest … Passed 0.19 sec
test 1208
Start 1208: vtkRenderingVolumeCxx-ProjectedTetrahedraZoomIn

1208: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “ProjectedTetrahedraZoomIn” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary”
1208: Test timeout computed to be: 3600
1208: Loading /home/user/src/VTK-build/ExternalData//Testing/Data/ironProt.vtk
1208: Loading /home/user/src/VTK-build/ExternalData//Testing/Data/neghip.slc
1208: Generic Warning: In /home/user/src/VTK/Rendering/Volume/Testing/Cxx/ProjectedTetrahedraZoomIn.cxx, line 197
1208: This test will always pass.
1208:
3/89 Test #1208: vtkRenderingVolumeCxx-ProjectedTetrahedraZoomIn … Passed 2.30 sec
test 1209
Start 1209: vtkRenderingVolumeCxx-TestFinalColorWindowLevel

1209: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestFinalColorWindowLevel” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestFinalColorWindowLevel.png”
1209: Test timeout computed to be: 3600
1209: 0Standard0.182799
1209: 0.342806
4/89 Test #1209: vtkRenderingVolumeCxx-TestFinalColorWindowLevel … Passed 0.59 sec
test 1210
Start 1210: vtkRenderingVolumeCxx-TestFixedPointRayCastLightComponents

1210: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestFixedPointRayCastLightComponents” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestFixedPointRayCastLightComponents.png”
1210: Test timeout computed to be: 3600
1210: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)
1210: 0Standard0.152745
1210: 0.236041
5/89 Test #1210: vtkRenderingVolumeCxx-TestFixedPointRayCastLightComponents … Passed 0.77 sec
test 1211
Start 1211: vtkRenderingVolumeCxx-TestGPURayCastAdditive

1211: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestGPURayCastAdditive” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestGPURayCastAdditive.png”
1211: Test timeout computed to be: 3600
1211: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)
1211: 5.27712Standard0.062804
1211: 0.128531
6/89 Test #1211: vtkRenderingVolumeCxx-TestGPURayCastAdditive … Passed 0.34 sec
test 1212
Start 1212: vtkRenderingVolumeCxx-TestGPURayCastCompositeBinaryMask

1212: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestGPURayCastCompositeBinaryMask” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestGPURayCastCompositeBinaryMask.png”
1212: Test timeout computed to be: 3600
1212: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)
1212: 0.00261438Standard0.0450981
1212: 0.061025
7/89 Test #1212: vtkRenderingVolumeCxx-TestGPURayCastCompositeBinaryMask … Passed 0.33 sec
test 1213
Start 1213: vtkRenderingVolumeCxx-TestGPURayCastCompositeMaskBlend

1213: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestGPURayCastCompositeMaskBlend” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestGPURayCastCompositeMaskBlend.png”
1213: Test timeout computed to be: 3600
1213: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)
1213: 0Standard0.069319
1213: 0.14196
8/89 Test #1213: vtkRenderingVolumeCxx-TestGPURayCastCompositeMaskBlend … Passed 0.38 sec
test 1214
Start 1214: vtkRenderingVolumeCxx-TestGPURayCastCompositeMask

1214: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestGPURayCastCompositeMask” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestGPURayCastCompositeMask.png”
1214: Test timeout computed to be: 3600
1214: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)
1214: 0.392157Standard0.078526
1214: 0.153189
9/89 Test #1214: vtkRenderingVolumeCxx-TestGPURayCastCompositeMask … Passed 0.33 sec
test 1215
Start 1215: vtkRenderingVolumeCxx-TestGPURayCastCompositeToMIP

1215: Test command: /home/user/src/VTK-build/bin/vtkRenderingVolumeCxxTests “TestGPURayCastCompositeToMIP” “-D” “/home/user/src/VTK-build/ExternalData//Testing” “-T” “/home/user/src/VTK-build/Testing/Temporary” “-V” “/home/user/src/VTK-build/ExternalData/Rendering/Volume/Testing/Data/Baseline/TestGPURayCastCompositeToMIP.png”
1215: Test timeout computed to be: 3600
1215: CTEST_FULL_OUTPUT (Avoid ctest truncation of output)

Now I have to finish my paper work !

chir.set · January 16, 2018, 6:17pm

Well the RX480 GPU won’t work with the radeon kernel module. I’ll try that on my laptop ASAP.

(BTW, the page you referenced is much outdated. Catalyst is dead, Xorg is versioned to 1.1.19.)

Sankhesh_Jhaveri · January 17, 2018, 8:36pm

We were able to test this on the same GPU on a mac with proprietary AMD drivers. No issues.
It could be a driver issue.

There have been many changes to the volume rendering code in VTK that went into commit 69e26835e. Now that you have a VTK build, would you be able to run git bisect to isolate the VTK commit that introduced this issue?

jcfr · January 17, 2018, 8:58pm

Thanks

Here are more details about git bisect: https://blog.kitware.com/regression-hunting-from-slicer-cdash-error-to-itk-master/

And an example of test script that could be adapted: https://gist.github.com/jcfr/6029698

@Sankhesh_Jhaveri I guess the bisect test script could rebuild and only execute the test process TestGPURayCastCompositeToMIP. Then, if it doesn’t stop after a while, it could be killed if it doesn’t exit. Do you have test script that could be reused for this type of problem ?

but if the machine freeze, I guess he would have to be a manual bisect with a systematic restart of the machine …

Topic		Replies	Views
VTK multi-volume ROI not working Support	5	1429	May 5, 2021
Slicer crash without any message when try to run my own module Support	0	209	April 22, 2023
Enabling volume rendering for a large volume makes the application crash Support volume-rendering	6	657	April 12, 2021
Volume Rendering : displaying ROI leads to a crash on self built Slicer Support	7	429	January 12, 2022
Slicer crashing during Model Making Support	1	376	July 6, 2021

Commit 69e26835e would burn my discrete GPU!

Related Topics