Yes, unfortunately for C++ loadable modules you do need to restart the application every time you rebuild. Technically you could unload and reload the shared libraries, but in development reloading is usually done because of some kind of bug or feature change so proper cleanup would be hard.
One way to approach this is to put the C++ code like itk classes, filters, io, etc into a helper library, like the logic classes you’ll find in some modules, which can usually be debugged using well defined tests. Then write the GUI as scripted modules that you can reload quickly. SlicerVMTK might be a good one to study, but many follow this pattern.