Then the desired result is to merge these two models, while preserving the internal root canal system.
But when I try to use the logical operator of segment editor or merge models method in 3D slicer, the internal cavity will disappear in the intersect part of merged model. So I am wondering if there is any method that could preserve the internal root canal system and what should I do to achieve this goal. Any help would be appreciated!
I would suggest that you make a copy of the hollow teeth and fill the root canal systems in the copy (could use “Smoothing → Closing (fill holes)”). Then you can use logical subtraction to subtract the filled teeth from mandible. Next, delete the filled tooth copies. Now the original (unfilled) teeth should fit into the mandible gaps without the root canal systems being filled.
I thought about this a little more and have another suggestion. If the outer surface of each tooth is closed (i.e. the root canal cavity does not extend to an opening at the bottom of the tooth), then you could use your initial process (without making copies of the teeth and filling them) and just use the “Islands” tool and “Keep largest island” on the mandible segment. As long as the root canal systems don’t have openings, the bits you don’t want should be separate islands and will disappear if you only keep the largest island. This method is simpler and more efficient if it will work for you.