vtkAppendPolyData just adds the list of points and surfaces together without resolving the geometry, so what you describe makes sense. Probably you need to cut and combine the surfaces. Exactly how you do that depends on what features you need to preserve in the data, but one option would be to convert both models to labelmap representation, combine them with logical operations in the segment editor, and then re-mesh to a single polydata.
To preserve the original shape you need to do a mesh-based boolean operation. I don’t do that myself much, but if you search for vtkbool and Slicer you will find a lot of discussion about when it works well and when it may have trouble with meshes. Maybe someone else can chime in with their experiences.
Combine Models module in Sandbox extension does not use vtkAppendPolyData, but it performs proper Boolean operations using vtkbool package. Advantage of this method is that all details of the input meshes are preserved. Disadvantage is that even though vtkbool is a quite robust package, in some special cases it may still fail.
You can set arbitrarily small spacing as segmentation geometry and then there will be no perceivable shape change. Disadvantage of this method is that memory need and computation time can increase very steeply as you reduce the spacing. The advantage of this method is that it works correctly every time - there are no problematic special cases.