Segmentation algorithm used in 3D Slicer

If you mean which algorithms are used for generating closed surface from binary labelmap representation of the segmentation: see implementation here. In short: vtkDiscreteFlyingEdges3D followed by vtkWindowedSincPolyDataFilter, vtkDecimatePro (optional), and vtkPolyDataNormals.