Masking STL for Registration

Hello and thank you for your reply.

I am trying to transform only part of a stl geometry. I want the other part not to be transformed (1 mm down from the top).

Example picture:
scenario

Here is what I have done:

-Create transform from two vessel volumes
-Import transform as volume
-Use segment to mask part that I want “static” and set inside to “0”
-Export masked transform volume
-import masked transform volume as transform
-Import STL of segment of vessel
-Apply transform to STL
-Harden transform

It seems like the mask is not working.
The static part does not stay static and it deforms so the top of the vessel is no longer flat like I need it to be.

I am not sure if my transform mask looks right. It is the gray circle but it seems like it still has a gradient and is not masking properly but I set the value to 0?
maskfail

Sorry for the confusion.