It appears that AntsPy does not use ITK’s Python package but instead builds ITK C++ from source and uses that along with wrapping it using a package called nanobind.
I observe similar warnings about possible incompatible factory load when I connect an OpenIGTLink connection using PlusToolkit’s PlusServer that uses a slightly different ITK version than the one used by Slicer core and corresponding SlicerOpenIGTLink extension. I observe the warning, but generally don’t face any issues because the versions are similar enough.
Here is the C++ ITK build that uses ITK 5.4.3.