Apparently MIR borrowck cares about at least one of these for checking variance. In runtime MIR, though, there's no need for them as `PtrToPtr` does the same thing. (Banning them simplifies passes like GVN that no longer need to handle multiple cast possibilities.)
For more information about how rustc works, see the rustc dev guide.