Remove more `ref` patterns from the compiler r? `@oli-obk` Previous PR: https://github.com/rust-lang/rust/pull/104500