Avoid accessing the HIR in the happy path of `coherent_trait`
Unfortunately the hir is still used in unsafety checks, and we do not have a way to avoid that. An impl's unsafety is not part of any query other than hir.
So this PR does not affect perf, but could still be considered a cleanup