Do not assert in op_to_const.
`op_to_const` is used in `try_destructure_mir_constant_for_diagnostics`, which may encounter invalid constants created by optimizations and debugging.
r? ``@oli-obk``
Fixes https://github.com/rust-lang/rust/issues/117368