In many cases, you don’t need an equivalent to finally, because the cleanup is automatically handled via the Drop trait, which runs cleanup code when the object is freed from memory (similar to “destructors” in some other languages).
Because of Rust’s whole ownership thingamabob, it’s generally entirely deterministic when this code will run (at the closing brace for the scope in which the object is defined, unless you pass that object outside of that scope).

In other cases, you don’t need a finally, because nothing forces you to bubble up errors instantly. You can make a call which fails, store the error in a variable, run your cleanup steps and then return the error at the end of you function.

Sometimes, however, you do want to bubble up errors right away (via ? or early return), typically so you can group them together and handle them all the same.
In that case, you can run the cleanup code in the calling function. If you don’t to want to make it the responsibility of the caller, then pull out a small function within your function, so that you become the caller of that small function and can do the cleanup steps at the end of your function, while you do the bubbling within the smaller function.
There’s also ways to make that less invasive via closures (which also serve as a boundary to which you can bubble errors), but those are somewhat complex in Rust, due to the whole ownership thingamabob.

I will say, I do sometimes feel like Rust could use a better way to handle doing something before the error bubbles up. But generally speaking, I don’t feel like a finally is missing.

scopeguard would be one way to get defer in Rust