I guess something changed in Rust and now this works. I'm not seeing any drop in FPS like I saw before and no errors in the console.