I think this is one of the biggest problems with VR gaming. In theory, swinging around like Spider-Man in VR sounds awesome. But in practice, it's just nauseating. You really do need to be able to see the screen with a definite border to combat motion sickness. At least I do, not sure about everyone else.
It sounds lame, but the only VR game that I really enjoy is Real VR Fishing. Mostly because it's about as calm as it gets. No movement what so ever, but it still benefits from being immersive.
There's a game called Climbey (https://store.steampowered.com/app/520010/Climbey/), that was kind of like this. I had a ton of fun playing it, but it's pretty much like you said: this game worked for me, not so much for some of my friends who'd get nauseous as soon as they had to move (let alone look down).