In Pixyz Review, you can stage your scene and set which objects can be manipulated to make it more real.

You need to set up an interaction behavior on an occurrence, make it grabbable in VR, and edit its constraint of movements.

Then, you can set the part which is pointable in order to avoid your teammate to teleport on your object.

See here for more informations & step-by-step tutorial: