I would venture to say that that's at least 90% of the issue. If the video isn't synced properly to the motions, your brain will basically crap itself in confusion.
I remember back when VR coasters were introduced, everyone was sure it would cause motion sickness, but as soon as they started rolling out, those fears generally went away. With the VR experiences they've done at Six Flags GA, I haven't heard of any motion sickness issues, and the videos were synced perfectly.