Yeah confidence is definitely it. In my example I thought I was in the right place so had no reason not to be confident. I was wearing my normal clothing but I don't normally look too touristy, although if they looked closely I was wearing the blue UOAP wristband they gave me.
I've spent a lot of time in backstage areas at Disney, Universal and many hotels in every instance people are generally reluctant to question you so long as you look like you belong. There's a video on YouTube of some guy that hosted his own backstage tour at Epcot. The only issue he had was everyone was taking photos, he was even approached by a manager but confidently explained what he was doing and the manager said ok just keep your group from taking photos.