For the record, I'm not accusing you of making any ill-advised comment. I guess I was just thinking out loud. It just made me think of SeaWorld's policy of "No adults without kids, no kids without adults" policy. Not sure if that's still a thing, but it was at one point. It was at least for SeaWorld San Diego.
I agree that as long as you don't draw any attention to yourself by staring or approaching kids, there shouldn't be an issue.