I agree. Especially when you look down the buildings of Hogsmeade, the Castle looks miles away.
I took this picture two weeks ago.
It's going to look amazing when you're in Hogsmeade. I'm going to love every step I take closer to it.
However, from the Jurassic Park side, it does look extremely tiny in person...even smaller than I thought it looked in pictures from that side.