I think we spend more time in USF for a multitude of reasons outside of Diagon.
The food across the board is significantly better in USF imo, as well as entertainment in general. Between Bourne, Horror Makeup, the parade and all the street shows you can fill a lot of time. Add in ET, Mummy, MIB, Transformers and Fallon and you haven't even touched Diagon yet.
(Plus, we love just hanging out in Springfield)