Even taking that game out the other 3 make sense.
Link defeats Ganon and goes back to his childhood. In that alternate future Ganon returns and wrecks havoc. Without a hero to stop him, the gods flood the world. This is all in Wind Waker’s opening.
In the childhood that Link returns to he warns everyone about Ganondorf’s true plans. Ganondorf is put on trial and exiled to the Twilight Realm. Majora’s Mask also takes place here directly following OoT.
All Skyward Sword did was explain the origin of the Master Sword and why there’s always an endless cycle of Link + Zelda vs. Ganon. Btw I think it’s crap too. Before Skyward Sword all of these links were still pretty clear.