Sound like water
When soundproofing a space the analogy often used is to imagine a boat in water. If there are any holes in the boat it will take on water. Similarly, if there are any holes in a building those will allow the passage of sound (outside to inside or vice versa). What this translates to in practice is a whole lot of caulk.