A blow off valve is not used because the throttle butterflies are upstream from the compressor. When they close, it is impossible for the turbo to surge because the compressor is spinning in a vacuum. This has the added benefit of reducing lag as well. Some of the turbo F1 teams put a throttle body on the compressor inlet just for that purpose even though they were using EFI.
As for air "not spending time" in the intercooler, this too is a non issue for the reasons mrmustang alluded to. It's the same argument that water flowing "too fast" through a radiator have enough time to cool off, when in fact flow over time has little impact on the effectiveness of the cooling system.