With the volume of air that you're flowing the volume of the cold side is totally irrelevant, you're arguing miliseconds. As for pressure drop, you're assuming a fixed air volume going in and that's just not the way it works since the control system fights to maintain a fixed pressure at whatever point you take your reference signal from. The only realistic possible downside of putting in an intercooler that the math says is "too big" is the added weight, cost and/or effort to actually install it. There is a point of diminishing returns, but it certianly won't hurt anything hence my request for an actual example. Bottom line is you won't find one.
The math is great until you get to the real world and realise that it's only about 1/3 of what's really going on