I'll bet what's really happening is the flow path is being diverted rather than the blower changing speeds. When you select floor/defrost/vent etc, vacuum is used to open or close trap doors in the heater & A/C box. I've seen this phenomenon happen in older cars, but for the life of me I can't recall the exact trouble. Bad vacuum dashpot maybe? The dashpots are little diaphragm canisters that actuate the trap doors I mentioned earlier. Check those, and the vacuum lines running to and from the selector switch.
