Agreed, switch or motor... you'd probably know if the linkage was slipping (partial movement, noise, etc).
Next time it happens, got out of the car and try to move the wipers by hand. If they suddenly start moving, it's most likely the motor. Otherwise it's probably the switch.