That's pretty wild. What brand is it? Do you have any other accessory drive modifications? Were the bolts loose on the pulley?
I had a similar problem, when the bolts loosened on my crank pulley. It eventually led to a crack in the pulley, and the resulting wobble had me confused for weeks (you couldn't see the crack while it was installed), because it looked straight when the engine was off, but apparently had an intermittent wobble while the engine was at higher rpm. Destroyed a few belts before I finally pulled it and the damper, and discovered the crack.