Should I be able to take him?

I agree, I would definately head on over to his place and demand the money, thats just weak. If he has any self respect he'll give you the money and ask for a rematch. You'll definately take him now, you got the psychological edge too!