Do you remember if you heard the pump priming? The relay could be going bad, that has happened to me before. I also had a fuel pump go out on me with similar symptoms, it would start fine cold, but after driving a while, it would not restart until it sat for a few minutes to cool off I guess.