One of two suggestions, hopefully one works for you.
1. what you describe, sounds EXACTLY like what mine did, to the tee, and it took years to find because it was too random.
There is a vapor return line that runs the length of the car, pass side if I rember right. Right about where the door meets the quarter panel, or right in frontof the rear wheels under the car, the vapor line has a break. About 2 inches of it are NOT connected for some reason. The vapor line runs from the engine compartment, along the lenth of the car, then stops, then two inches later picks up again and runs back to the tank. These two inches are connected via a rubber hose. You should see it when you crawl under. Your rubber hose may be rotted away, or even missing like mine was. Replace it, problem solved.
2. My bro has a similar problem too, but his filler neck to gas tank grommet is bad, and he dumps and leaks fuel after filling up.