IDK about holding pressure when off, but on my 96, I hear the fuel pump run momentarily and switch off after turning the ignition switch to ON prior to cranking. Pressure at the fuel rail for the '99 should be 35-50psi with the engine running and 25-40psi with the key on/engine off. If gauges show these in specs, your problem is likely elsewhere.
When measuring the pressure in my 99 stang, it drops down to zero pressure 1-2 minutes after shutting off car. Is this a problem or normal. It runs around 30-32 psi.