You could have worn cam phasers which is common on high mileage engines (and on lower mileage engines that have been driven hard). The symptoms are often intermittent, especially in the early stages.
Do you hear any ticking noise from the front of the engine at idle (not the normal clicking of the injectors)?
Another possibility is that the mesh screen inside the VCT solenoid oil control valves is clogged up. This is common on engines where oil changes have either been neglected or poor quality oil used.