Twist it to break the seal the rubber has on the plastic, it will come off easier.
Also, you arent trying to tug on the valve on the drivers side cover, correct? The PCV is on the passengers side, and it should pop right out and in. If not, rotate it a few degrees and try again.