Sounds like your problem is a vacumm leak or screwed up vacumm lines, in fact i'd say there's a 75% chance that's what it is. I've seen and had this problem before.
As others said - look for vacuum leak; or if it has a lot of cam you may not have enough vacuum at idle to operate the brakes.
Get it fixed and be safe. Or keep driving it this way and you'll shortly be in a situation where you'll have to buy another car -- then get one with good brakes.
A big cam in a computer controlled car won't run for shyz let alone have enough vac to power the brakes. Check for leaks or miss placed vac lines..it happens.