i think you got to get those 2 metal bent rods to take it out. Looks like two metal U's. Most stores have them and they come in the stereo installing kits.
Well, as it turned out, everything went alright. I just wasn't sure how that center trim came out, but after prying and pulling, found out it just pops out. (Always afraid of breaking things by doing that....) So replaced that and all the door speakers, and now it has some passable tunes. Thanks for the replies everyone.