What do I do with a dog is scared and aggressive toward everyone but me?

I adopted a Golden Retriever/Chow mix from an animal shelter three months ago. So far, the only people he will even let put a leash on him are me and my girlfriend. My roommate, my friends, my parents...he is too scared of everybody else in my life. It's gotten to the point where I can't even board him or have a friend take care of him if I'm away for work. Occasionally he will even get aggressive (lunge, snarl, etc...no bites so far) if someone makes a sudden move. I'm thinking clearly he was abused at some point in his life, but how do I reverse this and help him to love other people? Taking him for walks, being the person that feeds him, giving him treats...all the usual tricks I've tried, and he still is too scared of everyone. HELP! I want my dog to be people friendly!