Joe,

There's really only one you need to go to...
http://www.amoeba.com/  (1855 Haight St.)

There's also Mod Lang in the Easy Bay..
http://www.modlang.com/

If you feel like trekking across the GG Bridge to Marin county, 
Village Music in Mill Valley carries a massive collection of rare
vinyl and other obscurities. Worth checking out if you're
into that sort of thing: http://www.villagemusic.com/

-Jeff