This hits home for me, as I spent the first 40 years of my life apologizing for my taste in not just books but movies and music as well. Even now, I still find myself apologizing, though it's changed a bit. In the past I apologized for liking weird, geeky things. Oftentimes now I find myself apologizing for "normal" things like romance novels or TV procedurals.
But life is too short and there's only so much time to enjoy all the arts available to us. So enjoy what you enjoy, and never apologize.