The struggle is as old as art itself: Do you sacrifice artistic integrity to make what people want, or do you just do your own thing because it's something you care about.
I don't feel that I am unique among writers in feeling this way. My views on writing are a little behind the times in terms of things such as attributing a quote to someone and not including cursing in every third sentence. I'm pretty set in my ways. And I know I'm sacrificing chances to actually make a name for myself by sticking to what I like and what I believe in.
There are strong arguments to be made in either direction.
Polonius tells us in Hamlet: "To thine own self be true." Personally, I would find a lot more satisfaction in making it against the odds as a writer by continuing to do my own thing. It would serve as validation that my talent as a writer transcends pop culture and is actually quite good.
On the other hand, thumbing your nose at an audience is a good way to lose an audience. I use the musician Neil Young as a case study. After scoring his biggest hit in "Heart of Gold", he didn't really make another record like Harvest (on which Heart of Gold appears). It's hard to say whether it was intentional on his part. But his insistence on making music that he wants to make kept him off the charts for long years between hits.
I do not profess to be an expert, but my observations at this point among writing circles are that there are a ton of writers focused on Young Adult (YA) and Fantasy writing these days. The reasons might be more complex than this, but this is the generation that grew up with Harry Potter, one of the seminal collections and examples of this genre.
It doesn't help that series such as Twilight have also scored huge success, it validates the idea that people really want stories with fantastical elements that focus on coming of age. These two franchises alone account for several billion dollars of audience money fed into the respective author's bank accounts.
For those of us who are not necessarily interested in worlds with vampires and/or wizards, it seems like selling out to write stories in these genres. If you're talented, you could produce works in these genres that become popular, and make you rich.
Personally, I have a very hard time doing this. I don't discount the talent of someone like J.K. Rowling, but I just don't know if my audience is the same as her audience. I find myself reticent to want to wade into the crowded waters of these genres for the sake of making a name for myself.
One of the things I would fear is creating the expectation that I am going to continue to write in that genre. J.K. Rowling might want to write a science-fiction tale or a horror story, but she is probably somewhat limited in being able to do so. She's made her publisher a lot of money, and they certainly would want her to continue to do so. Name recognition will sell a few of her books, but there's always going to be a throng of people buying them in the hope of learning more about the universe she's created. They would be very disappointed if her newest book were a political thriller without the slightest hint of magic, even if it was spectacularly written.
I happen to believe in building your own audience more than I want to become famous. I know I could write great fantasy stories if I wanted to, but I just don't want to. I totally understand that the audience wanting more fantasy stories is much bigger than the ones who want sci-fi or historical fiction, but I love writing in those genres. It's also a much smaller pond in which to try to be a big fish.
It's not hard to understand arguments for "selling out" from an artistic standpoint. It would be nice to walk away from my day job and be a writer. It would be nice to sell one book so well that I would never have to work again for as long as I live.
But in the end I have to ask myself, would being rich and famous make me happy and feel like I have accomplished my goals as a writer? Or do I want to grind it out in relative obscurity, knowing that I enjoy what I'm writing?
It's a spiritual crisis that every artist faces, and I don't think there's a wrong answer to it.