Other answers deal with the "rules" of writing quite well. I'm not going to address that, I'm going to address the main part of your question:
What if the hero doesn't win?
Absolutely write this. This is the type of emotional hook, that when done well, will leave a lasting impression on your readers and keep them coming back for more.
There are four main types that, I believe, are the most powerful (warning - these are TV Tropes links and, as such, you may got lost for hours. It happens) - and while they are separate here, they are often interlinked.
1) Bittersweet Ending
2) The Bad Guy Wins.
3) Phyrric Victory
4) Downer Ending
The Bttersweet and Phyrric Victories work quite well together, and can be a very powerful tool. They make the cost of victory too high and can be a very emotional read.
Perhaps the hero manages to thwart the villain at the last possible moment, but in the process loses their friends/family/the world is destroyed.
Perhaps the hero fails at the end, his courage falters, and others pay the price. He is wracked by guilt.
Conversely, the villain triumphs and, against all reason, turns out to do a great deal of good. The hero, still opposing the villain, becomes the very thing he was trying to stop?
There are many, many examples, and the best part is because it isn't really the norm, that's the type of story that, done well, will live with you forever.
In terms of literary examples, my favourite is the death of Anomander Rake from Steven Erikson's Malazan Book of the Fallen series. His death, his purpose, the actions of not only himself, but those around him make it powerful. It grips the reader and doesn't let go. More than Anomander's actions and death, the real hero in that scene is Spinnock Durav. He fought an impossible fight, he lost it but delayed the unkillable villain long enough. He lost his individual fight, but through his actions the greater one was won. And it cost him everything - his people lost their leader, the greatest member of their race. But in doing so, hope for the future was also reborn. I remember it fondly, because I read the entire book (Toll the Hounds) in one sitting. I couldn't put it down. I had tears streaming down my face as I reached that scene (admittedly it was also about 4am at this stage). If you can get your hands on them, then I thoroughly suggest reading them.