- First point I want to address is a gross mischaracterization of people working on Bohmian mechanics or other hidden variable interpretations. I quote
Quantum mechanics remains unacceptable for many people, for quasi-religious reasons.
[...]
it's time for those people to seriously consider the hypothesis that quantum mechanics is really how the world works.
People for whom it is hard to swallow the hard reality of quantum mechanics, people who want to understand nature in terms of every day classical intuitions are the ones who often advocate Bohmian mechanics.
This is a straw man. Nobody I know working or having worked on hidden variable interpretations of quantum mechanics, be it Bell, Bohm, Goldstein, Bricmont, Maudlin, de Broglie or other, denies the reality of quantum mechanics. Quite the contrary, I would even say they are/were more aware of the implications than many people otherwise working with quantum mechanics. In particular they fully recognize the non-local nature of quantum mechanics. But I'll come back to that. I can of course not speak for ALL people dealing in hidden variable theories or other alternatives. I'm sure there will be a faire share of crackpots and other cranks. But there are many people who know their stuff.
Second point is the Zeilinger article that @Luboš Motl mentions. I've only skimmed through the article, but it seems that the only non-local hidden variable theories that are disposed of are ones devised by Leggett. Moreover, on page 3 of the article, the authors recognize that while their work indeed rules out a vast class of non-local hidden variable theories, it does not rule out Bohmian mechanics. So it is disingeneous to use this article to discredit Bohmian mechanics. The work in the article is however very valuable since it puts more strict constraints on the type of non-local hidden variable theories that can reproduce the results of quantum mechanics. This is an interesting question that is also discussed in the excellent book Quantum Paradoxes by Yakir Aharonov and Daniel Rohrlich. In one of the later chapters, they discuss the very special nature of non-locality in quantum mechanics, pointing out that there are many varying degrees of non-locality which are either stronger or weaker than the non-locality of quantum mechanics. They say it is still an open problem to find a precise characterization of the nature of quantum non-locality which could be turned into a postulate from which quantum mechanics could be derived in a way analoguous to the way special relativity is derived from its basic postulates.
Now, about Bell's theorem. I'll quote a piece of the following article which itself refers to a thought experiment proposed by Tim Maudlin. Of course, you'll recognize it is a variant of an EPR type experiment.
Here is a puzzle: two persons, call them $X$ and $Y$ , leave a room
through opposite doors; at that point, each is asked a question. The
precise nature of the questions does not matter, but there are three
possible questions (say, $A$, $B$ and $C$). Each person must answer yes
or no. This “experiment” is repeated many times, with sometimes
the same question, sometimes different questions being asked at the
two doors. The two persons are allowed to decide, before leaving
the room, to follow any strategy they want, but not to communicate
with each other, after they have heard the questions.
The statistics of answers have some strange properties. First
of all, it turns out that when the two people are asked the same
question, they always give the same answer. Is that mysterious? Of
course not; they simply decide, before leaving the room, to follow a
certain strategy: for example, to both say ‘yes’ if the question is $A$,
‘no’ if the question is $B$ and ‘no’ if the question is $C$. Altogether,
there are $8 = 2^3$ different such strategies. Before proceeding further, the reader has to answer for himself or herself the following
question: Is there any other way? Is there any way to account for
the perfect correlations between the results at the two doors without assuming that the answers were predetermined (if we assume
that the people cannot have any communication whatsoever with
each other once the questions are asked)? I have never seen any
suggestion of another possibility and I believe that if Bell’s theorem
is arguably the most widely misunderstood result in the history of
physics, it is precisely because this question is not answered before
proceeding further.
So, let us consider, for the time being, the assumption that the
answers are predetermined and let us call $v_i(\alpha) = \pm 1$, $i = X, Y$ ,
$\alpha = A, B, C$ those answers. These are “random variables”, namely
they may take different values when one repeats the “experiment”.
However, if one looks at the statistics of answers when different
questions are asked at the two doors, one finds that the frequencies
of the events in which the same answers are given is 1/4. And this,
combined with the perfect correlations is strange. Indeed, a version
of the no hidden variable theorems (similar to the one discussed in
the previous section), known as Bell’s theorem.
I'll leave out the proof, you can read it for yourself in the article. It is just the usual Bell type inequality.
What is the conclusion of all this? We started from one crucial
assumption: absence of “communication” between the two persons
once they are asked the questions. I will from now on revert to a
less anthropomorphic language and call this assumption “locality”
– assume that there is no causal connection whatsoever between the
two wings of the experiment. Then, we are led to a contradiction,
so that this assumption has to be dropped.
It is important to understand the logic of the argument: the
perfect correlations plus the absence of communication (i.e. locality) between the two wings of the experiments, leads us to postulate the existence of the variables $v_i(\alpha)$ [...]. However, merely assuming that
those variables exist leads to a contradiction with the experimental
results obtained when different questions are asked. To put it simply: locality plus perfect (anti)correlation implies hidden variables;
however, the latter plus statistics when different angles are measured implies a contradiction. Both the perfect correlations and
the statistics for different angles are empirical results; the theorem
is a theorem, namely a logical deduction; the only assumption was
the lack of “communication”, or locality. Hence, locality has to be
given up, period.
So, I'll stress again the most important point of this text: hidden variables are postulated to explain the correlations in the absence of communication (i.e. locality assumption). But this leads to a contradiction with the predictions of quantum mechanics. Quantum mechanics is right, we have tested it experimentally, therefore locality isn't right. I think that's fairly limpid. Anybody claiming that quantum mechanics doesn't force us to conclude non-locality is the one who doesn't really accept quantum mechanics, contrary to what detractors claimed in point 1.
In his book Speakable and unspeakable in quantum mechanics, which is really a collection of Bell's papers, there are various presentations of the theorem, some of which are clearer than others. Another book which does a good job at explaining the interpretation and consequences of Bell's theorem is Tim Maudlin's Quantum Non-Locality and Relativity.
- And I've barely addressed Bohmian mechanics yet and the question of the OP. Why do people still work on Bohmian mechanics? Well, the original intent of EPR was to show that quantum mechanics is incomplete. EPR showed that assuming locality inevitably leads to hidden variables, which is the first part of the previous argument. But the second part of the argument, provided by Bell shows that this is inconsistent with quantum mechanics. Therefore locality can't be right. One might wonder, even if we have to drop locality, why insist on supplementing the theory with hidden variables?
It's important to understand that the only variables added in Bohm's theory are the positions of particles. There are no hidden variables for spin, momentum, angular momentum, etc... It does however explain how the measured values for spin, momentum, etc... arise from the specific experimental configurations. This is what is called contextuality and has already been mentioned by @Sina Salek. Non-locality is explained by the fact that the wave-function is a function on configuration space and not in physical space. Hence the possibility of entanglement. You can read more as well as find further references in the article I mentioned.
Beyond the fact that Bohmian mechanics makes the non-locality in quantum mechanics more explicit, it also provides a deterministic interpretation of quantum mechanics, showing that quantum mechanics does not force randomness upon us.
Now, wether one likes Bohmian mechanics or not, one can not deny that these are definitely strong points of the theory. However, it does have its weaknesses as well. As @sb1 mentioned, Einstein thought it was cheap and in a way, I do agree that the way the extra equation for the positions is added is rather cheap and even ugly. It does also have its fair share of problems, for instance in trying to generalize into a QFT.
Anyway, I leave this link containing a short exposition of Bohmian mechanics and further references.
- Finally, I want to address @Matt Reece about QFT and locality. I'm no expert in QFT, but I do remember that locality is imposed on the level of operators by requiring that two observables localized within distinct space-time regions should commute. I'm not entirely sure what this implies w.r.t. entangled states, but I suppose it doesn't rule them out otherwise QFT would be in contradiction with experiment. (I wonder though if there is a complete treatment of entangled states within the context of QFT?) But if entangled states are allowed, then violations of Bell type inequalities still are possible and thus non-locality is a fact. What this means is that whatever the status of the principle of locality in QFT, it's a weaker form of locality than the one required in setting up Bell's inequalities.
