It seems to be common consensus that the world is non-deterministic and this is proved by Bell's theorem.
But even though Bell's experiments proved that the theory of quantum mechanics work, How does it prove the non-existent of local hidden variables?
Isn't it possible that there are hidden variables at work, and the results that were derived from these hidden variables coincide with the predictions of quantum mechanics?
Bell's theorems indeed rule out simple theories where hidden variables obey local equations. However, no matter how you reason, it's always at some point where you need another assumption. In its simplest form, it is the assumption that two observers, Bob an Alice, have the "free will" to choose along which axis they will measure the spin of a particle (photon, electron, or something else). Well, one could object that in a deterministic theory they have no such free will; their decisions were made in the far past.
But that does not invalidate Bell, because now you can say: Bell's theorem would imply that entangled photons emitted by a physical source are correlated non locally in an unnatural way with the nerves in Bob's and Alice's brains long before they made their decisions. That's called "conspiracy". So now the assumption is: there can't be conspiracy. Can't there? Spacelike non-local correlations in physical states are common in the physical world. In fact, in quantum field theory it's the propagators of all physical particles that describe correlations, and they do not vanish far outside the light cone. But the kind of conspiracy quantum systems seem to display (when described in terms of "hidden variables") looks disgusting to many researchers. So it is usually dismissed. Is "disgusting" a sound mathematical argument? You decide ...
This is a very specific question. Bell's theorem rules nothing out or in. Bell made the assumption that hidden variables existed, and using simple statistical arguments he derived a set of inequalities. If hidden variables existed they should make a measurable contribution to the correlatiions of spins. Therefore, if the measured correlations satisfied Bell's inequality it would support the existence of hidden variables. But if the inequality is violated then the assumption about the existence of hidden variable is false and quantum mechanical predictions are correct. This is similar to the proof by "reductio ad absurdum" in geometry or pure mathematics. All experiments that have checked Bell's inequality so far, have shown that the experimental data violate them. Thus hidden variables do not find support by experiment!
Bell made assumptions that need not be valid at all. One can dispute, for instance, his assumption of "statistical independence". That's, roughly, saying that if you don't know all the data, it is perfectly sensible to assume that all variables that you did not measure will come out with equal probabilities. Saying that this is disputable, is now ridiculed as being "unscientific". One thing I noted is that Bell made no reasonable models of deterministic theories. At least he doesn't mention any such attempts. What he uses is some strange mix of classical and quantum mechanical processes and then he arrives at contradictions. I did make models, and I am coming really close to models that predict the same dynamical processes as QM does. According to one such model you only need to assume the existence of dynamical variables that evolve too fast to allow us to follow them in detail. See arxiv:2103.04335 [quant-ph]. So we are tempted to simplify the dynamics, and then it seems as if variables go into superpositions. Superpositions merely are caused by our lack of knowledge, exactly as Einstein always suspected. I make no appeal to "conspiracy" or "retrocausality". My classical models ("hidden variables") wouldn't know how to conspire. It is often claimed that models like these would require non-locality. No, they don't. The most sophisticated models (classical cellular automata) are as local as one could ask for (nearest neighbour interactions only).
The term "local hidden variables" is a poor expression, and I haven't found the term used by John Bell anywhere. Bell showed that any theory with pre-existing spin cannot produce the right correlations predicted by quantum mechanics. Almost nobody I talk to seems to get this. Tim Maudlin is one. I highly suggest reading section 3 of this paper. It really explains Bell's theorem, and you won't walk around as so many do with the belief that "local hidden variables" is what Bell ruled out. He ruled out pre-existing properties (specifically spin) in any theory, local or not.
