Can the measurement problem be overcome?

Question

I was listening to some physicists discuss the issues with measurement in quantum mechanics and some of the earlier philosophical repercussions. However in most cases where measurement affects a system we can account for the perturbation. Why is this not possible in quantum mechanics(perhaps even statistically)? And has there been any experiments to attempt this?

Answer 1

In science, "why" is often quite an unsatisfying question. Science is not focused on the "why" as much as predicting behaviors. Why those behaviors happen is a philosophical question. In many cases, we can answer "why" by invoking a more fundamental physics, but at the present, quantum physics is as fundamental as we typically get.

We can speak mathematically:

Mathematically, in wave mechanics, the uncertainty relation between position and momentum arises because the expressions of the wavefunction in the two corresponding orthonormal bases in Hilbert space are Fourier transforms of one another (i.e., position and momentum are conjugate variables). A nonzero function and its Fourier transform cannot both be sharply localized at the same time.

If you are comfortable with the claim that measuring position and measuring momentum involves measuring a function and its Fourier transform, then the answer becomes quite simple. Mathematically a signal that is perfectly localized in time (an impulse) has an infinitely wide bandwidth in a frequency space, and a signal that is perfectly localized in frequency requires an infinitely wide range of time. And with that you say "case closed," put your pencil down, and get to drink your coffee before it gets cold.

If you're not comfortable with the idea that position and momentum form a conjugate pair (a fancy way of speaking to the Fourier-related relationship above), then we have to speak to the large quantity of evidence suggesting that relationship does indeed exist. Which is not always satisfying, but science does have to pay attention to the data more than it has to pay to satisfaction.

As to whether experiments have been done to challenge this, the answer would be a resounding yes. This limitation is a very frustrating one in quantum physics, and there is indeed a great deal of effort devoted to trying to deal with the measurement problem. The "best" approaches I am aware of are "soft" measurements, where one avoids measuring both values of a conjugate pair, and instead finds other measurements that are useful.

Answer 2

However in most cases where measurement affects a system we can account for the perturbation. Why is this not possible in quantum mechanics(perhaps even statistically)? And has there been any experiments to attempt this?

The measurement problem isn't primarily about perturbation. A measurement is an interaction that produces a record of some quantity that can be copied. In quantum theory a system can be in a state that is a superposition of two or more different possible measurement outcomes, e.g. a photon $p$ in a superposition two different arms of an interferometer: $|arm1\rangle_p+|arm2\rangle_p$. If the a measurement device $m$ interacts with this photon and records the result, then the state is $$|arm1\rangle_p|arm1\rangle_m+|arm2\rangle_p|arm2\rangle_m$$

But no physicist has ever seen a measurement device record that claims a photon is in both arms of an interferometer, so the problem is how we should understand such a state or come up with a replacement for quantum theory that doesn't appear to show two different results for the same measurement:

https://arxiv.org/abs/0712.0149

Some physicists want to pretend this problem doesn't exist and just do calculations of predictions with quantum theory. This approach leads to many problems. First, a prediction involves saying what is happening in an experiment and if you're not going to discuss that then it's unclear how you can seriously claim to be making a prediction. Second, if you want to make progress in physics you should want to understand the current best theory so you can find problems with it and come up with a replacement that solves those problems.

Some physicists want to replace quantum theory with a theory in which the measurement doesn't show two different measurement results, e.g. - pilot wave and spontaneous collapse theories:

https://arxiv.org/abs/2310.14969

https://arxiv.org/abs/1906.10761

Most predictions of quantum theory are now made with quantum field theories that include relativistic effects and neither of those variants of quantum theory reproduce those predictions:

https://arxiv.org/abs/2205.00568

In addition because of entanglement experiments such a theory would have to be non-local and non-Lorentz invariant, which means you have to replace relativity too:

https://arxiv.org/abs/1808.04966

The core of the problem is that the only existing account of what happens in reality to bring about the outcomes of quantum experiments involves systems being in multiple states that can interfere with one another. For example, changing the arm lengths in an interferometer with only one photon going through it at a time changes the probability of the outcome:

https://arxiv.org/abs/math/9911150

How do you explain that without saying there is something that acts like a photon in both arms even though you only see one of them when you do a measurement? One way to do this is to push the example I gave above further and work out what it actually implies for what you would expect to see. The measurement device will interact with the rest of the world including you, call this the environment $e$ so you get $$|arm1\rangle_p|arm1\rangle_m|arm1\rangle_e+|arm2\rangle_p|arm2\rangle_m|arm2\rangle_e$$ The process of copying information out of a system suppresses interference: a process called decoherence so you wouldn't expect to be able to see both versions of the result. Rather, there would be two versions of you and each version would see one result:

https://arxiv.org/abs/1111.2189

https://arxiv.org/abs/0707.2832

This is often called the many worlds interpretation of quantum theory and it doesn't have problems with locality or interference experiments:

https://arxiv.org/abs/quant-ph/9906007

https://arxiv.org/abs/1109.6223

Quantum experiments, such as interference and entanglement experiments, can be explained in terms of quantum theory but not in terms of theories in which every system is in only one state at a time. People don't like this explanation but the other attempts at explanations have problems like contradicting lots of existing physics without providing a replacement and that is the core of the measurement problem.

Answer 3

There are actually a number of problems, or different aspects of the measurement problem:

1. Certain combinations of observables, like position and momentum, cannot be independently measured.
2. Measuring a system will influence the system and unlike in classical physics we cannot (at least not always) make that influence arbitrary small.
3. It appears to us that a measurement projects the state into one of the possible outcomes, whereas the unitary time evolution of QM gives a different result: a superposition of all of the possible outcomes.

Point 1) was already discussed in [the other answer, #283846] and is actually resolved if we see wave mechanics as the more fundamental theory, at least more fundamental than QM based on postulates about observables. For wave functions and their Fourier transforms this behavior is natural, and most of us would in fact see it as a problem if we would not have this problem with measurements. So that covers point 1).

As for point 2), it is of course not unreasonable that interacting systems influence each other, so we could say that in classical physics we were just lucky to be able to reduce this influence to zero. And in practice we even couldn't, because we have no infinitely light test particles, so perhaps this is a bit of a red herring.

Point 3) then, is the main problem, which also leads to the different "interpretations" and brings us to the question: "Can the measurement problem be overcome?" Obviously for point 3) there are two ways out. Firstly, we could try to get rid of the unitary time evolution of QM. Changing the time evolution, however, is not easy and the [objective collapse theories] trying this, have problems with basic things like conservation of energy and Lorentz covariance. Alternatively, we could get rid of the collapse into one possible outcome, by arguing that this is only our subjective perception since we ourselves become part of the superposition. This is usually called a "many worlds" view, although it need not have all aspects of popular explanations, like the "splitting of the universe" by a measurement. In fact we always keep one and the same Hilbert space of states, we just have a superposition developing where the terms become gradually less interacting due to decoherence. Obviously (but sometimes forgotten) it's also possible that QM will turn out to be non-unitary but still have no collapse, so "many worlds" would still be true. I my view we can summarize it by saying that it's easier to envision that the macroscopic limit of QM will turn out to be a classical multiverse than a classical universe. But it is not generally accepted that this limit behavior has already been proven (the discussion about ["reproducing the Born rule"]). We will see...