It seems that the most natural way of choosing renormalization scale is to separate Lagrangian into physical Lagrangian and counterterms , like in Peskin & Schroeder's book, chapter 10.2, p.325, eq.(10.19), when discussing 4D $\phi^4$ theory, they define the the physical mass $m$ to be the position of pole of propagator, set field strength to $1$ and define the coupling constant $\lambda$ as the scattering amplitude at zero momentum at condition $s=4m^2,t=u=0$. The exact quote goes like:
My question is: after we have determine the renormalization scale using our observation data, how can we still move the renormalization scale, claiming that the physics does not change with renormalization scale and have renormalization group equation? if we define the renormalization scale using our observation, it seems to me that changing the scale will be equivalent to changing the condition we observe in the experiment, and the underlying physics might not necessarily be the same.
this answer and this answer mention that the renormalization scale have something to do with the energy scale of the system, yet if we change the energy scale of the system, it feels like we are also changing the underlying physics.
