my question is about dot product integrals such as work:
$$ \int_{\overrightarrow{a}}^{\overrightarrow{b}} \overrightarrow{F}(\overrightarrow{r})\cdot \overrightarrow{dr} $$
I want to write this as
$$ \int_{a}^{b} F(r) dr$$
where F is just dependent on the distance r from the origin (e.g. Coulomb-force).
You can not do this in general, because the integration does not just depend on the form of the force, it also depends on the exact path taken from $\vec a$ to $\vec b$ (in general); it is only path independent for a conservative force.
In the case of the coulomb potential due to a charge at the origin, you could perform the rewriting that you indicate. And, as you have correctly identified, you must properly assign the sign if by $F(r)$ you mean only the magnitude of the force.
Sometimes it seems that to go from one to the other you have to insert a minus sign, sometimes not. What is the rule here? Does it depend on the limits of integration, the direction of dr and/or the direction of F?
Yes, it can depend on these things. If the force is in the opposite direction as the displacement, then the dot product contains a $\cos(\pi)=-1$.
I find it helpful to be very explicit. You must also be ever vigilant against typos or other causes of accidental sign error.
Here is an example. Suppose I consider the classical force on an electron at location $\vec r$ due to a bare nucleus of atomic number Z fixed at the origin. This force is:
$$
\vec F_{e,Z} = \frac{-Z|e|^2}{4\pi \epsilon_0}\frac{\vec r}{|r|^3}\;,
$$
where we have a written $-Z|e|^2$ in the numerator of the first fractions since the charge of the electron is $-|e|$ and the charge of the nucleus is $Z|e|$. Note the force on the electron is directed radially inwards (which is explicitly accounted for by the minus sign, whereas everything else is manifestly positive (or is the vector $\vec r$)).
In this case, we happen to know that the force is conservative, but let's say for the sake of argument that we did not know that. Thus we specify the exact path $\vec r(s)$ from $\vec a$ to $\vec b$.
Supposing a linear path, we could write:
$$
\vec r(s) = \vec a + (\vec b - \vec a)s\;,
$$
where the parameter $s$ ranges from 0 to 1 as the electron moves from $\vec a$ to $\vec b$.
Thus we write the work as:
$$
W_{e,Z}^{\text{lin}}=\int_{\vec a}^{\vec b}\vec F_{e,Z}\cdot\vec{dr} = \int_{0}^1 \vec F_{e,Z}\cdot\frac {\vec dr}{ds}ds
$$
$$
=\int_{0}^1
\frac{-Z|e|^2}{4\pi \epsilon_0}\left(\frac{\vec a + (\vec b - \vec a)s}{|\vec a + (\vec b - \vec a)s|^3}\right)
\cdot(\vec b - \vec a)ds\;.
$$
Further supposing that the linear path is actual a linear radial path, we can write
$$
\vec r(s) = \hat r(a + (b - a)s)
$$
which further simplifies our expression for the work to:
$$
W_{e,Z}^{\text{lin,rad}}=\int_{0}^1
\frac{-Z|e|^2}{4\pi \epsilon_0}\left(\frac{a + (b - a)s}{|a + (b - a)s|^3}\right)
(b - a)ds = \int_{r=a}^{r=b}
\frac{-Z|e|^2}{4\pi \epsilon_0}\left(\frac{1}{r^2}\right)
dr\;,
$$
where the last equality follows from changing variables to $r=a+(b-a)s$.
The integral is easily evaluated to:
$$
W_{e,Z}^{\text{lin,rad}} = \frac{-Z|e|^2}{4\pi\epsilon_0}\left(\frac{1}{a}-\frac{1}{b}\right)\;,
$$
where we see that, if $b>a$ the work done on the electron by the force due to the nucleus is negative. This make sense because in this case the force and the displacement are in opposite directions. (On the other hand the work that I would have to do to actually make the electron move along the path is positive.)