Q. The gravitational field due to a mass distribution is given by $E=k/(x^3)$ in $x$-direction. Taking the gravitational potential to be zero at infinity, find its value at a distance $x$.
for the answer let the 'integral where the lower limit is infinity and upper limit x' be denoted as I
The answer my textbook gives to the above question is :
shouldn't the dot product of E and dx (because we are traveling from infinity to a point) give - sign so the final answer should be negative.
While we derive the expression for line intergral we don't care for the sign of $\Delta x$ because that already taken into account with limits of integration.For example : consider the example given below in the comment box.
So it's as given in the book.
The final answer is written correctly, without a negative sign, as shown in the printed solution. The field is pointing in the +x direction.
$\vec{E} = -\frac{\mathrm{d}U }{\mathrm{d} x} \hat{x}$
If the potential at x was negative as your are suspecting, and 0 at infinity, the derivative is > 0 and negative derivative is <0 , giving a field pointing in the wrong direction. See the link posted:
The question implies that there is a mass at origin. Since gravitational force is always attractive and when we move from $\infty$ to $x$ we travel in the direction of the force, the cross product $\vec{E}.\vec{dx}$ is positive.
Hence, $V(x)=-\int_{\infty}^{x}Edx$
The rest is according to the book.
The question states that "...mass distribution is given by ... in X-direction", which can be interpreted as: $$\vec{E}=\frac{K}{x^3} \hat i$$
At the very beginning of section 11.7 of the said book, it can be seen that the equation $V(\vec{r})=-\int_{r_0}^{r}\vec E.d\vec x$ has been derived by assuming that the particle is displaced from $\vec r$ to $\vec r +d\vec r$, but no assumption has been made about the direction of $\vec F$ or $\vec E$.
Hence the direction of $d\vec r$ would be implicitly taken care of by limits of the integral. But we can set the direction of the field aa per our need. Check J.G's post here
The integration would proceed like this:
$V(\vec{r})=-\int_{r_0}^{r}\vec E.d\vec x \\ \Rightarrow V(\vec{r})=-\int_{\infty}^{x}\frac{K}{x^3} \hat i . dx \hat i \\ \> \> \> \> = -\int_{\infty}^{x}\frac{K}{x^3}.dx$
SIDENOTE: In H.C. Verma's book, it is given that $V(\vec{r})=-\int_{\vec{r_0}}^{\vec{r}}\vec E.d\vec x$. But it must be noted that the limits of the integration cannot be taken as vectors.
