Experience tells that if we pull a piece of thread, it forms a straight line, a geodesic in the Euclidean space. If we perform a similar experiment on the surface of a sphere, we will get an arc of a great circle, which is also a geodesic.
How to show this in general, for any geometry?
As @Fardin pointed out, you only get a straight line in the absence of gravity. While gravity is active, the string will form a catenary.
In general, the string will try to follow the shortest distance between the 2 endpoints. If it didn't do that, the tension on the string would try to shorten it. The definition of a geodesic is that it is the shortest path between 2 points on a surface. In flat space, a straight line is that shortest distance; on a sphere it's a great circle.
EDIT
To answer a comment by @MonsieurPeriné, here's my intuitive answer why tension straightens the string.
Assume there is a point A on the string that is not on the straight line. At A, the tension obviously deviates from the straight line. If the string were a rod, that tension would create a bending moment that would try to twist the rod around one of its endpoints. That would try to move A closer to the straight line. As the string is not a rod, it will deform instead, until all points are on the line (or geodesic).
For the mathematical reason see the answer by @linkhyrule5.
@hdhondt gave a good conceptual answer to this question, namely "if it wasn't the shortest path, the tension would try and shorten it." Here's the same answer, but less concise and in math.
We begin with the Hamiltonian of a stretched string: given a differential element of relaxed length $dx$, if it is stretched by some length $\delta x$, the energy stored in that element is $\kappa \frac{\delta x}{dx}\delta x$, $\kappa = E A$ the Young's modulus times the cross-sectional area.
Okay, so we want the shape of the string in some real space. We can do that by defining a coordinate system (origin and unit length) along the idealized relaxed string, and then create a map $f_i: \mathbb{R}\to\mathbb{R}^n$ that takes in relaxed coordinates and returns $n$-dimensional spatial coordinates (indexed by $i$).
Here's where the metric comes in: for a small enough $dx$, the real length of the string-under-tension is the norm $\left\Vert f_i(x+dx) - f_i(x)\right\Vert$. Since $dx$ is just a scalar, we can therefore say
$$1+\frac{\delta x}{dx}=\left\Vert \frac{f_i(x+dx) - f_i(x)}{dx}\right\Vert=\left\Vert f^\prime_i(x)\right\Vert$$
Then the total energy of the stretched string is
$$H\left[f_i\right] = \kappa \int_0^L \left(\left\Vert f^\prime_i(x) \right\Vert - 1\right)^2 dx$$
Right -- but the physics of the problem tells us that when you apply tension to a string and permit it to reach equilibrium (constant tension throughout), the string is only ever going to get longer, i.e. the term being squared is nonnegative. Then minimizing $H$ amounts to to minimizing $\Vert f_i^\prime(x)\Vert$ -- i.e. minimizing the length of the string according to whatever metric it lives in, which means a geodesic.
Gravity mixes this up a bit in a couple of ways, mostly because of that pesky negative sign in the metric -- warped spacetime is allowed to make the string behave in ways that warped space alone isn't. (The easiest example is the classic problem about a space elevator, where the tension in the line does indeed vary throughout.) The fundamental issue, afaict, is that if you're tracing a trajectory through spacetime, it no longer makes much physical sense to minimize the energy; you should extremize the action instead, which I believe will give you a sign change that leads you back to the same line of logic.
If we pull a piece of thread, it should form a catenary curve (a curve resembling a hyperbolic curve) I guess!
The thread's length is denoted $l$. The shortest distance between the endpoints is called $d$
When you pull on the endpoints of a thread in opposite directions, you are ultimately increasing the distance between the endpoints. When $l>d$, the thread is necessarily not straight; if it were straight, where would the extra thread go?
If you pull the endpoints far enough apart, $l\le d$. For the case $l = d$, the thread becomes a line between the two endpoints, and is thus straight. One could say the case $l <d$ is not defined (if one views the length as dynamic), but if one were to define it, one could say $l$ is the non-stretched length. Then, for $l<d$, the thread is thinning and stretching, and will eventually break.
