Any metric space comes endowed with a function called the metric which measures distances between points, and by definition the metric is symmetric. The reason is simple: if you have two points $A$ and $B$, the distance between them shouldn't depend on your direction of travel. If it does, you aren't dealing with an ordinary sense of the word "distance", and need to use some more general concept.
The tangent space of a manifold is a metric space and the metric tensor measures distances between the endpoints of infinitesimal displacements in the tangent space (at least that's an intuitive if not-fully-rigorous way to say it). In order for the distance function in the tangent space $d(U, V) = g_{\mu\nu} U^\mu V^\mu$ to be symmetric $d(U,V)=d(V,U)$, the metric tensor $g_{\mu\nu}$ must be symmetric as well.
The above is a math answer since it uses a mathematical definition. The physics answer is that we use the mathematical abstraction of a metric space because it models the real world well. The distance measure we're interested in special and general relativity is the spacetime interval. It is worth pointing out (as others have) that the spacetime interval does violate one of the requirements for a definition of distance function defined above, namely that it is non-negative, and indeed we aren't interested in normal metric spaces leading to Riemannian geometry, but pseudo-Riemannian geometry. However, the spacetime interval reduces to more ordinary notions of distance up to a sign$^\dagger$, so the distance function should still be symmetric. We use this kind of mathematical object because it is a good model that makes predictions that agree with experiments. We expect differential geometry with a pseudo-Riemannian manifold to be a good model because the spacetime geometry we experience appears to be smooth and behaves like a manifold. However, you might notice nothing I'm saying is a proof it has to work like this -- if you want to explore the implications of the tensor field in general relativity having an antisymmetric part, you could join a whole field of researchers studying modifications of general relativity, but almost all modifications do not end up working for one reason or another and randomly changing an assumption in the math with no underlying physical motivation is usually a starting point of failure.
$^\dagger$ Depending on your sign convention, the spacetime interval between two points in the tangent space reduces to plus or minus the normal Euclidean distance on surfaces of constant time (and you can always do a (local) Lorentz boost to make two spacelike separated points lie on a surface of constant time (in the tangent space), and the interval is invariant under Lorentz boosts). Similarly, the spacetime interval reduces to minus or plus a 1-dimensional distance (time interval) on lines of constant spatial coordinates (and you can always do a (local) Lorentz boost to make two timelike separated points have the same spatial coordinates (in the tangent space)). And, of course, the spacetime interval is zero for all null separated distances.