Physicists use the word "potential" in different ways in different contexts, so there is no completely rigorous and general definition. There is a unifying idea, but unfortunately it's abstract enough that you probably won't understand all the jargon and concepts without some advanced physics training.
The general idea is this: consider a physical degree of freedom $x$, whose space of possible values forms some manifold $M$. Then a "potential" for $x$ is a field $V(x)$ defined on $M$ such that some kind of first derivative $V'(x_0)$ tells you how $x$ gets "pushed" if it takes on the value $x_0 \in M$.
When you first learn about potentials, it's almost always the case that the degree of freedom is the position ${\bf x}$ of a classical point particle, the manifold $M$ is physical space $\mathbb{R}^3$, the potential $V({\bf x})$ is a scalar potential energy field, and the "some kind of first derivative" is the negative gradient operator $-{\bf \nabla}$. In this case the potential $V(x)$ is simply a convenient way of encoding a position-dependent conservative force field ${\bf F}({\bf x}) = -{\bf \nabla}V({\bf x})$.
But as you get to more advanced applications, any of these can be generalized. To give a few examples:
- Rather than a single particle's position ${\bf x}$, the physical degree of freedom can be a field $\varphi(x)$.
- Rather than physical space $\mathbb{R}^3$, the manifold $M$ can be a proper subset, e.g. for a particle confined to the surface of a sphere. Even more abstractly, it doesn't need to be any kind of physical space at all; it field theory, the manifold is the set of values that the field can take on, which (regardless of the number of spatial dimensions), could be $\mathbb{R}$ for a scalar field, $\mathbb{R}^n$ for a vector field, or an even more abstract "spinor space".
- Rather than a scalar field, the potential $V(x)$ could be a vector field (as in the case of magnetism).
- Rather than the (negative) gradient operator, the "some kind of first derivative" could be the (negative) ordinary derivative (as in scalar field theory), or the vector curl (as in magnetism).
- Rather than a force, the "push" could be a force normalized by some suitable physical property of the degree of freedom, as in the electric field (force per unit charge) or gravitational field (force per unit mass, i.e. acceleration). More abstractly, it could be the generalized force that appears in the Euler-Lagrange equation in the Lagrangian formalism for particles, or the even more abstract one that appears in the Lagrangian formalism for fields.
