In this case, "measuring an operator" is meant to describe measuring the observable associated with the operator. More commonly one would put a "using" or "with" or "with respect to" between "measure" and "a given operator". As usual, the operator's eigenvalues correspond to measurement results for the observable, and measuring the observable collapses the quantum state to a corresponding eigenstate of the operator, with a probability for each orthogonal eigenstate equal to the probability amplitude of the eigenstate multiplied by its complex conjugate. The Pauli Z operator is the operator for measuring on the 1-qubit computational basis, with a measurement of $\left|0\right>$ being associated with an eigenvalue of 1 and $\left|1\right>$ being associated with an eigenvalue of -1, and the Pauli X operator is similarly associated with the Hadamard basis, to name two.
The reason the expectation value for this is given by $\left<\psi\right|A\left|\psi\right>$ can be seen through noticing what happens when $A$ acts on each of its associated eigenstates, which $\left|\psi\right>$ is in general in a superposition of. For the finite case, with the set of eigenvalues of the operator being denoted $S$, you can represent the quantum state as $\left|\psi\right> = \sum_{\lambda \in S}a_\lambda\left|s{_\lambda}\right>$, where $A\left|s_{\lambda}\right> = \lambda\left|s_{\lambda}\right>$ and $a_\lambda$ are the associated probability amplitudes. Then $A\left|\psi\right> = \sum_{\lambda \in S} \lambda a_\lambda \left|s_\lambda\right>$. With $\left<\psi\right| = \sum_{\lambda \in S}\bar a_\lambda\ \left<s_\lambda\right|$ and all the different $\left|s_\lambda\right>$ being orthogonal, we get $\left<\psi\right|A\left|\psi\right> = \sum_{\lambda \in S} \lambda a_\lambda \bar a_\lambda$. With the probability amplitudes multiplied by their complex conjugates being the probability of measurement, this is the traditional expected value formula.