When trying to explain electricity, it is common to use the water analogy. This can be seen here and in Wikipedia. There are some more good links, explanations, and pictures in this Q&A.
To explain your battery example, think of the battery as a reservoir of water that is capable of some pressure (pressure is analogous to voltage). Think of your meter as a very long pipe with a device to measure pressure at the end. If you just connect the pipe to the output of the reservoir, the water will go up the pipe to a certain level but never reach the pressure sensor.
In your experiment with the battery and the PC chassis, you had two reservoirs with pipes. One being the battery the other being the PC power supply. The only difference is that the power supply had the pipe connected to the input. Neither sensor has water flowing past it so neither sensor can measure any pressure.
The only way to measure the pressure of the battery reservoir is to loop the pipe back to the input. That way the water flows from the output back into the input continuously. This way the water continuously passes the pressure sensor and can be measured.
The reservoir always has a pressure. But that pressure cannot be measured unless it is allowed to flow. And in electricity, in order for there to be flow, the electrons must be allowed to return to the source.
I've never seen an electrical analogy to software. But since you say you've worked purely in software, I'm going to attempt one. Let me know if this helps your understanding (or if it only confuses you more) because I'm always in need of a better way to explain electronics to software engineers.
As has been explained in other answers, voltage is a potential difference. The key word is difference.
Take the following code:
cout << "abc";
What is the difference between the location of 'a' and the location of 'c' in the stream?
Answer: 2 characters
Now take this code:
cout << "abc";
cerr << "xyz";
What is the difference between the location of 'a' and the location of 'z'?
Answer: Undefined because "abc" and "xyz" are in separate streams. You can't measure the location of a character in one stream in reference to the location of a character in a different stream.
Location of a character in a string isn't a great analogy for voltage/pressure. But hopefully the concept of measuring between two separate streams translates to measuring the pressure of one reservoir with respect to another or the voltage of a battery with respect to a power supply.
Edit:
In your question, you state, "wired a single terminal of an AAA battery through a meter to the chassis of my PC". This leads me to believe you connected one lead of the meter to the + of the battery and the other lead of the meter to the chassis of the PC.
If there was a complete circuit and if the meter leads were in teh correct connectors on the meter, this would be how you measure current, not voltage. Have a look at this page for a general discussion on the difference between measuring voltage and measuring current. And have a look at this tutorial for some basics on using a meter.
If you connected the + of your battery to your chassis through a wire and then measured from chassis to earth ground, you would get 0V. This is because chassis and earth ground would be a the same potential and therefore there would be no difference.
Electrons won't go on a trip unless they know they can come back home. When you connect the battery to the chassis using only one wire, the electrons won't leave the battery because they know that earth is not their home. The - of the battery is their home and they won't budge unless they know they can return there.
If you connect the - of the battery to the chassis and measure from the + terminal to earth ground, then you should see a voltage because you are connecting earth ground to the electrons home base at the - terminal. And as long as there is a path back home, they will take the trip, current will flow, and voltage can be measured.
Edit 2:
First off, DO NOT connect an ammeter directly to a battery without a load. The ammeter is a very low resistance and you will essentially short out the battery. The battery will then get very hot in short order. Always connect a resistive load in series when using an ammeter.
You are correct in stating that an ammeter is just a galvanometer. A voltmeter is just an ammeter with a series resistor. So when you are using the voltmeter directly across the battery, you have a complete circuit with a load. That is why you measure a voltage when you measure directly across the battery.
If you use an external resistor in series with your ammeter and complete the circuit, current will flow and you will read some amount of milliamps depending on the value of the resistor.
I'm an engineer and you're a physicist. So when you ask "is this due to the chemical nature of the battery?", we're both getting out of our areas of expertise. Although you're probably better off because I can tie this back to physics principles that I probably don't understand as fully as you do. So while I can't give you a good, complete answer, I can point you towards some search terms to help you understand it a bit more.
The field of study is known as electrochemistry.
The flow of electrons is always from anode to cathode outside of the
cell or device, regardless of the cell or device type and operating
mode, with the exception of diodes
And the reason for this is the physical principle of charge conservation:
Charge conservation is a physical law that states that the change in
the amount of electric charge in any volume of space is exactly equal
to the amount of charge flowing into the volume minus the amount of
charge flowing out of the volume.
Sorry about all the Wikipedia links but it's as good a place as any to start from. Once you have the terminology, you can ferret out more complete and definitive sources of information.
