Looking at the picture below, you can totally see that these are tiny boats.
The water is practically washing over the hull of these boat models.
But the water has boundaries that are very far away, and even if the water is very deep, it would (at least in my mind) not make a big difference to the size of the waves after just a few inches of depth.
What dictates the size waves in the open water? Is it the molecular properties of the water?
This is really just a footnote to Ross' answer as Ross is quite correct and the link he provided contains the information you asked for.
In the open sea waves are normally produced by the wind. When the wind hits the sea surface it creates essentially random patterns of pressure variation and these lift some parts of the surface up and press others down. Over time this creates waves. The important variable is the wind speed. There isn't a precise windspeed - wavelength relationship because the random nature of wave generation means you get a spread of wavelengths. However it's generally true that low wind speeds produce short waves and high wind speeds produce long waves.
I did a quick graph of the average wavelength against speed from the data in the Wikipedia article and I get:
The data doesn't include low speeds, but the trend is obvious.
I don't think there's a simple explanation for exactly why the wavelength depends on wind speed. According to the Wikipedia article the pressure variations created by the wind create small waves, but the wind then interacts with these waves and increases their wavelength. This is known as the Miles mechanism, and if you're sufficiently interested there is a paper about it here. However you'll need to pay to see more than the first two pages.
I like your photograph, but as I'm sure you realize that is not the open ocean. (I didn't notice the painted background the first time I looked at it!) In the pictured situation, the waves are going to be highly influenced by the nearby man-made structures, the shore, and the bottom (if this area is relatively shallow.) Wave reflection, refraction, and the driving action of incoming waves from outside (this bay?) will together determine how the surface waves behave in this local area. After all, consider that this is the reason structures such as breakwaters are built at the mouth of harbors - to control wave action within the harbor. When marine engineers and oceanographers design a harbor they have to use computer models to predict the net effect of natural bathymetry and proposed structures and external wave sources - simple models are just inadequate for the job.
Although your suggestion of simplifying the problem by removing it to the 'open ocean' is a reasonable one, in fact the causes of the wave spectrum at a particular point and time in the open ocean is usually complex too. Waves can travel across the world's ocean basins - and the source of the waves locally can be far removed from where you are observing them. As Ross says, ocean waves can originate as wind waves, they can be generated by weather patterns, and sometimes an earthquake generates a tsunami. Wind velocity, duration, and fetch are important for creating wind waves. After waves are instantiated, they travel far distances from the area of their source undergoing reflection, refraction, diffraction, and dispersion over many distances.
Ocean wave data can now be gathered from space, so there much more observational information than 20 years ago when I studied physical oceanography. At that time, the explanations were not much more than Ross's answer, but I think the opinion (given the new availability of global data) was that the field was set to rapidly expand and advance. I would be interested to hear from someone current in this topic about what we have learned about ocean waves in the last decade!
This page on the WikiWaves.org site discusses the physics of ocean wave spectra. Ocean-Wave Spectra