Just as the title of this question suggests; this post is a resource recommendation.
Does this belong on Physics SE or Mathematics SE? Well, that is a bit of tricky question to me in that there is undoubtedly a good deal of overlap between the mathematical and physical communities regarding the mathematical aspects of quantum mechanics. That being said, I am looking for resources specifically tailored to physicists (as has been noted below) as opposed to resources that teach quantum physics to mathematicians.
I am a little unsure of exactly how to ask this question in that I am not precisely sure of what type or types of mathematical methods that physicists address such issues.
I want to begin learning about how to define a well posed quantum mechanical problem. For example, there is more to the domain of a Hamiltonian than the square integrability of the states on which the operator acts. One must know about the domain of the operator, I am under the impression that the domain is dense, however, I don't know what that means. Perhaps it means something similar to when Hilbert space is said to be dense, i.e. without "holes"; every Cauchy sequence converges to an element of Hilbert space?
Basically, I want to properly define the Hamiltonian for a quantum system and to know whether or not some wave function or the other is within the Hamiltonian's domain of action.
I have had the usual courses in quantum mechanics; however, it appears that these more mathematical issues are usually swept under the rug. It is time for me to get a better grasp on them. I would prefer some good PDFs that can be found free online in addition to relevant and cogent stuff here on SE.
My thoughts are that operator theory and functional analysis are the proper mathematical branches to look into, however, these disciplines are rather large and perhaps not all of the material relevant to the mathematician is equally relevant to the theoretical physicist, thus something to tailored to physics would be what I am looking for.