There are various tangential ways in which this could be illegal, for example if your subjects are celebrities, you take a picture of them and commercially exploit it without permission in a product endorsement. Leaving aside such fringe cases, in the US, the legal right to privacy comes about, at the first cut, by premise trespass law. If the proprietor tells you to go away, you have to go away; if the proprietor tells you that you cannot take pictures, you cannot take pictures (your right to enter is conditional). Neither of those circumstances holds in your case.
There are other tort-law bases for a right to privacy: numerous privacy laws regarding privacy and financial transactions (not relevant here), the aforementioned right of publicity (commercial exploitation of likeness), false light (like defamation, about creating a false impression – I don't see what false information is conveyed by a photo). There is also public disclosure of private fact, but that cat is out of the bag because the subject has self-disclosed the supposedly offensive fact revealed by the picture by eating in public. Intrusion of solitude and seclusion does not exist in the circumstance, since the subject is eating in public where everybody can see: there is no reasonable expectation of privacy.
This page sums us Virginia law. The main take-away from that is that there is no common law action for privacy in Virginia, and only an action for unauthorized used of likeness or name.
