There are at least three different grounds on which someone running a private game server without permission might be successfully sued by the original developers or operators of the game. These are Copyright infringement, Trademark infringement, and Interference with contracts.
Copyright Infringement
It is unlawful to make a copy of a protected work without permission. It is also unlawful to make a derivative work from a protected work without permission.
If a server is reverse-engineered starting only with the observable behavior, it is probably not a copy in the sense used in copyright law.
US copyright law (17 USC 101) says:
A “derivative work” is a work based upon one or more preexisting works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which a work may be recast, transformed, or adapted. A work consisting of editorial revisions, annotations, elaborations, or other modifications, which, as a whole, represent an original work of authorship, is a “derivative work”.
The law in other countries is similar.
However, reverse-engineering has in a number of US cases been held to be a fair use. This is a strictly US legal concept which does not apply elsewhere. Sega Enterprises v. Accolade and Sony Computer Entertainment v. Connectix held that in particular circumstances reverse engineering of video game systems to allow interoperability was a fair use. However in Compaq Computer Corp. v. Procom Technology, Inc. and Blizzard v. BnetD reverse engineering efforts were held to be improper. In the Blizzard case this was based on violations of the ELUA and ToS by people who had agreed to them, and then proceeded to do reverse engineering contrary to specific provisions of those agreements.
Whether something is fair-use is a complex, fact-driven issue. Anyone planning on undertaking such an effort based on a fair-use claim would be wise to secure competent legal advice in advance.
Some non-US countries have a legal concept of fair dealing which is somewhat similar to, but more limited than, the US concept of fair use.
Trademark Infringement
The name of a published game is usually a protected trademark. Logos, images, slogans, and tag lines may also be protected as trademarks. Using a trademark to advertise a competing product – even a free one – is trademark infringement.
However, if the original product is no longer on the market, its trademark protection may have lapsed. Trademarks are only protected when 'used in commerce". If the mark is no longer used, and there is no indication of future intent to use the mark, it may no longer be protected.
If a trademark is used in a comparative sense such as a statement that "Game X is more fun than Game A" this is generally not trademark infringement, as long as no reasonable person would be confused into thinking that the product or service was made by, authorized by, or endorsed by the makers of the protected product. This sort of use is generally limited to the name, and does not permit the use of a logo or other image, nor of slogans or the like.
Similarly, a product may be described as being compatible with a product from a different source, without this being trademark infringement. (E.g., "Battery A can be used with Brand X tools"; "GreatBlade fits all SmoothShave razors.")
Both of these are examples of Nominative use.
Relying on a determination that a mark has lapsed can be tricky, and it would be wise to consult a trademark lawyer before doing so. Nominative use is a bit clearer, but consulting an expert is still often wise. Both of these are fact-based situations and different cases may come out differently.
Interference with Contracts
If an existing game operator has contracts with game users, and a clone attracts such users causing them to break contracts with the operator, there may be a case for Tortious interference with a contractual relationship. This would allow the original operator to sue the clone operator and obtain damages.
However, this theory is based on inducing users to break existing contracts. If the previous game is not in operation, and there are no current contracts, then such a suit would not be viable.
