“Venomancer” was used prior to Dota; it’s a fairly common gaming term for a poison-using mage. It’s a generic combination of “venom” and “-mancy,” a productive suffix in English that just means “type of magic.” Dota was far from the first to use that particular combination. The term was used (interchangeably with “poisonmancer”) in the Diablo II community as a term for poison-focused necromancers, for instance—and that wasn’t the origin of it, either, I’m sure.
Even in the Dota context, IceFrog got the character from Guinsoo’s Dota All-Stars, who got it from Eul’s original Defense of the Ancients.
This matters because intellectual property law often cares about “prior art,” and the Venomancer character—even specifically the Dota one—isn’t really Valve’s. Warcraft III map-makers were constantly stealing each other’s work, and there’s no way that Valve tracked down everyone who had a hand in this and purchased whatever rights they had to things. Eul and IceFrog work for Valve now but Guinsoo works for competitor Riot, for example. And it’s not as if any of this was chronicled or recorded terribly well. And beyond the specific Dota character, the term “venomancer” is so generic—and so widely used—that no one can have any rights to it.
So if you write a character who uses poison magic and is called “venomancer,” or even “The Venomancer,” you’re really just using a word. An uncommon one, but by no means necessarily specifically reference to any given character, even if Dota’s is the most well-known at this point. After all, the Dota Venomancer has a name (Lesale Deathbringer); “venomancer” is just a title. And even if it did, Valve would struggle to defend their claim to that character anyway.
Whether or not you should is a separate question, though. A large portion of your audience might immediately think of the Dota character, which may not be what you want.
I would check to see if there is a trademark affixed to the word 'venomancer.' A tentative google search seems to deny it. If there isn't, I don't see a problem with using it.
Another thing: if you want, you can spell it differently: Venommancer, Vainomancer, etc. It doesn't really work well with this name, though.
If you do not want to change the name and you cannot find if it is trademarked, email/call the people who created the name and check. If you make it different in appearance than the creature in the video game, I don't see a problem with it--assuming there is no trademark, of course.
In addition to the legal question, there's also the question of whether you should, as in, will it be off-putting to readers?
I think the rule of thumb there is that if it's not a really widely-known IP then it won't be a problem. Or on the other end of the spectrum, if it's something very generic and deep in the target culture's mythology (like elves for instance) then it's also ok.
The MMO "Perfect World" has a spellcasting class called Venomancer. They can tame pets and turn into foxes.
The way copyright lawsuits work is, the plaintiff makes a list of all the alleged similarities between the two works.
So you can still get in trouble for using elements that weren’t original to the work you’re being sued for copying, if you use enough different elements from the same work. Famously, J.R.R, Tolkien took inspiration from his mythology. Gary Gygax tried to defend himself against a copyright lawsuit by the estate of J.R.R. Tolkien by claiming that he hadn’t stolen from Tolkien, he’d stolen from the same sources Tolkien stole from, but he ended up needing to settle the lawsuit anyway. One reason most observers didn’t buy Gygax’s explanation is that, in many cases, Tolkien had borrowed a name from the public domain, but done something original with it, and then Gygax had clearly ripped off, for example, Tolkien’s Orcs rather than the sea monster from Greek mythology, Tolkien’s Hobbits instead of the Brownies from Medieval England, Tolkien’s particular version of Elves and Dwarves, and so on.
So, if there’s something called a venomancer in both books, but they’re very different, that’s probably fine. If “Venomancer” is not on the cover, it’s probably not even being used in a trademark context. If they’re basically the same character with the same name, though, that might be a problem. And if there are enough other similarities between the two, something blatant like using the same name could convince a jury that the others are no coincidence, either.
