You mention in the comments that your actual problem wasn't with '$HOME'. That's quite important since it matters if it is an environment variable or not.
Bash will complete possible environment variables first if you started with a dollar sign, finally followed by filename completion. So if it couldn't find a variable to complete for you, it could have found a filename, in which case it will escape the dollar sign with a backslash since that's a special character.
If you started your command with a program followed by some variables, completion could be handed over to programmable completion, which could also cause this. For example, typing gzip $HOME/ followed by tab will insert a backslash. The files handling programmable completion are owned by the package 'bash-completion', so either you can uninstall that package (I don't have in installed), or you can convince them it's a bug.
Maintainers: http://bash-completion.alioth.debian.org/
Update: on Arch Linux, I tried out bash-completion version 2.0 and that works as expected for ls $HO<TAB>, but not for ls $HOME/<TAB> so I guess they still not solved this bug (Ubuntu uses version 1.3). You can try manually installing version 2.0, but note that it doesn't work completely. (Since the bug report also mentions it could be bash, I'm using version 4.2.37 instead of 4.2.24)