I haven't addressed the root of the problem, but just kind of made it a virtually unnoticable.
So, the hack is quite simple. As the problem persists only when there is no playback, I came up with a hack to always have a playback!
Now, this should be true only when the headphones are connected. And the playback should stop when headphones are removed.
Follow these simple steps-
Step #1: Install mplayer and configure it. (mplayer because I am very sure that I won't be using it for playing music ever!)
sudo apt install mplayer
nano ~/.mplayer/config
type in that file-> lirc=no
Step #2: Download this mp3 file from here (silence-10sec.mp3)
Step #3: Create a folder in home directory and name it .hush
Copy the mp3 file in your .hush directory
cp ~/Downloads/silence-10sec.mp3 ~/.hush/
In that folder, create a script file and name it silence.sh
Copy paste this code in the script file.
#!/bin/bash
plugged="jack/headphone HEADPHONE plug"
unplugged="jack/headphone HEADPHONE unplug"
acpi_listen | while IFS= read -r event;
do
if [ "$event" == "$plugged" ]
then
mplayer -really-quiet -loop 0 ~/.scripts/silence-10sec.mp3 &
elif [ "$event" == "$unplugged" ]
then
pkill -f mplayer
fi
done
Make it executable with the command chmod +755 silence.sh
Step #4: Type startup applications in the Dash and add the path of silence.sh script to it.

So, basically what we are doing is listening to events of headphones been connected and disconnected and playing or stopping the blank mp3 file in a loop. Now, I have used mplayer because I know that I am never gonna use that for any other purpose and killing its process is completely safe for me.