MCU: ATTiny13
I noticed this after trying to debug why pushing my switch (connected via R2, a 507kOhm pulldown resistor) makes the LED dimmer while depressed. The switch was powered by the same supply line as the Vcc input to the microcontroller.
Upon disconnecting the Vcc input (Pin 8), I noticed that the LED was still lit when the switch was depressed. If I removed a connection from the ground pin 4, the LED still lit up, but less brighter.
The circuit below represents what I observed. The switch is removed to simplify the problem:

Why does this happen, and how can I stop it? It is interfering with the output when the button is depressed.
Here is a picture of the circuit on a breadboard. The supply line (5V is the red wire, Ground is black):



