John Rennie already gave the practical answer considering the atmosphere, noting that without doing anything objects near the ISS will deorbit quickly from drag. But that's letting reality get in the way of a good physics problem. I'll show that while a human can't send a ball crashing into the surface in one orbit, they can come close.
The ISS is listed as having a typical orbital velocity of $v = 7.66\times10^3\ \mathrm{m/s}$. For a test particle in orbit around the Earth, gravitational parameter $GM_\oplus = 3.98\times10^{14}\ \mathrm{m^3/s^2}$, a circular orbit (which we'll assume for concreteness and simplicity) will have a semimajor axis (i.e. radius) of
$$ a_0 = \frac{GM_\oplus}{v^2} = 6.79\times10^6\ \mathrm{m}. $$
It will have a specific energy of $\epsilon_0 = -GM_\oplus/2a_0 = -v^2/2$, and a specific angular momentum of $h_0 = a_0v$.
Your first instinct1 might be to throw the ball down with speed $\Delta v$. This will add velocity perpendicular to its current velocity, so the change in energy is simple: $\epsilon_\mathrm{down} = \epsilon_0 + \Delta\epsilon_\mathrm{down}$, $\Delta\epsilon_\mathrm{down} = (\Delta v)^2/2$. Because the added velocity is along the radial direction, angular momentum does not change: $h_\mathrm{down} = h_0$.
Given $\epsilon$ and $h$, we can calculate the corresponding $a$ and $e$ (eccentricity) according to
$$ a = -\frac{GM_\oplus}{2\epsilon}, \qquad e = \sqrt{1-\frac{h^2}{GM_\oplus a}}. $$
From there it's a simple matter to find the perigee according to
$$ r_\mathrm{per} = (1-e) a. $$
Plugging in numbers for $\Delta v = 100\ \mathrm{mph} = 44.7\ \mathrm{m/s}$, this gets us
$$ r_\mathrm{per,down} = 6.75\times10^{6}\ \mathrm{m}. $$
However, you can do better by throwing directly backward. This is the most efficient way to lower perigee. In this case, the new specific kinetic energy is $k_\mathrm{back} = (v-\Delta v)^2/2$, meaning the energy changes by $\Delta\epsilon_\mathrm{back} = k_\mathrm{back} - v^2/2$. The angular momentum also changes in this case: $h_\mathrm{back} = a_0 (v-\Delta v)$. Plugging in numbers gets us
$$ r_\mathrm{per,back} = 6.64\times10^6\ \mathrm{m}. $$
Now it turns out the highest point on Earth is Chimborazo, with an elevation above the center of the Earth of $r = 6.38\times10^6\ \mathrm{m}$. Thus you could not force the ball to hit any part of the Earth with with velocity. However, the effect is not negligible. If we look at values of $r_\mathrm{per}/r$, we started out at $1.064$ and got to either $1.059$ (throwing down) or $1.040$ (throwing backward).
How far into the atmosphere does throwing backward get you? According to this tool for the NRLSISE-00 atmosphere model, the density of the atmosphere going from an altitude of $415\ \mathrm{km}$ to $259\ \mathrm{km}$ increases by a factor of about $30$. Thus whatever drag the ISS experiences can be increased quite a bit with just a small change to the orbit.
How fast would you need to throw the ball to deorbit it in the absence of an atmosphere? If we throw backward, our old $a = a_0$ will be our new apogee. We want our new perigee to be the altitude $r$. Solving $r_\mathrm{apo,per} = (1 \pm e) a$ for $a$ tells us we want a new semimajor axis of $a = (a_0+r)/2$. Backtracking through $\epsilon = -GM_\oplus/2a$ this tells us what the new energy is and thus what the change in velocity must be (since we can only change kinetic energy with an instantaneous impulse). The answer is $120.\ \mathrm{m/s} = 268\ \mathrm{mph}$. This relatively small number compared to the orbital velocity is a reflection of just how close low-Earth orbit is to the surface compared to the radius of Earth.
1Unless you've played Kerbal Space Program.
