Pandas in a virtual environment does not require any global packages. Everything that is required is installed inside the virtual environment by pip including all dependencies. Also your /home/ubuntu/.virtualenvs/my-project/lib/python3.8/site-packages package location shows that pandas has been installed globally by pip3 outside the virtual environment, the same as would be the case if you had not activated the virtual environment with source bin/activate first.
Either you didn't activate your virtual environment before trying to install packages in it or your virtual environment is trashed. If activating the Python virtual environment doesn't help, then delete the virtual environment and create a new one from scratch. This time you will get the latest version of pip3 installed in the virtual environment by default, and you should be able to install Python packages in it with pip3 without using sudo.