By "referring to notes", I don't believe that means notes that have been taken by the witness during the trial. In certain cases, the witness can refer to notes prepared ahead of time to refresh their memory about something. They can't simply read content directly from the notes, though, they have to have an "independent recollection" of the facts. "Notes" here can actually refer to more than just something the witness wrote for the purposes of the trial, it could also be a report, a photograph, or a physical item. As an example, in State v. Brown, 350 N.C. 193 (1999) a witness saw a letter she had previously written, which refreshed her memory about certain statements she had made.
If this is done during testimony, Rule 612 of the Federal Rules of Evidence enables the opposing party to inspect the notes in question, cross-examine the witness about the notes, and enter any relevant portions of the notes into evidence. That means you have to be careful about what notes you use because if you (for example) had notes of a conversation between yourself and your attorney, you would waive attorney-client privilege the moment you use them on the stand.
From a practical standpoint, I'd imagine this makes it very difficult for a witness to take notes while on the stand. Opposing counsel will see you with notes, they'll demand they be turned over for inspection, and now you don't have anything to write on any more.