By increasing the frequency of electromagnetic wave in visible range, human eye sees it as a color change.
But how human eye will see transition after violet wavelength to ultraviolet? As far as I understand ultraviolet (at least middle and far ranges) is invisible for eye.
So if imagine some source that can smoothly change frequency of the radiation from violet to (far) ultraviolet, how human will see it?
Will it just smoothly disappear?
The sensitivity of your cone cells (the ones that see color) varies with frequency, as shown in this nice graph from Wikipedia.
If you had a monochromatic light source of constant intensity, but could smoothly vary the frequency, it would appear to your eyes as becoming brighter and dimmer across different frequencies.
On the far left of the graph you can see that the sensitivity of the S cones smoothly (but quite steeply) goes to zero. This is exactly the brightness you would see if you were to tune your light source from violet to ultraviolet, i.e. you would observe smooth dimming of the light until it is gone.
Broad spectrum
It very much depends on the spectrum of the light source. If the spectrum is rather broad - e.g., a black body radiation that has a maximum that shifts towards the ultraviolet or even an ultraviolet lamp, then the source will remain violet but become pale and possibly invisible after some time (the latter scenario is less likely for a black body spectrum).
Narrow spectrum
On the other hand, if we talk about a monichromatic light source, then the transition will be more abrupt, as it will be determined not by the spectrum of the light, but by that of the photoreceptors in our eye - see How does light combine to make new colours?. Note also the related question: Is it possible that there is a color our human eye can't see?
