Unused memory is wasted memory; so Ubuntu (generally Linux) tries to use it in the best way it can.
As I know, whenever you open a program, after closing it a lot of data related to its process still lives in memory in case that you run it again, so Linux doesn't have to load all that data again into your memory because it's already there.
As long as you have a lot of data in Buff/Cache you are safe to go, because actually that part will be freed immediately in case of need.
You can use vmstat -S M to see the buffer and cache separately. first see which one is having a higher value, then it's easier to guess what is happening.
Both bellow quotes are from here:
"Buffers" represent how much portion of RAM is dedicated to cache disk block.
"Cached" is similar like "Buffers", only this time it caches pages from file reading.
Which is quoted itself from here. And a more clear quote is:
Buffers are associated with a specific block device, and cover caching of filesystem metadata as well as tracking in-flight pages. The cache only contains parked file data.
The buffers remember what's in directories, what file permissions are, and keep track of what memory is being written from or read to for a particular block device. The cache only contains the contents of the files themselves.
Read more
In your case, I guess a large part of this 24G is related to cache, it may be caused by a specific service running on your system.