Is large language model and foundation model the same thing?

Question

I read a lot about foundation model and large language model.

However, I dont find a clear definition what exactly is a foundation model. Is large language model and foundation model the same thing?

Answer 1

At this point in time, there does not appear to be a really widely-agreed-upon definition of "Foundation models". If you want one, the best place to go would be this paper from Stanford that coined the term. Generally at least some of the following ideas apply (according to some people they must all apply, according to others only some need to apply):

• Trained using unsupervised or self-supervised learning.
• Large (deep neural network) model.
• Not intended to be used directly for any particular end-goal.
• Intended to serve as a basis ("foundation"). You can think of this as "warm-starting", starting from a well-trained initial model, that you can then fine-tune with further training (for example, supervised learning) for any specific task you personally have in mind.
• Trained on multimodal data (not just text, not just images, not just audio, etc., but a mix of things). In my personal opinion this point really is not strictly necessary, and I think most people would agree, but I've seen some people saying this.

Large Language Models would typically be trained specifically on language-related data (text). So, I suppose an LLM could sometimes serve as a Foundation model, but it's not necessarily the same thing.

NOTE: I would like to remark that, from many experts outside of Stanford, there is quite a bit of pushback against the particular term "Foundation model". There seems to be a bit of PR involved, where it seems like they may like to coin a new term for something that really doesn't need a new term. Just saying it's, for example, a large pre-trained model, may be more clear.