Should encoder only or decoder only transformers be used for name origin identification?

Question

Like, identifying what mythology is a god from by providing it's name.

Answer 1

Since your domain specific task is essentially name summarization and classification, assuming you insist LLMs with your own sufficient labeled training data, fine-tuning encoder-only transformers such as BERT or RoBERTa are the more natural choice which are pre-trained with massive morphological & phonetic patterns capturing subword units even for a short name like suffixes and designed to learn their rich contextual embeddings to be able to associate with a target mythology label by adding a classification head.

Decoder-only transformers are optimized for generation tasks in zero-shot or few-shot prompt settings, but when you have a specific classification task with your own sufficient labeled training data, encoder-only models are usually more direct and efficient with much less computation compared to token-based generation.