Which of the following statements is correct about pre-training and fine-tuning?

Question

BlackTom AI · Accepted Answer

Question restatement: Which of the following statements is correct about pre-training and fine-tuning?

Option 1: 'Pre-training is an unsupervised approach.' This is generally accurate for many large language models, where pre-training often uses unsupervised or self-supervised objectives (e.g., predicting missing tokens). The key idea is learning broad representations from unlabeled data, so describing pre-training as unsupervised captures the typical method, though some setups may include weak supervision or auxiliary signals. Overall, this statement aligns with common practice.

Option 2: 'Fine-tuning can use both supervised and reinforcement learning approaches.' This is correct in practice. Fine-tuning can be done with supervised learning using labeled data, and can also involve reinforcement learning techniques (e.g., reinforcement learning from human feedback, RLHF) or other RL-based objectives to optimize behavior beyond strict labeled data constraints. This option reflects the flexible toolkit used in fine-tuning.

Option 3: 'GPT is a pre-trained encoder.' This is misleading. GPT models are typically described as decoder-only architectures (transformer decoders) rather than encoders. They are pre-trained with autoregressive objectives, predicting the next token, which distinguishes them from encoder-based architectures or encoder-decoder models. Therefore, this statement is inaccurate for the typical GPT family.

Option 4: 'Both A and B.' This statement asserts that both the first and second statements are true. Given the explanations above, Option 1 is a reasonable description of pre-training as unsupervised, and Option 2 correctly describes the potential for fine-tuning to use supervised or reinforcement learning approaches. Therefore, combining these two accurate claims is true, making this option a correct synthesis of the previous points.

Summary: Option 1 is largely correct in describing pre-training as unsupervised. Option 2 accurately states the possible methods for fine-tuning. Option 3 contains a factual inaccuracy about GPT being an encoder. Option 4 correctly combines the true aspects of A and B, reflecting the standard understanding of pre-training and fine-tuning practices. The reasoning shows why each option stands as it does, highlighting the conceptual distinctions and common practices in modern NLP model training.

BU.330.760.52.SP25 Final- Requires Respondus LockDown Browser

Which of the following statements is correct about pre-training and fine-tuning?

查看解析

登录即可查看完整答案

类似问题

Transfer learning is invariably effective. eg. Irrespective of the amount of data, we can always rely on transfer learning.

When attempting to transfer learn for an image captioning task, we must use a source dataset for image captioning or visual question answering, since image classification by itself is not similar enough

Which of the following statements about transfer learning in CNNs are correct? (mark all that apply)

What is the key difference between pre-training and fine-tuning stages in transformer model development? Hint: Lec 19, Slide 50.

In a consumer society, many adults channel creativity into buying things

Economic stress and unpredictable times have resulted in a booming industry for self-help products

更多留学生实用工具

考试浏览器助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单