All of the BERT results presented so far have used
the fine-tuning approach, where a simple classification layer is added to the pre-trained model, and
all parameters are jointly fine-tuned on a downstream task. However, the feature-based approach,
where fixed features are extracted from the pretrained model, has certain advantages. First, not
all tasks can be easily represented by a Transformer encoder architecture, and therefore require
a task-specific model architecture to be added.
Second, there are major computational benefits
to pre-compute an expensive representation of the
training data once and then run many experiments
with cheaper models on top of this representation.
