| dc.description.abstract | This dissertation aims to solve two related questions that carry great significance for applied researchers: how do transfer learning models perform on textual classification and frame analysis under small training sizes. Transfer learning is deemed as one of the most innovative ideas in NLP (Natural Language Processing) and has broken numerous records in miscellaneous NLP tasks. It has expedited the NLP research by saving time for model training. Transfer learning may also achieve better results than prior practices on small training sizes. However, to date, there is few thorough investigation of transfer learning’s performances on small training sizes.
This dissertation bridges the gap by conducting 2641 experiments of textual classification on performances of 6 different machine learning models across 5 diverse datasets and 8 different small training sizes utilizing different annotation schemes. Transfer learning models consistently outperform traditional machine learning (ML) models across different datasets and training sizes. Having said that, there are notable differences across Transfer Learning models. Two representative transfer learning models are used in this dissertation: BERT and XLNet. BERT model suffers a cold start problem with a larger variance in performances at moderately small training sizes (e.g. 400, 800) compared to other models. XLNet model should be our benchmark model in future practices because it achieves the best results across different training sizes and datasets with acceptable variances. A more compact annotation scheme, by collapsing categories into smaller number of groups, proves to increase model performances consistently across datasets and training sizes.
The second study suggests that transfer learning also benefits frame analysis greatly. With a compact annotation scheme and using a contextual Twitter dataset, which is unbalanced with 5 frames to classify, with a training size of 600, this research has achieved better than 72% accuracy with XLNet. This is optimistic for future research because even though each piece of text only contains the length of a normal tweet, which is significantly shorter than other sources of data, transfer learning could still achieve a satisfying level of result. This level of result could be used as a springboard for an iterative process that incorporates human relabeling to achieve more accurate results with less human labor.
This dissertation casts light on future research on textual classification and specifically frame analysis by offering guidance on model selection, performance evaluation, and annotation strategies. The visualization app (https://yikaizhao.shinyapps.io/simulation_app/) made specifically for this dissertation could be used as a reference for future related research. | en |
