Methodological Reform in Quantitative Second Language Research: Effect Sizes, Bayesian Hypothesis Testing, and Bayesian Estimation of Effect Sizes

Abstract

This dissertation consists of three manuscripts. The manuscripts contribute to a budding “methodological reform” currently taking place in quantitative second-language (L2) research. In the first manuscript, the researcher describes an empirical investigation on the application of two well-known effect size estimators, eta-squared (η2) and partial eta-squared (ηp2), from the previously published literature (2005 - 2015) in four premier L2 journals. These two effect size estimators express the amount of variance accounted for by one or more independent variables. However, despite their widespread reporting, often in conjunction with ANOVAs, these estimators are rarely accompanied by much in the way of interpretation. The study shows that ηp2 values are frequently being misreported as representing η2. The researcher interprets and discusses potential consequences related to the long-standing confusion surrounding these related but distinct estimators. In the second manuscript, the researcher discusses a Bayesian alternative to p-values in t-test designs known a “Bayes Factor”. This approach responds to pointed calls questioning why null hypothesis testing is still the go-to analytic approach in L2 research. Adopting an open-science framework, the researcher (a) re-analyzes the empirical findings of 418 L2 t-tests using the Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts. The results show considerable differences arising in the rejections of the null hypothesis in certain cases of previously published literature. The study provides field-wide recommendations for improved use of null hypothesis testing, and introduces a free, online software package developed to promote Bayesian hypothesis testing in the field.

In the third manuscript, the researcher provides an applied, non-technical rationale for using Bayesian estimation in L2 research. Specifically, the researcher takes three steps to achieve my goal. First, the researcher compares the conceptual underpinning of the Bayesian and the Frequentist methods. Second, using real as well as carefully simulated data, the researcher introduces and applies a Bayesian method to the estimation of standardized mean difference effect size (i.e., Cohen’s d) from t-test designs. Third, to promote the use of Bayesian estimation of Cohen’s d effect size in L2 research, the researcher introduces a free, web-accessed, point-and-click software package as well as a suite of highly flexible R functions.