A Meta-Analysis of the Cognitive, Affective, and Interpersonal Outcomes of Flipped Classrooms in Higher Education

Abstract

This paper aims to quantify the effects of flipped classrooms in higher education by reviewing 43 empirical studies of students’ cognitive, affective, and interpersonal outcomes. The innovative pedagogy of a flipped classroom in higher education fosters a sustainable, interactive, and student-centered learning environment (as opposed to the traditional lecture style, in which there is little room for interaction). This study’s results show the positive effects of flipped classrooms and highlight the improvement in students’ educational outcomes between 2012 and 2017. Overall, effect sizes were medium—effect size (ES) = 0.35, 95% confidence interval (CI) = 0.24 to 0.47—across three outcome domains using a random effects model. In the outcomes, affective (ES = 0.59), interpersonal (ES = 0.53), and cognitive (ES = 0.24) domains were of a higher order than the effect sizes. However, the results indicated that flipped classrooms benefitted students studying chemistry, engineering, mathematics, and physics less than they did students studying other subjects.

1. Introduction

The flipped classroom is an innovative instructional model that is gaining popularity in higher education because it provides active and student-centered learning and enhances students’ educational outcomes [1]. Rahman, Mohamed, Aris, and Zaid [2] state that flipped classrooms were initially introduced in college-level technology classes. In the flipped classroom, students study instructional materials before class, typically online lectures, and apply what they learned in in-class activities [3]. Unlike teacher-centered teaching (e.g., the traditional college lecture style), flipped classrooms provide students with engaging, interactive learning experiences in which they can develop complex reasoning, written communication, and critical thinking skills [4].

The needs of students and society often evolve faster than traditional teaching methods. Thus, there is an urgent need to reconstruct college education [5]. An increasing number of stakeholders, including students and instructors, see the traditional, teacher-centered lecture style as obsolete. Consequently, universities are responding by developing, systematizing, and delivering courses and programs in new and innovative ways, which they hope will engage students as well as meet their educational needs and demands. However, transitioning from traditional lecture-based learning to a new classroom model requires a paradigm shift from teacher-centered to student-centered learning [6]. Although some scholars debate about whether the dichotomy of lectures versus active learning is meaningful in today’s higher education classrooms [7,8], this paper assumes that flipped classrooms represent a different instructional model that can complement, rather than replace, traditional approaches to education.

2. Literature Review

2.1. Defining the Flipped Classroom

The flipped classroom model does away with most teacher-centered instructional activities to create time for student-centered learning activities, such as computer-based individual instruction prior to in-class sessions [3,9]. Despite the substantial variation in the definitions of a flipped classroom [10], most flipped classrooms involve students watching instructional videos online and before class, and then participating in related interactive educational activities during the class [10,11]. Some researchers define flipped classrooms as those in which students receive computer-based instruction [12] or read materials before class [3]. Thus, in flipped classrooms, instructors perform various roles: they are curricular designers, instructors, and media developers.

A flipped classroom is also called a “backwards classroom” because teachers sometimes record lectures in advance and upload them online for students to study on their own time. This frees up class time for student-centered educational works, such as inquiry-oriented strategies, problem-based learning, quizzes, or assignments [4]. The essential point is that, within a flipped classroom, active learning activities are expanded. The reciprocal actions that occur during the class distinguish flipped classrooms from traditional models of education [13].

Flipped classrooms are generally understood through the concept of blended learning [14]. Blended learning refers to how instructors and other educational leaders integrate new technologies to adopt and develop problem-based, active learning approaches to engage students [15]. Some researchers have asserted that flipped classrooms not only enhance educational outcomes but also inspire students to excel [11]. Thus, flipped classrooms are being implemented on university campuses [16].

However, there are some barriers to the implementation of flipped classrooms in practice. Professors may be hesitant to adopt new approaches to teaching and learning [6]. Students have diverse learning needs, and some students may want instructors to provide more (or less) challenging material [6]. Other students find it difficult to keep up with the fast pace of traditional, lecture-style learning [17]. Flipped classrooms appear to provide a great solution to the major problem faced by universities: satisfying the educational needs and demands of thousands of students who study and learn at different levels.

2.2. Effects of Flipped Classroom Studies

Previous studies on flipped classrooms have found that this instructional model has positive, negative, and mixed results.

2.2.1. Positive Outcomes of Flipped Classrooms

By measuring students’ cognitive, affective, and interpersonal outcomes, a growing body of studies has found that flipped classrooms have positive effects. Cognitive outcomes are not limited to learning outcomes (e.g., learning performance and test scores); they also refer to educational outcomes in which students develop and acquire meta-cognitive abilities, such as critical thinking skills. Flipped-classroom instruction improves students’ meta-cognition and collaborative learning strategies [18], their domain knowledge and critical thinking skills [19], and their understanding of content [20]. Hsieh, Wu, and Marek [11] note that flipped classrooms enhance student learning performance and improve learning outcomes. Flipped classrooms have also been shown to spark statistical improvements in students’ vocabulary and grammar [21] as well as boost their subject matter test scores [20]. However, several studies have reported that flipped classrooms have a negative effect on students’ cognitive outcomes. Jensen, Kummer, and Godoy [16] reported that introducing a flipped classroom does not improve learning gains, scientific reasoning ability, higher level conceptual knowledge, or understanding any more than traditional learning and teaching styles.

Affective outcomes refer to educational outcomes regarding students’ satisfaction, confidence, motivation, emotions, attitudes, and feelings toward learning, the subject matter itself, or educational activities [22]. Affective outcomes have been shown to strongly influence cognitive outcomes [23]. Flipped classrooms have been shown to improve student motivation [24], student satisfaction [21,25], and confidence [21]. However, some studies have shown that flipped classrooms had a negative impact on students’ satisfaction and attitudes [16,26].

Interpersonal outcomes refer to learning that aims to improve student action and performance, including interaction and engagement (e.g., active learning). Flipped classrooms have been found to improve student–teacher interaction, student engagement, student-to-student interaction, individual education, active learning, and debate competence [6,21,27].

2.2.2. Negative Outcomes of Flipped Classrooms

Not all studies on flipped classrooms report positive results. Some report mixed or negative results. Ryan and Reid [28] demonstrated that low-achieving students in flipped classrooms performed better on exams. However, Jensen, Kummer, and Godoy [16] indicated that flipped classrooms did not improve student performance outcomes regardless of whether students were high achievers or low achievers. Missildine, Fountain, Summers, et al. [26] showed that introducing flipped classrooms improved learning gains but did not improve students’ satisfaction. Lucke [29] indicated that students enjoyed their flipped classes but showed no improvement in cognition and understanding. Vliet, Winnips, and Brouwer [18] pointed out that positive learning gains from flipped classroom environments were only temporary.

Few meta-analyses exist on the effects of flipped classrooms. Further, there is little empirical evidence regarding flipped classrooms’ utility in improving student performance in higher education [30]. This study is the first to examine the effects of flipped classrooms in higher education using a meta-analysis.

3. Research Problem

This study conducts a meta-analysis to explore the effects of flipped classrooms on cognitive, affective, and interpersonal educational outcomes. The meta-analysis synthesizes the effects of flipped classrooms in higher education and attempts to answer the following research questions: (a) what is the overall effect of the flipped classroom approach in the context of higher education? (b) What outcome variables have the most influence on measurable flipped classroom effect size? And (c) are any effects of the flipped classroom approach moderated by studies’ characteristics or variables (e.g., department, subject area, and publication year)?

4. Method

Meta-analysis involves formulating a problem, collecting data, coding data, analysis, and interpretation [31]. This study’s meta-analysis followed the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analysis) guidelines [32].

4.1. Literature Search

This paper examines journal articles and dissertations about flipped classrooms in the context of higher education that were published between 2012 and 2017. The authors searched five electronic databases for empirical articles: The Education Resources Information Center (ERIC), PROQUEST, Web of Science, PsychInfo, and Google Scholar. To capture a range of potential eligible studies, we employed the following search keywords in titles and abstracts: “flipped classroom,” “flipped class,” “flipped learning,” “inverted class,” “inverted classroom,” “smart learning,” and “blended learning.” The authors found forty-three meaningful studies that met the study’s inclusion and exclusion criteria (Figure 1).

4.2. Inclusion and Exclusion Criteria

Studies with the following features met this study’s inclusion and exclusion criteria: they must be quantitative studies on student learning or reasoning processes in flipped classrooms; they must provide sufficient information to calculate effect sizes; they must define the flipped classroom approach as including the use of video or audio materials before class and featuring in-class activities; they must compare flipped classrooms’ effects with those of traditional classrooms; they must feature students in higher education settings; they must have been published between January 2012 and June 2017; and they must be an empirical, peer-reviewed journal article or dissertation.

4.3. Coding Studies

The data were extracted from studies that met the inclusion criteria (

Table 1. Characteristics of studies.

First Author	Year	Publication	Effect Size	Data Format	Major	Subject	Outcome
Choi, Y. [33]	2016	Thesis	0.566	PP	Education	Education	A
Chun, K. [27]	2016	Journal	0.368	PP	Medical	Medicine	A
Kim, N. [34]	2014	Journal	0.220	PP	Education	Physiology	C
Lee, H. [35]	2015	Journal	0.634	PP	Business	English	C
Lee, S. [36]	2015	Journal	1.321	PP	English	English	N/A
Lim, J. [37]	2015	Journal	−0.017	PP	Multiple	English	C
Lim, J. [38]	2015	Journal	−0.353	PP	Multiple	English	C
Oh, J. [39]	2015	Journal	0.059	PP	Multiple	Multiple	C
Son, E. [40]	2015	Journal	1.666	PP	Multiple	Multiple	A
Pierce, R. [30]	2012	Journal	0.856	PP	Pharmacy	Pharmacy	C
McLaughlin, J. [4]	2014	Journal	0.270	PP	Pharmacy	Pharmacy	C
Souza, M. [41]	2015	Journal	1.098	PP	Engineering	Computer	C
Mattis, K. [42]	2015	Journal	−0.079	PP	Nursing	Math	A
Sletten, S. [43]	2015	Journal	0.382	PP	Multiple	Multiple	C
Mortensen, C. [44]	2015	Journal	0.459	PP	Animal Science	Science	C
Kang, N. [21]	2015	Journal	0.196	SMCD	Multiple	English	C
Kim, E. [45]	2016	Thesis	0.641	SMCD	Education	Education	A
Park, W. [46]	2016	Journal	0.261	SMCD	Multiple	Multiple	AC
Sun, J. [47]	2017	Journal	0.128	SMCD	Multiple	Physics	A
Hsieh, J. [11]	2016	Journal	0.709	SMCD	English	English	C
Davies, R. [48]	2013	Journal	0.229	SMCD	Management	Computer	C
Brooks, A. [49]	2014	Journal	0.225	SMCD	Multiple	English	C
Sengel, E. [50]	2014	Journal	0.145	SMCD	Education	Physics	C
Kim, Y. [51]	2015	Journal	0.780	SMCD	Multiple	English	A
Suh, M. [52]	2016	Journal	0.467	SMCD	Education	Education	AC
Overmyer, G. [53]	2014	Dissertation	0.192	SMCD	Multiple	Math	C
Winter, J. [54]	2013	Dissertation	0.242	SMCD	Multiple	Physics	C
Lee, S. [55]	2016	Thesis	0.953	SMCD	Education	Education	A
Jensen, J. [16]	2015	Journal	0.076	SMD	Multiple	Science	C
McLaughlin, J. [4]	2014	Journal	0.270	SMD	Pharmacy	Pharmacy	ACI
Mason, G. [5]	2013	Journal	0.033	SMD	Engineering	Engineering	C
Souza, M. [41]	2015	Journal	1.098	SMD	Engineering	Computer	C
Missildine, K. [26]	2013	Journal	0.436	SMD	Nursing	Nursing	C
Ryan, M. [28]	2016	Journal	0.024	SMD	Multiple	Chemistry	C
Talley, C. [56]	2013	Journal	0.447	SMD	Psychology	Psychology	C
Albert, M. [57]	2014	Journal	0.168	SMD	Business	Management	C
Al-Zahrani, A. [58]	2015	Journal	0.484	SMD	Education	Education	AC
Mason, G. [5]	2013	Journal	0.033	SMD	Engineering	Engineering	ACI
Osman, S. [59]	2014	Journal	−0.933	SMD	Business	Business	C
Sahin, A. [60]	2015	Journal	0.509	SMD	Engineering	Engineering	C
Leicht, R. [61]	2012	Journal	0.080	SMD	Engineering	Engineering	C
Winquist, J. [62]	2014	Journal	0.427	SMD	Psychology	Math	C
Kim, Y. [51]	2015	Journal	0.780	SMD	Multiple	English	AC
Suh, M. [52]	2016	Journal	0.467	SMD	Education	Education	C
Mattis, K. [42]	2015	Journal	−0.079	SMD	Nursing	Math	C
Flynn, A. [63]	2015	Journal	0.107	SMD	Chemistry	Chemistry	C
Prashar, A. [64]	2015	Journal	0.165	SMD	Management	Management	C
Fraga, L. [65]	2014	Journal	0.360	SMD	Education	English	C
Overmyer, G. [53]	2014	Dissertation	0.192	SMD	Multiple	Math	C
Winter, J. [54]	2013	Dissertation	0.242	SMD	Multiple	Physics	C
Mortensen, C. [44]	2015	Journal	0.459	SMD	Animal Science	Science	C
ALRowais, A. [66]	2014	Journal	0.405	PP	Education	Education	C

Note: SMES = standardized mean difference; PP = pre–post; SMCES = standardized mean change difference; A = affective, C = cognitive, I = interpersonal.

). The studies’ characteristics were coded as possible moderating variables to investigate the variance of flipped classrooms’ effects. Two researchers independently coded each study. We developed a coding manual to maintain reliability of the coding procedures, which included study characteristics, effect size calculation, and report characteristics. Discrepancies between the two coders were resolved prior to data analysis without exception and were resolved by an independent third expert if no agreement could be reached between the two coders.

4.4. Computation of Effect Sizes

The effect size of this meta-analysis includes three different data formats: treatment vs. control group design, pre–post design, and standardized mean change difference (pre–post measure with both treatment and control group), where the pooled estimate of standard deviation was used to consider different sample sizes between flipped and non-flipped classroom groups. All effect sizes were calculated using the Comprehensive Meta-Analysis (CMA) program to estimate a mean effect size [67]. Effect sizes were reported as positive when flipped classroom students performed better than students in the control groups. The effect size was evaluated as follows: 0.20 = small effect, 0.50 = medium effect, and 0.80 = large effect [68].

4.5. Analysis

4.5.1. Combining Effect Sizes

We employed a two-step process to synthesize the effects of flipped classroom outcomes. First, it calculated the effect size and variance of each outcome in the primary study. Second, it calculated the weighted mean effect size (ES) using inverse variance weight. To select its analysis model, the study conducted a homogeneity test using two measures of variability: Q and I². The Q test examined whether the variability in an average weighted ES exceeds sampling error alone [69]. I² is an alternative measure of homogeneity, which is less sensitive to sample size than Q. I² shows whether the proportion of the observed variance reflects differences in true effect sizes [67]. To evaluate I² statistics, this study followed Higgins and Green’s [70] guidelines: 0% to 40% might not be important; 30% to 60% may represent moderate heterogeneity; 50% to 90% may represent substantial heterogeneity; and 75% to 100% may represent considerable heterogeneity. The null hypothesis of the homogeneity test was that all outcomes came from the same population. If homogeneous, this study used a fixed effects model that had a common effect size and only considered sampling variance. If heterogeneous, this study used a random effects model that had no common effect size and considered sampling variance and true difference between studies [71]. Based on the homogeneity test and investigation of flipped classroom primary studies, this study used random effect models to synthesize the main effects and sub-group analyses.

4.5.2. Publication Bias

Publication bias happens when the results of published studies are different from the results of unpublished studies because studies with positive results, large effects, and large sample sizes are overrepresented in the literature [67,72]. To examine publication bias, this study adopts a funnel plot, exploring symmetrical distributions around the weighted mean effect sizes [73]. Funnel plots are scatter plots of effect sizes from studies in the meta-analysis, where the horizontal axis represents effect sizes and the vertical axis represents standard errors [72]. An asymmetrical pattern in the results of the funnel plot indicates a possible publication bias.

4.5.3. Analyzing Variances in Effect Sizes Across Studies

Finally, this study examined the variances in the effect sizes using sub-group analysis and meta-regression [74]. Meta-analyzers should prove whether the effect sizes are homogenous in order to calculate the overall effect size in a meta-analysis. This study used homogeneity test results to select an analysis model and decide whether reviewers would perform a sub-group analysis. Q-statistics were used to assess the heterogeneous structure of the average effect sizes. When the Q statistic is significant (p < 0.05), it suggests that the studies in the meta-analysis are heterogeneous effects. A random effects model was adopted to calculate the overall effect size in this study. The homogeneity calculation formula is as follows:

$$Q={\displaystyle \sum }_{i=1}^k\frac{{(g_i-g.)}^2}{v\left(g_i\right)}={\displaystyle \sum }{w}_i{(g_i-g.)}^2$$

(1)

where $w_i=1/v\left(g_i\right)$ and w_i is an inverse variance weight. The Q statistic is used to determine whether the primary results are homogeneous for subgroup analysis. The magnitude of effect sizes interpreted 0.2 as small, 0.5 as medium, and 0.8 as large according to Cohen’s rule of thumb [68].

4.5.4. Dependence

This meta-analysis included a total of 43 studies and 218 effect sizes. When a primary study has more than one effect size, reviewers should explain the assumption of independence because multiple effect sizes have dependence within the study. To maintain the assumption of independence, the reviewers should select only one effect size per study, which will cause information loss. To keep multiple effect sizes within the study, this choice will cause a violation of independence assumption. To avoid this violation, this study adopted the “shifting unit of analysis” method [75]. This method proposes a compromise between the issues of information loss and violation of independence assumptions. To calculate the overall effect size, “study” will be used as an analysis unit to determine the independence assumption. To perform sub-group analysis, the effect size of each sub-group will be used as a unit of analysis.

5. Results

As mentioned earlier, the 43 studies included in the meta-analysis synthesized a total of 218 effect sizes: an average of 5.1 effect sizes per study. As multiple effect sizes existed within studies, the reviewers considered the dependence of effect sizes in each study. Figure 2 shows the study characteristics for all 43 studies, including effect size (i.e., standard difference in means), standard error, variance, confidence interval, Z-value, and p-value in a forest plot. Black squares in the forest plot’s horizontal lines show the effect size of an individual study, and the horizontal lines indicate the confidence interval for each estimate. The small diamond shape at the bottom represents the overall effect size of all studies. According to the forest plot, the smallest effect size value is −0.933, and the highest effect size value is 1.666. Thirty-nine studies had positive effect sizes, while four had negative effect sizes. Consequently, the implementation of flipped classrooms had a significant effect in 39 of the 43 studies.

5.1. Overall Effect of Flipped Classroom (Research Question 1)

Table 2. Results of the homogeneity test.

N	Q	p-Value	−95% CI	ES	+95% CI	SE
43	1022.9	<0.000	0.30	0.32	0.34	0.011

Note: N = number of studies; Q = homogeneity statistics; ES = effect size; SE = standard error; CI = confidence interval.

provides the following statistics: number of studies; Q statistic assessing the heterogeneity of the distributions of the effect sizes; statistical significance of ES; and confidence interval (CI) for each ES, ES, and standard error (SE). The effect sizes were heterogeneous (Q [42] = 1022.9, p < 0.05), I² = 96.0, among the 43 studies. Thus, all studies in the analysis did not share a common effect size, which means the null hypothesis of the homogeneity test can be rejected. We used the random effects model to estimate the overall effect size and compare sub-group differences using the study characteristics (e.g., outcome variables, report characteristics variables, and study characteristics variables). The results of the homogeneity test show that the effect sizes are heterogeneous (Table 2).

The results of the random effects model analysis are displayed in

Table 3. Overall result of meta-analysis.

N	−95% CI	ES	+95% CI	SE
43	0.24	0.35	0.47	0.060

Note: N = number of studies; ES = effect size; CI = confidence interval; SE = standard error.

. The overall effect size of flipped classrooms was 0.35, indicating that flipped classrooms had a medium effect in terms of the Cohen’s rule of thumb [68]. The effect size showed an overall significant difference in outcomes from flipped classrooms and traditional lecture-based classrooms in higher education (ES = 0.35, 95% CI = 0.24 to 0.47).

5.2. Outcomes of Flipped Classroom (Research Question 2)

This meta-analysis used a random effects model to investigate the differences between sub-groups, as the results from each sub-group were heterogeneous. The categorical variables are as follows: outcome domains (cognitive, affective, and interpersonal), department, subject, data format, and publication status. We conducted a meta-regression analysis using publication year as a covariate. In the random effects categorical analysis by outcome, shown in

Table 4. Effect size by outcome.

Outcomes	k	−95% CI	ES	+95% CI	SE
Affective Outcomes
Attitude	11	0.52	0.64	0.75	0.058
Confidence	7	0.13	0.25	0.36	0.059
Immersion	3	1.29	1.52	1.76	0.122
Impression	5	0.08	0.28	0.48	0.28
Independence	2	0.33	0.57	0.81	0.123
Motivation	11	0.65	0.76	0.88	0.057
Subtotal	39	0.53	0.59	0.65	0.030
Cognitive Outcomes
Achievement	8	0.44	0.59	0.75	0.080
Application	8	0.16	0.25	0.34	0.046
Analysis	3	0.28	0.46	0.63	0.092
Evaluation	1	−1.46	−0.93	−0.40	0.270
Knowledge	109	0.12	0.15	0.18	0.017
Metacognition	16	0.17	0.26	0.35	0.046
Synthesis	4	0.26	0.49	0.71	0.115
Understanding	4	0.49	0.60	0.71	0.057
Subtotal	153	0.19	0.24	0.36	0.014
Interpersonal Outcomes
Interaction	3	0.41	0.54	0.67	0.066
Participation	10	0.54	0.63	0.73	0.048
Response	5	0.18	0.32	0.45	0.068
Subtotal	18	0.47	0.53	0.59	0.031

Note: k = number of effect size; CI = confidence interval; ES = effect size; SE = standard error.

, the results of implementing flipped classrooms varied. In the outcomes, the respective effect sizes of affective (ES = 0.59), interpersonal (ES = 0.53), and cognitive (ES = 0.24) domains were in descending order.

In the context of higher education, flipped classrooms appear to have more significant effects on students’ affective and interpersonal outcomes than on their cognitive outcomes. Regarding affective outcomes, students’ immersion (ES = 1.52), motivation (ES = 0.76), attitude (ES = 0.64), independence (ES = 0.57), impression (ES = 0.28), and confidence (ES = 0.25) were in descending order of effect sizes. Regarding interpersonal outcomes, students’ participation (ES = 0.63), interaction (ES = 0.54), and response (ES = 0.32) were in descending order of effect sizes. In the cognitive domain, understanding (ES = 0.60), achievement (ES = 0.59), synthesis (ES = 0.49), analysis (ES = 0.46), meta-cognition (ES = 0.26), application (ES = 0.25), knowledge (ES = 0.15), and evaluation (ES = −0.93) were in descending order of effect sizes.

5.3. Effects of Characteristics (Research Question 3)

Table 5. Effect sizes by subgroup related to the department.

Subgroup	Categories	k	−95% CI	ES	+95% CI	SE
Department	Animal Science	4	0.35	0.46	0.57	0.055
	Business	6	0.12	0.19	0.26	0.038
	Chemistry	1	−0.01	0.11	0.23	0.062
	Education	37	0.51	0.58	0.64	0.032
	Engineering	38	0.33	0.42	0.50	0.044
	English	2	0.46	0.83	1.20	0.189
	Management	15	0.11	0.25	0.39	0.072
	Medical	9	0.27	0.37	0.47	0.052
	Nursing	7	−0.01	0.13	0.28	0.074
	Pharmaceutical Med.	17	0.24	0.29	0.34	0.025
	Psychology	2	0.15	0.44	0.72	0.147
	Mixed	71	0.17	0.21	0.25	0.021

Note: k = number of effect size; CI = confidence interval; ES = effect size; SE = standard error.

and

Table 6. Effect sizes by subgroup related to the subject area.

Subgroup	Categories	k	−95% CI	ES	+95% CI	SE
Subject	Business	1	−1.46	−0.93	−0.41	0.270
	Chemistry	19	0.02	0.10	0.18	0.039
	Computer Science	7	0.80	0.96	1.11	0.077
	Education	26	0.72	0.80	0.88	0.041
	Engineering	32	0.07	0.17	0.27	0.051
	English	34	0.04	0.12	0.20	0.041
	Humanities	2	1.75	1.95	2.14	0.098
	Management	17	0.12	0.19	0.26	0.036
	Mathematics	13	0.03	0.13	0.23	0.051
	Medicine	9	0.27	0.37	0.47	0.052
	Nursing	1	0.21	0.44	0.66	0.116
	Pharmaceutical Med.	17	0.24	0.29	0.34	0.025
	Physics	9	0.04	0.15	0.25	0.054
	Physiology	9	0.11	0.22	0.33	0.054
	Psychology	1	0.00	0.45	0.89	0.226
	Science	5	0.33	0.43	0.54	0.053
	Mixed	9	0.37	0.44	0.51	0.037

Note: k = number of effect size; CI = confidence interval; ES = effect size; SE = standard error.

list the effect sizes measured by this study, separated by department and subject area. The highest proportion of effect sizes (k = 71) applies to students in mixed departments. The effect sizes by department were English, ES = 0.83, Education, ES = 0.58, Animal Science, ES = 0.46, Psychology, ES = 0.44, Medical, ES = 0.37, Pharmaceutical Medicine, ES = 0.29, Management, ES = 0.25, Mixed, ES = 0.21, Business, ES = 0.19, Nursing, ES = 0.13, and Chemistry, ES = 0.11.

This study investigated a variety of subject areas to determine whether the flipped classroom approach is more beneficial in some contexts or subjects than it is in others. The effect sizes by subject were Computer Science, ES = 0.96, Education, ES = 0.80, Psychology, ES = 0.45, Nursing, ES = 0.44, Mixed subject, ES = 0.44, Science, ES = 0.43, Medicine, ES = 0.37, Pharmaceutical Medicine, ES = 0.29, Physiology, ES = 0.22, Management, ES = 0.19, Engineering, ES = 0.17, Physics, ES = 0.15, Math, ES = 0.13, English, ES = 0.12, Chemistry, ES = 0.10, and Business, ES = −0.93. However, the effect sizes of Business, Nursing, and Psychology require careful interpretation because they showed one effect size (k = 1) per subject.

In the primary studies reviewed in this research, the data are generally represented in three different formats: pre–post design, treatment vs. control group design, and pre–post with treatment vs. control group (standardized mean change difference). The effect sizes for each type are as follows: treatment vs. control, ES = 0.25 (95% CI = 0.21 to 0.28), pre–post design, ES = 0.38 (95% CI = 0.35 to 0.42), and standardized mean change difference, ES = 0.47 (95% CI = 0.41 to 0.53). The difference was not small, and study design may factor into this difference in effect sizes. Regarding publication type, the effect size of dissertations (ES = 0.61, 95% CI = 0.54 to 0.68) was larger than the effect size of journal articles (ES = 0.29, 95% CI = 0.26 to 0.31), but the difference was not significant (

Table 7. Effect sizes by subgroup: data format and publication type.

Sub-Group	Categories	k	−95% CI	ES	+95% CI	SE
Data Format	Pre–post	67	0.35	0.38	0.42	0.018
	Treatment vs. control	106	0.21	0.25	0.28	0.017
	Mean change difference	45	0.41	0.47	0.53	0.029
Publication Type	Dissertation	29	0.54	0.61	0.68	0.036
	Journal	188	0.26	0.29	0.31	0.012

Note: k = number of effect size; CI = confidence interval; ES = effect size; SE = standard error.

).

Regarding year of publication, this study conducted a meta-regression analysis in which the regressing effect sizes of flipped classrooms on year of publication served as a moderator. The slope of the meta-regression by publication year is negative overall, but it is statistically significant (

Table 8. Results of the random-effects regression analysis by publication year.

Standard Parameter	Estimate	Error	z-Value	p-Value
Intercept	1021.4	141.8	7.20	0.001
Publication year	−0.51	0.070	−7.20	0.001

) and has a significant moderating effect on the relationship between flipped classrooms and a study’s year of publication.

5.4. Publication Bias

The funnel plot (Figure 3) shows the symmetry of effect size distribution in the mean effect size whether publication bias in the overall effect size exists, providing no evidence for publication bias. This meta-analysis shows no missing studies and finds no imputations of effect size for publication bias.

6. Discussion

This study conducted a meta-analysis of the effects of flipped classrooms on students’ cognitive, affective, and interpersonal outcomes in higher education. It extends the discussions and findings from recent meta-analyses that found that flipped classrooms had a significant effect on students’ cognitive outcomes in higher education: for example, by improving their test scores, grade, knowledge, skills, and self-directed learning (e.g., [9,76,77]). This study expands the evidence for flipped classroom effectiveness in improving college students’ academic outcomes as compared to traditional, lecture-based classrooms.

The first research question was regarding the overall effect of flipped classrooms on students’ cognitive, affective, and interpersonal outcomes. The study found that flipped classrooms had a medium effect on academic outcomes; the average scores of students in flipped classrooms were 0.35 standard deviations above the average scores of students in traditional, lecture-based classrooms. It also confirmed the results of previous, related studies (e.g., ES = 0.36 [3]; ES = 0.35 [9]; ES = 0.53 [77]; ES = 0.21 [78]). In short, its findings demonstrate that flipped classrooms can improve college students’ academic outcomes in various ways, could provide an effective way to inculcate essential 21st-century skills in students [79], and may assist students with special educational needs in performing better than they would in traditional, lecture-based classrooms.

The second research question was regarding the outcomes influenced by the introduction of the flipped classroom method. The overall effect sizes of the affective outcomes (ES = 0.59, SE = 0.03, 95% CI = 0.53 to 0.65]), interpersonal outcomes (ES = 0.53, SE = 0.31, CI = 0.47 to 0.59), and cognitive outcomes (ES = 0.24, SE = 0.24, 95% CI = 0.19 to 0.36) were the descending order of the overall effect sizes. This study’s results suggest that flipped classrooms improve college students’ cognitive, affective, and interpersonal outcomes and that flipped classrooms have more significant effects on affective and interpersonal outcomes than on cognitive outcomes. This result can be explained by the features of the flipped classroom that encourage active engagement and learner-centered interactions. Furthermore, this study’s findings indicate that flipped classrooms indirectly affect cognitive outcomes because affective outcomes have a strong influence on cognitive outcomes [23], in part by improving students’ motivation and willingness to learn [80]. However, affective outcomes (e.g., attitudes and satisfaction) in the flipped classroom are not necessarily positive in higher education.

This study’s results regarding the high effect sizes of interpersonal outcomes in flipped classrooms are consistent with the results of Shi, Ma, Macleod, et al. [77]. Further, the results can be explained by the instructors’ tendency to design active in-class activities in flipped classrooms to increase student participation and interaction [61] through discussion, small group activities, feedback, group discussion, collaborative group work, and group projects [81]. These active, in-class activities enhance students’ interpersonal skills and encourage them to become active and self-directed learners who are deeply involved in the learning process [82,83].

This study’s third research question addressed the effects of study characteristics on how the effect sizes of flipped classrooms were measured. To answer this question, the study performed subgroup analyses using subject area, department, publication year, and study design as moderators. These moderators accounted for a small amount of the relatively large levels of heterogeneity between studies. The results indicated that flipped classrooms can be applied in a variety of subject areas and still effectively improve educational outcomes, as discussed in Rahman, Mohamed, Aris, and Zaid [2]. Although instructors’ individual approaches can influence the success of flipped classrooms, this study found that English, Engineering, Math, Physics, and Chemistry classrooms showed small effect sizes. These results are in line with other meta-analyses of flipped classrooms (e.g., [3,78]).

Regarding publication bias and publication type, this study found that the primary literature on flipped classrooms did not indicate publication bias, even though dissertations (ES = 0.61) had a greater effect size than journal articles (ES = 0.29). This study also performed a funnel plot to examine the possibility of publication bias but did not find evidence for publication bias. Thus, publication type can be treated as a moderator in future flipped classroom interventions.

7. Limitations and Future Research Directions

This meta-analysis has several limitations. First, the meta-analysis gains ecological validity by including only quantitative field studies (experimental or quasi-experimental research), which examine whether the study results can be generalized to real-life settings. However, some internal validity relative to more controlled laboratory studies is sacrificed: for example, randomized controlled trials [84]. Second, this meta-analysis includes only quantitative findings despite the fact that there are many flipped classroom studies that employ qualitative research methods [31,85]. Because this study excluded qualitative studies from its analysis, its results should be interpreted with caution. Qualitative findings help researchers arrive at deeper understandings [86] and generate new knowledge [87]. Some studies show that flipped classrooms have been particularly effective among the learner demographic [28] because low achievers require more interaction and motivation to attain good learning outcomes. We recommend and encourage researchers to implement flipped classrooms with various student bodies in a variety of academic settings to better define the degree to which these results are transferrable [16].

The flipped classroom is not a panacea, and its effectiveness depends in large part on whether students actually use the available pre-class time effectively [30]. We therefore propose repeated use of flipped classrooms and related, modified strategies on a trial-and-error basis. Ratta [6] insisted that flipped classroom instruction is congruent with today’s digital-savvy college student; moreover, it is also important to understand the various influences of today’s student culture, study style, study habits, and use of devices. Further study may be warranted to allow more detailed conclusions about student performance to be drawn [88].

8. Conclusions

This study synthesized the results of 43 studies regarding the effects of flipped classrooms on students’ cognitive, affective, and interpersonal outcomes in higher education. It examined the overall effect sizes of flipped classrooms compared to traditional, lecture-based classrooms and found that flipped classrooms had a medium effect on various student learning outcomes. Particularly, the study identified that the flipped classroom shows a more significant effect on affective and interpersonal outcomes than on cognitive outcomes. This result can be explained by the features of the flipped classroom that encourage active engagement and learner-centered interactions. Instructors and other educational leaders in higher education institutions can pursue instruction redesigns and educational supports to implement flipped classrooms as an effective pedagogical practice. Additionally, the mixed results of adopting the flipped classroom instruction in departments and subjects show that various instructional forms and strategies are factors that determine the effectiveness on educational outcomes. Thus, future research must explore the relationship between various forms of flipped classrooms and educational outcomes to arrive at pedagogical decisions for instructional development.