机器学习技术能有效地分析定性焦点小组数据吗?

挑战

比以往任何时候都多, 研究人员可以接触到各种各样的广泛信息, 丰富的, 来自多个来源的教育相关文本数据, 如文献数据库(e.g., 教育资源资讯中心, 在线课程/调查的开放式回答, 网上论坛, 面对面课程或焦点小组的转录音频, 数字的文章, 还有社交媒体.

Such advances in data availability (coupled with emerging analytic techniques) can dramatically increase the possibilities for discovering new patterns, 研究效率, 以及在教育环境中检验新理论. 例如, 新兴的分析技术能否基于机器学习, 比如主题建模, 有效地分析定性焦点小组数据, 以及可以确定哪些限制和/或建议?

解决方案

We provided support to Worcester Polytechnic Institute by extending the use of topic modeling on data collected from focus groups with teachers who have implemented a number of math technology interventions with their students: From Here to There (FH2T), a game-based perceptual learning intervention; DragonBox 12+ (DragonBox), a widely used game-based technology application; and 2 versions of ASSISTments (Immediate Feedback and Active Control).

我们比较了定性编码和主题建模的结果. 与文本挖掘中通常使用的文本数据不同, 如上所述, 这项研究中的焦点小组涉及动态的人类交流, 老师们在学习的过程中以一种有意义的方式分享思想和经验, 处理, 并对主持人和其他参与者做出回应. 在这个过程中, 演员(在这种情况下), teachers) constantly take turns (and listen simultaneously) in roles as speaker and listener (Watzlawick et al., 1967*; DeVito, 2016*). 像这样, 交流的模式在不断演变, and the directions and depth of information exchanged during the focus group can drastically differ depending on the group of teachers and the skills of the facilitator.

This investigation used information collected as part of a larger randomized controlled trial (RCT) conducted in partnership with Worcester Polytechnic Institute, 缅因大学, 和印第安纳大学在COVID-19大流行期间.

更大的研究是在10所中学进行的随机对照试验(9所面对面，1所虚拟)。, 包括3,600+ 7年级学生. It tested the impacts of 3 educational technology interventions on algebraic understanding among 7th-graders, (a) FH2T, (b) DragonBox, (c)即时反馈, (d)主动控制. FH2T和DragonBox条件代表基于游戏的应用程序的使用. 即时反馈包括使用在线作业系统ASSISTments的问题集. 为了本研究的目的, the Active Control condition mimics traditional homework assignments while still using technology.

作为更大研究的一部分, teachers participated in focus groups to discuss (a) teachers’ perspectives of students’ reactions to the mathematics technologies, (b) challenges teachers and students encountered while using the mathematics technologies, (三)各种应用对学生学习的影响, (d)疫情对学生教学的独特影响, (五)对数学技术的改进建议. 在34名参与研究的教师中, 16个国家(47%)参加了4次焦点小组会议中的1次.

在这个调查中, we examined if topic modeling could extract patterns that were consistent with more qualitative analysis approaches from teacher focus group data. 我们在本研究中探讨的具体问题是:

• 定性编码方法和主题建模方法产生了哪些主题?
• 关于焦点小组数据的分析，主题建模有什么局限性?
• What recommendations do we have for other researchers who may attempt to use the topic modeling on data collected from focus groups?

Unlike other forms of text data such as literature, books, or essays where a type of information (e.g., 演员信息, 关于数据源的信息, 关于发现的信息被组织在一个部分中, 焦点小组产生的数据反映了动态的人际交流(1).e., 参与者以有意义的方式分享想法和经验, 处理, 并回应主持人和其他参与者). The facilitator can also steer the participants back to the focus group questions or go along with the direction of the focus group discussions, 取决于所提出的研究问题.

结果

The topic modeling results showed a high degree of agreement with the qualitative coding results. The biggest difference between the topic modeling and qualitative coding was the organization/classification of themes at the higher level (e.g.， 5个主题vs. 3主题). 在较低的层次, 两种方法都确定了许多相同的子主题和, 因此, the results tell similar stories about the students’ reactions to working with the different mathematics technologies. In examining the coding and narrative findings from the 2 different methods (topic modeling and qualitative coding), the 2 researchers found that the topic modeling method was less effective in capturing the nuanced information that the qualitative coding was able to identify. 例如, the qualitative coding was better able to provide details about students’ reactions specific to each of the learning tools and, 因此, 能够提供比主题建模更细微的发现吗. 类似的, the qualitative coding was better able to identify differences and information specific to student populations (i.e., students in special education and accelerated students) than the topic modeling methodology.

This study successfully extended the application of topic modeling on data collected from focus groups. 一起使用, study results demonstrated that topic modeling is a viable method for coding focus group data in a variety of ways. 它可以很容易地在定性分析之前使用，以确定节点(如.e., themes) or in parallel or after qualitative analysis to identify strengths and weaknesses that might be inherent in the qualitative analysis. Of benefit to the qualitative coder is the rapid nature of technology that allows for faster coding using topic modeling.

*注:

DeVito J.A. (2016). 人际沟通书 (14日艾德.). 英国伦敦:培生教育有限公司.

罗瓦兹拉威克,P.巴维拉斯，J.B.和杰克逊，D.D. (1967). Pragmatics of human communication: A study of interactional patterns, pathologies, and paradoxes. 纽约，纽约州:W.W. 诺顿 & 公司.