Research On Problem-Solving Competence Assessment Based On The Learning Analytics

Twenty-first century is commonly regarded as a knowledge economic century. Knowledge work, global market, remote connection, and mutually influenced culture reorganize our connected world. When the world becomes more and more complicated and connected, the changing society also requires higher qualified professionals with high-level cognitive skills to adapt to the fast-changing environment and its complexity. Complex problem solving is one of the core competences and draws widespread attention all over the world. Various countries and organizations, such as UNESCO, EU, Australia, USA, have released related policy to promote teaching and assessment on problem-solving competence in K-12 education, which, in turn, helps youths gain advantage in future global competition. Because problem-solving process is implicit cognitive activities, the traditional "black box" testing can only offer summative assessment results of problem-solving performance. In terms of knowledge, skills, abilities, behavior, and other cognitive and/or non-cognitive factors, we are lack of knowledge about how this process actually occurs or why the underlying mechanism leads to that kind of performance. Researchers face this challenge in understanding complex problem solving and further assess one’s problem-solving competence.With digital revolution shifting to data revolution, information and communication technology (ICT) plays more significant role in learning assessment. ICT-supported learning assessment means we can observe more fine-grained data and generate time-stamp for data traces, which offers unprecedented opportunities for large-scale assessment and microanalysis. Thanks to innovative technology and computing methods, researchers can now obtain insightful and policy-making findings through mining and interpreting big and interrelated data. This study investigated cognitive and non-cognitive factors and built assessment model of problem-solving competence based on cognitive patterns and mental representations of K-12 problem-solvers. Then the study designed an assessment approach to analyze problem-solving competence from two dimensions. The research foci are four-fold:First, the study investigated complex problem-solving competence and its cognitive processing process and modeled the online problem-solving behavior. From the perspective of mental representing and behavior pattern, and in combining with problem-solving language, semantic proposition and contextual features, online problem-solving activities and its related semiotics, methods, coding schemes and conceptual model were designed. Specifically regarding spatial continuity, functional relevancy and information dynamicity in online problem-solving process, two types of mapping relations and three levels of logical structure were formulated. The former includes information seeking-problem understanding and solution planning-variable manipulation; while the latter includes levels of problem-solving process, online operation and knowledge construction.Second, the study proposed a two-dimensional assessment model of problem-solving competence. From cognitive dimension, the study designed a three-component analyzing framework including knowledge content, knowledge organization and information processing, which interprets levels of problem solving actions (micro), skills (meso) and performance (macro). From non-cognitive dimension, the study investigated impacts and mechanisms of the non-cognitive factors on problem-solving competence and verified the casual influence model of environment-individual-behavior based on social cognitive theory.Third, the study designed a micro-world for online problem-solving competence assessment using prototyping method from software engineering. The overall system adopted Browser/Server architecture, and was developed in using Rich Internet Application (RIA) technique called Adobe Flex and Java Sever Pages (JSP) for browser and server sides respectively. The complex problem tasks were designed using human-computer interaction (HCI) and simulation techniques for assessing the knowledge of qualitative reasoning, linear equation model and finite state machine. A pilot study was conducted through recruiting 39 primary students in Shanghai. The acceptance and effectiveness of this micro-world was verified.Fourth, this study adopted the evidence-centered assessment design framework to analyze problem-solving competence. Five hundred and fifty-four students of grade three to grade five from a Shanghai primary school were participated in this large-scale online problem-solving assessment. The three components in terms of knowledge content, knowledge organization and information processing were further analyzed based on five assessment indicators that are conceptual breadth, correctness, integration, accessibility and information processing strategies. Non-cognitive factors and their causal relations were evaluated using structural equation modeling.The major findings revealed that problem-solving competence of primary school students in Shanghai were generally in the stage of metacognitive development and knowledge structure integration. Only few of them had reached the stage of knowledge structure optimizing, while another few stayed behind in the stage of concept development who needed additional intervention during their problem-solving learning. Regarding demographic factors, this study found no significant difference in overall performance between male and female students, but male students had significant higher meta-cognitive abilities than their female counterparts. The study also revealed that students from three grades had significant difference in all three aspects including cognition, metacognition and cognitive efficiency, which corroborates our hypothesis that children in between 9 and 12 years old experience important stage of brain’s functional structure development. The cognitive diagnostic analysis showed that these participants had good performance on exploring-understanding, monitoring-reflecting but poor on representing-planning. Based on the first stage clustering of student groups, the study conducted sequential analysis and found that different cluster students had their typical behavior patterns respectively. For example, high-quality-high-efficiency cluster’s students performed in expert-like model; impulse-high-efficiency cluster’s students focused on knowledge construction and reconstruction; potential-active cluster’s students were inadequate in metacognitive abilities such as planning and knowledge accommodation, but relied heavily on their experience to solve the new problem; low-quality-passive cluster’s students had incoherent problem-solving behavior which suggested their lack of conflict identification, metacognition and regulation abilities.From non-cognitive dimension, the study identified that personal characters such as learning motivation, learning emotion and self-regulation, value expectancy such as achievement goals, task value and control belief, as well as environmental support including parent and teacher support all play directive positive role in problem-solving behavior. Besides, both personal characters and environmental control had indirect impact on problem-solving behavior through the mediate variable of value expectancy and personal characters respectively. Another intriguing finding is that environmental control had negative impact on personal characters and value expectancy, which in turn, affected problem-solving behavior negatively. Generally speaking, value expectancy had the largest positive effect on problem-solving behavior, followed by personal character and environmental support. Meanwhile, negative effect of environmental control should not be overlooked as well.