Using “Concept Sorting” to Study Learning Processes and Outcomes

Department of Medicine, University of Calgary Faculty of Medicine, Calgary, Alberta, Canada.
Academic Medicine (Impact Factor: 2.93). 09/2002; 77(8):831-6. DOI: 10.1097/00001888-200208000-00017
Source: PubMed


First, to evaluate "concept sorting" as a tool for assessing knowledge organization in the memories of first-year medical students, and second, to study the relationship between knowledge organization and examination performance.
During 2001, first-year medical students taking the Renal Course at the University of Calgary Faculty of Medicine were given a questionnaire on scheme use and were given a concept-sorting task in the domain of metabolic alkalosis. The sophistication of their concept sorting was graded using the number of physiology-based groups they formed. Review of the course's examination scores allowed correlation with concept-sorting scores. Statistical analyses used Fisher's exact test and the two-sample t-test. Pearson's correlation coefficient and the kappa statistic were used for correlation between raters.
A total of 81 of 99 students completed the study. The concept-sorting score (mean +/- SEM) for students who used the scheme was higher than was the score for students who did not (2.5 +/- 0.14 versus 1.91 +/- 0.12, p =.016). Students who scored higher in the concept-sorting task, referred to as "deep learners," scored higher than did "surface learners" on exam questions on metabolic alkalosis (2.81 versus 2.29, p =.02). There was no difference in the overall examination performances between the two groups.
Concept sorting may be a useful tool for studying the learning process. Scheme use by students produces a positive outcome on examination performance.

4 Reads
  • Source
    • "Another implication of considering knowledge structure for medical diagnoses a milestone on the journey to medical expertise is that educational interventions can be specifically geared towards improving this surrogate end-point. Visuospatial cognitive strategies such as diagnostic schemes appear to be effective in the presentations studied here (McLaughlin et al., 2002). Knowledge structure of medical diagnoses assessment tools, such as ConSort Ó , may be useful for comparing the effectiveness of different educational interventions on the development of knowledge structure for medical diagnoses. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Evolution from novice to expert is associated with the development of expert-type knowledge structure. The objectives of this study were to examine reliability and validity of concept sorting (ConSort) as a measure of static knowledge structure and to determine the relationship between concepts in static knowledge structure and concepts used during diagnostic reasoning. ConSort was used to identify static knowledge concepts and analysis of think-aloud protocols was used to identify dynamic knowledge concepts (used during diagnostic reasoning). Intra- and inter-rater reliability, and correlation across cases, were evaluated. Construct validity was evaluated by comparing proportions of nephrologists and students with expert-type knowledge structure. Sensitivity and specificity of static knowledge concepts as a predictor of dynamic knowledge concepts were estimated. Thirteen first-year medical students and 19 nephrologists participated. Intra- and inter-rater agreement for determination of static knowledge concepts were 1.0 and 0.90, respectively. Reliability across cases was 0.45. The proportions of nephrologists and students identified as having expert-type knowledge structure were 82.9% and 55.8%, respectively (p=0.001). Sensitivity and specificity of ConSort((c)) in predicting concepts that were used during diagnostic reasoning were 96.8% and 27.8% for nephrologists and 87.2% and 55.1% for students. ConSort is a reliable, valid and sensitive tool for studying static knowledge structure. The applicability of tools that evaluate static knowledge structure should be explored as an addition to existing tools that evaluate dynamic tasks such as diagnostic reasoning.
    Advances in Health Sciences Education 09/2007; 12(3):265-78. DOI:10.1007/s10459-005-6029-3 · 2.12 Impact Factor
  • Source
    • "An important assumption [and criticism] of CMA is that the map generated represents the 'true' knowledge structure of the subject rather than merely a creation of the research tool; a hypothesis that is difficult to prove. The observation, however, that the map generated by CMA correlates with performance provides some support for this assumption upon which CMA is based [7,8]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: During evolution from novice to expert, knowledge structure develops into an abridged network organized around pathophysiological concepts. The objectives of this study were to examine the change in knowledge structure in medical students in one year and to investigate the association between the use of a conceptual framework (diagnostic scheme) and long-term knowledge structure. Medical students' knowledge structure of metabolic alkalosis was studied after instruction and one year later using concept-sorting. Knowledge structure was labeled 'expert-type' if students shared >or= 2 concepts with experts and 'novice-type' if they shared < 2 concepts. Conditional logistic regression was used to study the association between short-term knowledge structure, the use of a diagnostic scheme and long-term knowledge structure. Thirty-four medical students completed the concept-sorting task on both occasions. Twenty-four used a diagnostic scheme for metabolic alkalosis. Short-term knowledge structure was not a correlate of long-term knowledge structure, whereas use of a diagnostic scheme was associated with increased odds of expert-type long-term knowledge structure (odds ratio 12.6 [1.4, 116.0], p = 0.02). There was an interaction between short-term knowledge structure and the use of a diagnostic scheme. In the group who did not use a diagnostic scheme the number of students changing from expert-type to novice-type was greater than vice versa (p = 0.046). There was no significant change in the group that used the diagnostic scheme (p = 0.6). The use of a diagnostic scheme by students may attenuate the loss of expert-type knowledge structure.
    BMC Medical Education 07/2006; 6:37. DOI:10.1186/1472-6920-6-37 · 1.22 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Cognitive psychology research supports the notion that experts use mental frameworks or "schemes", both to organize knowledge in memory and to solve clinical problems. The central purpose of this study was to determine the relationship between problem-solving strategies and the likelihood of diagnostic success. Think-aloud protocols were collected to determine the diagnostic reasoning used by experts and non-experts when attempting to diagnose clinical presentations in gastroenterology. Using logistic regression analysis, the study found that there is a relationship between diagnostic reasoning strategy and the likelihood of diagnostic success. Compared to hypothetico-deductive reasoning, the odds of diagnostic success were significantly greater when subjects used the diagnostic strategies of pattern recognition and scheme-inductive reasoning. Two other factors emerged as independent determinants of diagnostic success: expertise and clinical presentation. Not surprisingly, experts outperformed novices, while the content area of the clinical cases in each of the four clinical presentations demonstrated varying degrees of difficulty and thus diagnostic success. These findings have significant implications for medical educators. It supports the introduction of "schemes" as a means of enhancing memory organization and improving diagnostic success.
    Medical Education 09/2003; 37(8):695-703. DOI:10.1046/j.1365-2923.2003.01577.x · 3.20 Impact Factor
Show more