The Relationship of Critical Thinking and Clinical Judgment

This document is a best practices essay from the international, multidisciplinary collection of teaching and training techniques, “Critical thinking and Clinical Reasoning in the Health Sciences.” Each essay in this set provides an example of training reasoning skills and thinking mindset described by

international experts in training clinical reasoning.

The Relationship of Critical Thinking and Clinical Judgment

Noreen C. Facione & Peter A. Facione

Lives depend on competent clinical reasoning. This makes it a moral imperative for health care providers to strive to monitor and improve their clinical reasoning and care related judgments. Knowing that this is the agreement owed to the public trust, agencies responsible for the accreditation of professional training programs and for the oversight of health care delivery have mandated the need to demonstrate competence in clinical reasoning in health care clinicians and students. This focus on competent reasoning and problem solving is not unique to health care. Sparked by a meeting of the United States Governors in the late 1980’s, educational mandates to teach and assess thinking and problem-solving have become increasingly pervasive. In this effort, the health sciences and military science have led the way. Nearly all performance-based credentialing programs and performance-based funding initiatives require thinking and problem solving as one of the educational outcomes that is worthy of assessment (Ackerman, Rinchuse, & Rinchuse, 2006). This focus on assessing competence in reasoning and problem-solving is also becoming a standard in the workplace.

The language of thinking

Critical thinking and reflective problem-solving are two common terms for the cognitive processes involved in clinical reasoning. Excellence in professional judgment is the result of the sound use of critical thinking skills and the reliable and strong disposition to use those critical thinking skills. The alternative (acting without adequate analysis of the problem, repeating a previous care delivery behavior unreflectively, or continuing to carry out a care delivery behavior without evaluating its effect) is not a standard of practice any of us would uphold. The discussion below outlines what has been learned to date about how humans engage high risk problems and arrive at competent judgments about what to believe and what to do. It also explores the challenge we face as researchers and educators to facilitate improvements in clinical reasoning for ourselves, our students, and our peers.

There are many prior accounts of the development of a consensus description of critical thinking, research carried out as a Delphi Study in the late 1980’s (American Philosophical Association, 1990). and replicated by an independent study at Penn State University (Jones & Ratcliff, 1993). We recommend that those unfamiliar with this literature seek out any of these previous papers (Facione & Facione, 1996a; 2006; Facione, Facione & Giancarlo, 2000). All our work in instrument development and in the theoretical and practical study of human reasoning stems from this seminal study focused on the importance of everyday competence in reasoned judgment. Here we offer a brief overview integrating our research on defining and measuring evidence of everyday reasoning and judgment with the emerging consensus of research attempting to explain human reasoning processes. The result is informative for training critical thinking and clinical reasoning.

We begin with a definition of critical thinking derived from a consensus of disciplines, and used widely to ground teaching and assessment of critical thinking:

Critical thinking is the process of purposeful, self-regulatory judgment. Using this process, an individual or a group gives reasoned consideration to evidence, contexts, conceptualizations, methods, and criteria in order to decide what to believe or what to do.

(American Philosophical Association Delphi Report, 1990).

In other words, critical thinking is a judgment process. Its goal is to decide what to believe and/or what to do in a given context, in relation to the available evidence, using appropriate conceptualizations and methods, and evaluated by the appropriate criteria. One way of describing how critical thinking relates to clinical judgment would be: Critical thinking is the process we use to make a judgment about what to believe and what to do about the symptoms our patient is presenting for diagnosis and treatment. This language is discipline free, because it refers to cognitive capabilities that can be generalized to all problem frames and all situational contexts. Here our interest is applying this terminology to the health sciences. To arrive at a judgment about what to believe and what to do, a clinician should consider the unique character of the symptoms (evidence) in view of the patient’s current health and life circumstances (context), using the knowledge and skills acquired over the course of their health sciences training and practice (methods, conceptualizations), anticipate the likely effects of a chosen treatment action (consideration of evidence and criteria), and finally monitor the eventual consequences of delivered care (evidence and criteria).

Adequate time to think

Newell (1990) provided us with some concrete data on how long it takes to process a novel observation or a novel problem demanding of a response. When humans are queried on a novel issue or problem, they require eleven to sixteen seconds to interpret the situation at hand and formulate even the most rudimentary reflective response. With forewarning they can summon relevant memories and content knowledge to inform their response, but otherwise processing time is required. Humans also frequently rely on heuristic maneuvers in an attempt to optimally address high stakes issues. Heuristic reasoning is believed to be most prevalent in time limited situations that do not admit of more reflective thinking, and in uncertain contexts when reflective thought fails to resolve ambiguities in the direction of a seemingly certain judgment (Gilovic, Griffin & Kahneman, 2002). More on this topic in the section below entitled ‘Two systems of reasoning’ but for now we return to the issue of ‘time to think.’

Sixteen seconds is far longer than we are accustomed to waiting for a response after we pose an important question to a clinician, or even a student who is supposed to be prepared for the clinic. Both may feel the desire to respond thoughtfully and provide the optimal opinion, but far more often they first feel the pressure to respond quickly. So, what is forthcoming usually begins as only half-thought-out, with late breaking insights and necessary edits coming later as additional ideas are formulated. If the problem we pose is novel, and the clinician values accuracy and comprehensiveness as a component of the response, we may hear, “Now let me think about that for a moment.” Hearing this response should engender confidence, but often instead it engenders doubt.

Learning to ‘think aloud,’ supplying evidence of the process of one’s thinking and subsequent judgment (the assumptions made, the evidence base applied, the logical framing) offers a way for the listener to both evaluate the quality of the judgment and to learn to reason better themselves, This is demonstrated in discussions of think aloud exercises in some of the chapters to follow. The accuracy of Newell’s findings (the need for time to think) can be readily observed by asking anyone a novel question that requires reflective thought, and then recording the time to a response. This is true, regardless of expertise level, when the question or problem is truly novel. The physiological realities of human thinking make it important for educators to control the tempo of teaching and learning sessions if they are to effectively lead to improved clinical reasoning. Those who answer too quickly may not have thought well.

Clinical reasoning and expertise

When clinical problems are familiar, we can rely on externally developed protocols and internal ‘mental scripts’ to assist us in deciding what to believe and what to do about the problem. The externally developed protocols are elaborate and rise to the status of standards when the consequences of error are high and when society is concerned with safety. There is still need for reflective thought when using protocols to assure that they remain appropriate to the case and that expected results occur.

Internally developed ‘mental scripts’ are a function of expertise. The Dreyfus and Dreyfus model of expertise, which has been adapted by Benner for Nursing, (Benner 1994, 2004) is a phenomenological model that provides a description of the increasing sense of ease experienced over time by the clinician, moving from novice practice to more expert practice. Most models of expertise describe the novice who encounters a problem as attending indiscriminately to data in an attempt to recognize key relationships that will then allow the application of knowledge which they believe to be relevant. The expert, in contrast, recognizes most problems by pattern, and resolves them without a significant awareness of reflective thinking. An expert does this through the retrieval of similar cases examples stored in episodic memory, a larger array of relevant knowledge stored in semantic memory, and the use of other heuristic thinking processes.

Benner’s work describing the ‘lived experience’ of clinical reasoning notes the seeming inability to reflect on the thinking process that occurs in the expert clinician, describing it as ‘intuiting.’ In contrast, other cognitive science models of human reasoning explain this lack of conscious reflection as a function of several cognitive processes: heuristic reasoning (thinking maneuvers and shortcuts discussed below), automatic thought (the ability to accomplish an array of tasks without conscious attention), and the absence of perception of meta-cognition (listening to or thinking about your thinking). In the case of automatic thinking, familiarity with the tasks required frees cognitive resources to focused more specifically on only the unique aspects of the situation or perhaps even a different problem altogether. Recall the experience of driving home from work rehearsing approaches to resolving an interpersonal issue. Possibly you exit the car realizing that you really didn’t ‘see’ the road and the other drivers for the majority of the thirty-minute drive. We even have language for this, ‘running on autopilot.’ But clearly some cognitive process, outside of your awareness, was monitoring your driving, making lane changes, braking, using turn signals, seeing the other cars. It is not known how often the autopilot function impacts clinical reasoning, nor what percentage of those impacts are negative.

Models of expertise help us to understand how different groups of clinicians are likely to approach clinical problems. A high level of expertise does not assure flawless reasoning in the clinician, any more than we can be sure frequent errors will be made by the novice. Novices are known to be slower to come to a judgment because they require more time for reflective thought and additional data searching. Novices err through problem misidentification and uncertainty about knowledge application. But experts also err due to problem misidentification, and they are more prone to being inattentive to those differences in the problem which make it the odd exception to the pattern, and which render the modal responses inappropriate.

Understanding the cognitive effort entailed by the novice or expert state suggests several things about the training of clinical reasoning. Feelings of comfort when working on familiar problems in familiar contexts should not be confused with genuine clinical expertise. A person may be comfortable doing roughly the same thing repeatedly, as demanding as that may be, but not have the expertise to be able to resolve new problems, to adapt old ways to new situations, or even to recognize limitations or shortcomings in the way he or she has always gone about doing those familiar things.

Expert clinicians are never beyond the need to actively monitor the soundness of their clinical reasoning. While we might allow ourselves to fold the laundry or cut the grass automatically, we can’t allow this type of disconnect when the life or health of others is at stake. Hence, we need to continuously build the cognitive skills and habits of mind inherent in critical thinking as the preferred tools of the clinical judgment process, the conscious reflection about what to believe and what to do in the clinical context. Novice clinicians will have far more novel problems to address, but those who have stronger critical thinking skills will progress toward higher levels of competence and expertise.

Two systems of reasoning

Newer research in human reasoning finds evidence of the function of two interconnected ‘systems’ of reasoning. ‘System 1’ is conceptualized as reactive, instinctive, quick, and holistic. System 1 often relies on highly expeditious heuristic maneuvers which can yield useful response to perceived problems without recourse to reflection. By contrast, ‘System 2’ is described in the cognitive science literature as more deliberative, reflective, analytical, and procedural. System 2 is generally associated with reflective problem-solving and critical thinking. In its decision making processes System 2 also uses some heuristic maneuvers. We offer a fuller discussion in Thinking and Reasoning in Human Decision Making: The Method of Argument and Heuristic Analysis, (Facione & Facione, 2007) but will recap key elements here.

In humans these two systems never function completely independently. One is not naturally “better” than the other; in fact there are situations where each offers something of a corrective effect on the other. Because both systems rely on cognitive heuristics and because these maneuvers are known to have the potential to introduce error and biases into human reasoning, knowing something about heuristic reasoning is important to those who are attempting to train or to measure clinical reasoning. There is a growing literature on this research but reading several of the foundational books and papers will provide the needed insight into how we believe humans actually think and make clinical judgments (Gilovic, Griffin & Kahneman, 2002; Kahneman, Slovic & Tversky, 1982; Montgomery 1998). Here we provide only the briefest overview.

Some hypothesize that lacking claws, fangs, skeletal armor, protective fur, poisonous secretions, natural camouflage, strength, or speed, the human species survived, because of some other evolutionary advantage. One factor was the fast, efficient, and effective problem-solving made possible by heuristic reasoning. When used well, heuristic thinking helps us survive, but misuse of this type of reasoning, when not overridden by reflective thought (System 2), leads to predictable error. For example, consider the influence on behavior of the affect heuristic. This heuristic might function well pre- consciously like this: “unprotected needle –> BAD! (stop)” Twenty years after the AIDS epidemic, no reflective argument should be needed for a trained clinician to recognize the immediate danger presented by an unshielded needle. A misuse of this heuristic might be “comfort food –> GOOD,” depending on how much one is trying to lose weight. Favoring choices that avoid loss, recognizing similarities, guessing about future events by playing a movie in your head of what will happen, and assuming one is able to control all threats, are examples of heuristic maneuvers that are typically below the level of conscious thought. This system 1 thought has a powerful effect on behavior as documented in these references here and others at the end of this chapter. (Montgomery, 1998; Tversky & Kahneman, 1973; Kahneman, Slovic & Tversky, 1982; Weinstein, 1982).

Perhaps the current preoccupation with heuristic reasoning results from being relatively unaware of it in the past. While cognitive and social psychology has been working to impact the understanding of human reasoning, many have been holding to early descriptions by Plato and Aristotle of humans as always striving to be deliberative, reflective, and logical. Not true. Even when we are making high stakes clinical judgments, this is not true. The current conceptualization of two interacting and complementary systems better explains the evolutionary success of our species. A new method of argument analysis has emerged that includes an examination of the entire decision-making process, both System 1 and

System 2 and the influences of cognitive heuristics along with argument making on decision outcomes (Facione & Facione, 2007). It’s likely that this method of decision analysis will bring new insights into how some common clinical errors occur. The important thing to realize is that although you may not yet have heard much about this in the past, this is how we think. Effectively mixing System 1 and System 2 cognitive maneuvers to identify and resolve clinical problems is the normal form of mental processes involved in sound, expert, clinical reasoning. Misusing heuristic reasoning maneuvers, in the context of poor logic and misinformation is a description of poor clinical reasoning. Figure 1 below is a diagram locating the thinking processes we have been discussing. Even good thinkers make both System 1 and 2 errors from time to time. We misinterpret things, we overestimate, or we underestimate our chances of succeeding, rely on mistaken analogies, reject options out of hand, rely too heavily on feelings and hunches, judge things credible when they are not, etc. And there is one more strategy we humans use to become confident about decisions. This additional strategy needs to be factored in before the story of clinical reasoning is fully told.

Figure 1: The Argument and Heuristic Analysis Model of Decision-Making

Dominance structuring

Richly considered judgments about what to believe or do are typically structured around one dominant conclusion. In the case of a clinical judgment made under risky and uncertain conditions, that judgment emerges as a function of eliminating possible choices based on the evidence available. Subsequently, even when new evidence becomes available that changes the value of the chosen alternative, it proves difficult to override one’s original conclusion. This remains true, even when new information renders the supporting reasons for the initial decision questionable at best. Creation of a ‘dominance structure’ (Montgomery, 1989) around one’s choice of action (or inaction) can sustain confidence in the judgment even when the negative consequences of error are extremely high.

We all do this. We need to do this to attain significant confidence to act under uncertain conditions. Otherwise, we would be more likely to delay a needed judgment or fail to maintain our resolve, thus making errors of omission. This would constitute a breach in the trust placed in us as health care providers. But there are dangers here. We can all think of situations where an ineffective plan of care was continued too long to be optimal, and was even harmful for a given individual. If we add the realities surrounding the interpersonal power structure necessary for the function of a medical team, there is an added pressure of responsibility on team leaders to be aware of dominance structures around particular diagnostic or treatment decisions which they may be sustaining long past their utility for improving the health of individual patients. The same situation could be described in relationship to the retention of policies and practices well beyond their appropriate application, or negative judgments against co-workers because initial negative impressions are wrongly sustained.

Problem parameters

When we interpret presenting symptoms, we explore their characteristics (frequency, severity, persistence, duration…), knowing that these characteristics modify the symptoms’ meaning. So, it is with the characteristics of clinical problems, or all of life’s problems for that matter. A problem’s attributes pose differing challenges for the thinking skills and habits of mind required for successful problem resolution. We have already mentioned above that new, or novel, problems and situations are approached differently than familiar ones. Other key characteristics of problem situations are the associated risk, the problem’s complexity, the spontaneity of its occurrence, accompanying time constraints, and the need for specialized knowledge or collaboration required to address a response. Reflect on the likely characteristics of the typical problems presented in clinical practice and recall your own initial clinical experiences as a student. When you were a health science student yourself, many of the problems you encountered in the clinical setting appeared to you to be: 1) novel, 2) complex, 3) high stakes, 4) time constrained, 5) spontaneous, and 6) requiring of more specialized knowledge than you had at your fingertips. Finally, in spite of being a trainee, often you probably felt you had to resolve problems individually rather than relying on collaboration. Your responses to those same problems now will depend in part on the nature of your current practice and the expertise you have developed. The perceived risk attached may be similar, as most clinical judgments are high stakes for you as well as your patients. But there are probably a higher proportion of those problems which are now, for you, highly familiar, less complex, more anticipatable, more within your knowledge base, where time to think is less of an issue, and you can rely on collaboration with other members of the health care team.

Training clinical judgment across all of these possible problem parameters requires a careful pedagogical approach. We need to remember to provide time for trainees to think. Scaffolding the complexity of problems presented to students and novice staff will improve their ability to think well. Debriefing case outcomes as to the embedded clinical reasoning (surfacing assumptions, preliminary diagnoses, suspected interacting factors) externalizes the reasoning process so that it can be critiqued or praised. In the end, training health professionals to think well in clinical practice is a delicate dance, balancing the need to function in swiftly evolving real world cases with the need to allow every promising student time to develop their critical thinking skills.

The emphasis here is on ‘promising.’ One thing we have learned in the course of our work in critical thinking measurement is that many students admitted to health science programs do not have the requisite thinking skills to become great or even competent diagnosticians. We know this by examining thousands of critical thinking test scores from students across the health science disciplines (Facione & Facione, 1997; Chirema, 2006) and from the research that has been done to link critical thinking test scores with success on licensure examinations in the health sciences (Williams, Schmidt, Tilliss et. al., 2006).

Taking a critical thinking approach to clinical practice entails two linked goals: accurate problem identification and optimal problem resolution. The first is essential. Taking action to solve the wrong problem may work occasionally in politics, but will not work for the sick and dying. The second is also essential. What are the consequences of not taking a critical thinking approach to developing a clinical treatment plan? If clinicians or our health science students do not have the possibility to think reflectively about clinical situations, they will use other methods for problem resolution. Some alternatives to critical thinking are: 1) to ask someone else what to do; 2) to do nothing; 3) to keep on doing something which is failing to achieve our desired outcome; or 4) to do something, anything, new just because it has not been tried yet. The first three are a recipe for mediocrity or failure through omission. The fourth is perhaps most dangerous if the presumed diagnosis is mistaken or if the chanced upon trial treatment turns out to be not simply ineffectual but actually harmful.

At its best, a focus on reflective thinking, and some attempt to meta-cognitively monitor our use of heuristic thinking, allows one to be thoughtful about intellectual honesty, analytically anticipating what happens next, demanding the wisdom of making decisions in a fair-minded and timely manner, and the attempt to eliminate personal biases. These habits of mind have been identified as those of the ideal critical thinker (Facione, Facione & Sanchez, 1994; American Philosophical Association, 1990).

Multiple measurement modalities

The assessment of critical thinking lends itself to the full array of measurement methods. Here as in all areas of measurement, multiple measures allow the assessment of critical thinking in many clinical practice contexts. Multiple choice (Facione & Facione, 2006; Facione, 2000; Watson & Glaser, 1980; Ennis, Millman & Tomko, 1985) or short answer essay tests (Ennis & Weir, 1985), can be used to take one measure of critical thinking skill. These are particularly useful as diagnostic tests for reasoning competence for newly hired clinicians, health science students, and even health care clients who are not cognitively impaired. Some of these instruments use multiple choice questions requiring test- takers to apply critical thinking skills not only to solve a problem but to evaluate the quality of the solution and provide the evidence for that quality. Likert-style attitudinal measures can gauge critical thinking habits of mind (Facione & Facione 1992; Giancarlo 1998). Others have reported the utility of the multiple-choice format to test reasoning process when the items are written well (Leung, Mok & Wong, 2007). Rubrics can be constructed to assess particular critical thinking skills or to obtain a holistic ratings of critical thinking skills and disposition. When care is taken to train rater and assure their valid and reliable observation of critical thinking as it presents in real time, these rubrics can be used to assess critical thinking exhibited by clinicians or students in routine case conferences, planned classroom presentations, written assignments, or immediately after addressing a spontaneous bedside situation (Facione & Facione 1996b; Facione & Facione, 1994).

Each assessment device has different potential for assessing critical thinking in relation to more or less authentic clinical judgment situations. Any test of critical thinking must call forth evidence of critical thinking itself and not merely evidence of content knowledge if they are to assess an individual’s ability to think well. Psychological measures of critical thinking disposition can provide a barometer for whether a given individual is disposed to use their critical thinking skills rather than to rely on some other way of dealing with problems. These test an individual’s willingness to try to think well. We need clinicians who are both willing and able to think well.

Summary

The focus on the need to train clinical judgment per se is rather new. At every level from novice to expert, clinical judgment regarding diagnosis, treatment, and on-going evaluation of patient outcomes is a fundamentally complex reasoning process which is applied to problems characterized by a multiplicity of potentially varying parameters, and which consumes cognitive resources including time to think as it relies upon core critical thinking skills and habits of mind, integrating our two systems of decision-making, susceptible to the benefits and shortcomings of cognitive dominance structuring. How clinical reasoning is experienced, even by the expert, is not a reliable measure of either the complexity or the quality of reasoning process. We would make an analogy to the practiced use of customized software on a computer. The apparent ease of the experience belies the cognitive, physiological, and mechanical processes at work. We cannot make this point strongly enough: because the potential implications of overconfidence in one’s expertise in clinical reasoning could not be more grave for the sick and dying. Previously we were overly confident that students and novice clinicians would somehow “naturally” advance in their clinical reasoning as they were introduced to typical clinical case scenarios. But we have learned that without a direct focus on the critical thinking processes used to interpret, analyze, infer, evaluate, and explain what is going on, progress in clinical reasoning is an uncertain outcome. True, this progress may come entirely from the learner’s own awareness of how she or he needs to go about internalizing and growing their clinical reasoning expertise. But when wise instructors and mentors facilitate reflective problem-solving by prompting meta-analysis and evaluation of clinical reasoning through their course assignments and pedagogical approaches, progress is more certain. Changes in health science curricula to case-based pedagogies and problem based learning are relatively recent, but we are already seeing evidence of improved outcomes as educational researchers report that their pedagogical efforts to improve clinical reasoning skills and dispositions have been demonstrated in a variety of health science disciplines and contexts (Jenicek 2006; McAllister 2005; Tiwari, Lai, So & Yuen, 2006, Shin, Jung, Shin, & Kim, 2006; Torre, Daley, Stark-Schweitzer, Siddhartha, et al., 2007; Ozturk, Muslu & Dicle, 2007; Suliman, 2006; Velde & Wittman, 2006). Expanding our knowledge of how to do this well will surely follow.

References

Ackerman, M., Rinchuse D. & Rinchuse D. (2006). ABO certification in the age of evidence and enhancement. American Journal of Orthodontics, Dentofacial, and Orthotics. Vol. 130 (2), 133-140.

American Philosophical Association [APA] (1990). Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction, Recommendations Prepared for the Committee on Pre-College Philosophy. ERIC Doc. No. ED 315-423.

Benner, P. From Novice to Expert: Excellence and Power in Clinical Practice. Menlo Park, CA: Addison Wesley., 1984. Benner, P. (2004). Using the Dreyfus Model of Skill Acquisition to Describe and Interpret Skill Acquisition and Clinical Judgment in Nursing Practice and Education, Bulletin of Science, Technology & Society, Vol. 24, No. 3, 188-199.

Chirema, K. (2006). The use of reflective journals in the promotion of reflection and learning in post-registration nursing students. Nursing Education Today, 27,192-202.

Ennis, R. Millman, J. & Tomko, T. The Cornell CT Test (Levels X and Z). Pacific Grove, CA: Midwest Publications, 1985.

Ennis, R & Weir, E. The Ennis-Weir CT Essay Test: An Instrument for Testing and Teaching. Pacific Grove, CA: Midwest Publications, 1985.

Facione, N. & Facione, P. (1996a). Externalizing the critical thinking in knowledge development and clinical judgment. Nursing Outlook , 44, 129-136.

Facione, N. & Facione P. Critical Thinking Assessment in Nursing Education Programs: An Aggregate Data Analysis. Millbrae, CA: California Academic Press, 1997.

Facione, N. & Facione, P. (2006) The cognitive structuring of patient delay. Social Science & Medicine, Vol. 63, pp. 3137-3149.

Facione, N. & Facione P. The Health Sciences Reasoning Test (HSRT). Millbrae, CA: California Academic Press, 2006. Facione, N, Facione, P & Sanchez, C. (1994). Critical thinking disposition as a measure of competent clinical judgment: The development of the California Critical Thinking Disposition Inventory. Journal of Nursing Education, 33 (8), 345-350.

Facione, P. The California Critical Thinking Skills Test (CCTST). Millbrae, CA: California Academic Press, 2000. Facione, P. & Facione, N. The California Critical Thinking Disposition Inventory (CCTDI). Millbrae, CA: California Academic Press, 1992.

Facione, P. & Facione, N. The Professional Judgment Rating Form: Novice/Intern Level. Millbrae, CA: California Academic Press, 1996b.

Facione, P. & Facione, N. Thinking and Reasoning in Human Decision Making: The Method of Argument and Heuristic Analysis. Millbrae, CA: The California Academic Press, 2007.

Facione, P, Facione, N, & Giancarlo, C, (2000). The Disposition toward critical thinking: Its character, measurement, and relationship to critical thinking skill. Informal Logic, Vol. 20 (1), 61-84.

Giancarlo, C, (1998). The California Measure of Mental Motivation (CM3). Millbrae, CA: California Academic Press. 1998.

Gilovic T, Griffin D, & Kahneman D, Heuristics and biases: The psychology of intuitive judgment. Cambridge, UK: Cambridge University Press, 2002.

Jenicek, M. (2006). How to read, understand, and write ‘Discussion’ sections in medical articles. An exercise in critical thinking. Medical Science Monitor. 12(6): SR28-36. E-publication, 2006, May 29.

Jones, B. & Ratcliff, J. National Center on Post Secondary Teaching, (1993). Learning and Assessment, The Pennsylvania State University. US Department of Education’s Office of Educational Research and Improvement (OERI) Grant No.: R117610037, CFDA No.:84.117G.

Kahneman, D. Slovic, P. & Tversky, A. Judgment under Uncertainty: Heuristics and Biases. Cambridge, UK: Cambridge University Press, 1982.

Leung, S.F., Mok, E. & Wong, D. (2007). The impact of assessment methods on the learning of nursing students. Nursing Education Today, E-publication, ahead of print at the time of this printing.

McAllister, M. (2005). Transformative teaching in nursing education: leading by example. Collegian, Vol. 12 (2), 11-6. Montgomery, H. From cognition to action: The search for dominance in decision making. In, Process and Structure in Human Decision-Making, (pp 23-49). John Wiley & Sons Chichester, UK, 1989.

Newell, A. Toward Unified Theories of Cognition. Cambridge, MA. Harvard University Press, 1990.

Ozturk, C., Muslu, G.K., Dicle, A. (2007, Dec 1). A comparison of problem based and traditional education on nursing students’ critical thinking dispositions. Nursing Education Today (e-pub ahead of print at this printing).

Shin, K, Jung, D, Shin, S, & Kim, M. (2006). Critical thinking dispositions and skills of senior nursing students in associate, baccalaureate, and RN to BSN programs. Journal of Nursing Education, Vol. 45 (6), 233-7.

Suliman, W. (2006). Critical thinking and learning styles of students in conventional and accelerated programs. International Nursing Review, Vol. 53, 73-79.

Tiwari, A., Lai, P. So, M., & Yuen, F. (2006). A comparison of the effects of problem-based learning and lecturing on the development of students’ critical thinking, Medical Education, 40, 547-554.

Torre, DM., Daley, B., Stark-Schweitzer, T., Siddhartha, S., Petkova, J., & Ziebert, M. (2007). A qualitative evaluation of Medical student learning with concept mapping. Medical Teaching, 29(9), 949-955.

Tversky, A. & Kahneman D, (1973). Availability: A Heuristic for Judging Frequency and Probability. Cognitive Psychology. Vol. 5: pp. 207-232.

Velde, B. Wittman, P. & Vos, P. (2006). Development of critical thinking in occupational therapy students. Occupational Therapy International. Vol. 13 (1), 49-60.

Watson, G. & Glaser, E. Watson-Glaser Critical Thinking Appraisal, Forms A and B. New York: Psychological Corporation, 1980.

Weinstein, N. (1982). Unrealistic optimism about susceptibility to health problems. Journal of Behavioral Medicine. Vol. 5. pp. 441-460.

Williams, K, Schmidt, C, Tilliss, T, Wilkins, K, & Glasnapp, D. (2006). Predictive validity of critical thinking skills and disposition for the national board dental hygiene examination: a preliminary investigation. Journal of Dental Education. 70(5), 536-44.

* * * * * *

Although this series focuses on health science content, the techniques are transferrable to all types of training programs and educational projects.