Fyfe, E., de Leeuw, J. R., Carvalho, P. F., Goldstone, R., Sherman, J., … Motz, B. (2021). ManyClasses 1: Assessing the generalizable effect of immediate versus delayed feedback across many college classes. Advances in Methods and Practices in Psychological Science, 4, 1-24. https://doi.org/10.31234/osf.io/4mvyh
Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.
Download PDF of paper
Carvalho, P. F., & Goldstone, R. L. (2021). The most efficient sequence of study depends on the type of test. Applied Cognitive Psychology, 35, 82-97.
Across three experiments featuring naturalistic concepts (psychology concepts) and naïve learners, we extend previous research showing an effect of the sequence of study on learning outcomes, by demonstrating that the sequence of examples during study changes the representation the learner creates of the study materials. We compared participants’ performance in test tasks requiring different representations and evaluated which sequence yields better learning in which type of tests. We found that interleaved study, in which examples from different concepts are mixed, leads to the creation of relatively interrelated concepts that are represented by contrast to each other and based on discriminating properties. Conversely, blocked study, in which several examples of the same concept are presented together, leads to the creation of relatively isolated concepts that are represented in terms of their central and characteristic properties. These results argue for the integrated investigation of the benefits of different sequences of study as depending on the characteristics of the study and testing situation.
Download PDF of paper
Tullis, J. G., & Goldstone, R. L. (2020). Why does peer instruction benefit student learning? Cognitive Research: Principles and Implications, 5:15, 1-12. https://doi.org/10.1186/s41235-020-00218-5
In peer instruction, instructors pose a challenging question to students, students answer the question individually, students work with a partner in the class to discuss their answers, and finally students answer the question again. A large body of evidence shows that peer instruction benefits student learning. To determine the mechanism for these benefits, we collected semester-long data from six classes, involving a total of 208 undergraduate students being asked a total of 86 different questions related to their course content. For each question, students chose their answer individually, reported their confidence, discussed their answers with their partner, and then indicated their possibly revised answer and confidence again. Overall, students were more accurate and confident after discussion than before. Initially correct students were more likely to keep their answers than initially incorrect students, and this tendency was partially but not completely attributable to differences in confidence. We discuss the benefits of peer instruction in terms of differences in the coherence of explanations, social learning, and the contextual factors that influence confidence and accuracy.
Download PDF of paper
Goldstone, R. L. (2019). Building inner tools. In M. Colombo, E. Irvine, M. Stapleton (Eds.)Andy Clark and his Critics. Oxford, England: Oxford University Press (pp. 113-127).
Humans show a striking penchant for creating tools to benefit our own thought processes. Andy Clark (2003, 2008) has convincingly argued that the tools that we as humans recruit become integrated parts of an extended cognitive system that includes us as just one component. By extending cognition beyond our brains, Clark presents an “embiggened” perspective on what it means to be a cognizer and a person more generally. This perspectival shift runs counter to some recent forms of argumentation that in effect work to minimize personhood. For example, arguments for lack of personal culpability can take the form of “It wasn’t my fault. It was the fault of my ___ ” to be filled in, perhaps, by “upbringing,” “genes,” “neurochemistry,” “diet,” or “improperly functioning amygdala.” Instead, Clark (see also Dennett 1989) offers the opposite line of argumentation, according to which we consist not only of our amygdalae and hippocampi but also potentially our glasses, notebooks, friends, supporting technologies, and culture.
Download PDF of this chapter
Lara-Dammer, F., Hofstadter, D. R., & Goldstone, R. L. (2019). A Computational Model of Scientific Discovery in a Very Simple World, Aiming at Psychological Realism. Journal of Experimental & Theoretical Artificial Intelligence, 1-22. 10.1080/0952813X.2019.1592234
We propose a computational model of human scientific discovery and perception of the world. As a prerequisite for such a model, we simulate dynamic microworlds in which physical events take place, as well as an observer that visually perceives and makes interpretations of events in the microworld. Moreover, we give the observer the ability to actively conduct experiments in order to gain evidence about natural regularities in the world. We have broken up the description of our project into two pieces. The first piece deals with the interpreter constructing relatively simple visual descriptions of objects and collisions within a context. The second phase deals with the interpreter positing relationships among the entities, winding up with elaborated construals and conjectures of mathematical laws governing the world. This paper focuses only on the second phase. As is the case with most human scientific observation, observations are subject to interpretation, and the discoveries are influenced by these interpretations.
Download PDF of paper
Carvalho, P. F., & Goldstone, R. L. (2019). When does interleaving practice improve learning? In J. Dunlosky & K. A. Rawson (Eds.) The Cambridge Handbook of Cognition and Education. Cambridge, England: Cambridge University Press. (pp. 411-436).
As you flip through the pages of this handbook you will notice that the content does not seem to be randomly organized. The content of the handbook is sequenced in a particular way: foundations before general strategies, background before applications, etc. The editors envisaged a sequence of topics, the authors of each topic envisaged a sequence of information in each chapter, and so on. We selected a particular sequence because we considered it to be effective. Deciding how to sequence information takes place all the time in educational contexts, from educators deciding how to organize their syllabus to educational technology designers deciding how to organize a piece of educational software, from handbook editors and writers deciding how to organize their materials, to students making decisions as to how to organize their study. One might imagine that as long as all students study the same materials, regardless of the sequence in which they study it, they will all learn the same information. This could not be further from the truth. In this chapter, we will review evidence of how and why the sequence of study changes what is learned. In doing so, we will try to uncover the powerful ways in which sequence can improve or deter learning.
Download PDF of Chapter here
Motz, B. A., Carvalho, P. F., de Leeuw, J. R., & Goldstone, R. L. (2018). Embedding experiments: staking causal inference in authentic educational contexts. Journal of Learning Analytics,5, 47-59. doi: 10.18608/jla.2018.52.4
To identify the ways teachers and educational systems can improve learning, researchers need to make causal inferences. Analyses of existing datasets play an important role in detecting causal patterns, but conducting experiments also plays an indispensable role in this research. In this article, we advocate for experiments to be embedded in real educational contexts, allowing researchers to test whether interventions such as a learning activity, new technology, or advising strategy elicit reliable improvements in authentic student behaviours and educational outcomes. Embedded experiments, wherein theoretically relevant variables are systematically manipulated in real learning contexts, carry strong benefits for making causal inferences, particularly when allied with the data rich resources of contemporary e-learning environments. Toward this goal, we offer a field guide to embedded experimentation, reviewing experimental design choices, addressing ethical concerns, discussing the importance of involving teachers, and reviewing how interventions can be deployed in a variety of contexts, at a range of scales. Causal inference is a critical component of a field that aims to improve student learning; including experimentation alongside analyses of existing data in learning analytics is the most compelling way to test causal claims.
Download PDF of article here
McColeman, C., Michal, A., Goldstone, R. l., Schloss, K., Kaminski, J., & Hullman, J. (2018). Data visualization as a domain to research areas in cognitive science. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 35-36). Madison, Wisconsin: Cognitive Science Society.
How people are able to turn information in the environment into meaning is a critical question for cognitive science. That environment is increasingly data-driven. Using data to inform decisions and improve understanding of the world is a valuable component of critical thinking, and serves as the foundation of evidence-based decision making. Designing graphical representations can make those data more accessible, such that users may engage the visual system and capacity for visual pattern recognition to discern regularities and properties of data. We ultimately want to understand the connection between the initial perception of data visualizations and conceptual understanding of information. Data visualizations, broadly, are the representation of recorded values in visual form, including scientific visualizations such as brain scans, or live visualizations such as stock market monitoring; the work discussed through this symposium is of the type used in science, business, and medical settings to display data abstractly.
Download PDF of paper here
Yu, J., Landy, D., & Goldstone, R. L. (2018). Visual flexibility in arithmetic expressions. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 2750-2755). Madison, Wisconsin: Cognitive Science Society.
We investigated whether, and in what, ways people use visual structures to evaluate mathematical expressions. We also explored the relationship between strategy use and other common measures in mathematics education. Participants organized long sum/products when visual structure was available in algebraic expressions. Two experiments showed a similar pattern: One group of participants primarily calculated from left to right, or combined identical numbers together. A second group calculated adjacent pairs. A third group tended to group terms which either produced easy sums (e.g., 6+4), or participated in a global structure. These different strategies were associated with different levels of success on the task, and, in Experiment 2, with differential math anxiety and mathematical skill. Specifically, problem solvers with lower math anxiety and higher math ability tend to group by chunks and easy calculation. These results identify an important role for the perception of coherent structure and pattern identification in mathematical reasoning.
Download PDF of article here
Yu, J., Goldstone, R. L., & Landy, D. (2018). Experientially grounded learning about the roles of variability, sample size, and difference between means in statistical reasoning. Proceedings of the 40th Annual Conference of the Cognitive Science Society. (pp. 2744-2749). Madison, Wisconsin: Cognitive Science Society.
Despite its omnipresence in this information-laden society, statistics is hard. The present study explored the applicability of a grounded cognition approach to learning basic statistical concepts. Participants in 2 experiments interacted with perceptually rich computer simulations designed to foster understanding of the relations between fundamental statistical concepts and to promote the ability to reason with statistics. During training, participants were asked to estimate the probability of two samples coming from the same population, with sample size, variability, and difference between means independently manipulated. The amount of learning during training was measured by the difference between participants’ confidence judgments and those of an Ideal Observer. The amount of transfer was assessed by the increase in accuracy from a pretest to a posttest. Learning and transfer were observed when tailored guidance was given along with the perceptually salient properties. Implications of our quantitative measures of human sensitivity to statistical concepts were discussed.
Download PDF of article
Motz, B.A., de Leeuw, J.R., Carvalho, P.F., Liang, K.L., Goldstone, R.L. (2017). A dissociation between engagement and learning: Enthusiastic instructions fail to reliably improve performance on a memory task. PLoS ONE, 12(7): e0181775. doi: 10.1371/journal.pone.0181775
Despite widespread assertions that enthusiasm is an important quality of effective teaching, empirical research on the effect of enthusiasm on learning and memory is mixed and largely inconclusive. To help resolve these inconsistencies, we conducted a carefully-controlled laboratory experiment, investigating whether enthusiastic instructions for a memory task would improve recall accuracy. Scripted videos, either enthusiastic or neutral, were used to manipulate the delivery of task instructions. We also manipulated the sequence of learning items, replicating the spacing effect, a known cognitive technique for memory improvement. Although spaced study reliably improved test performance, we found no reliable effect of enthusiasm on memory performance across two experiments. We did, however, find that enthusiastic instructions caused participants to respond to more item prompts, leaving fewer test questions blank, an outcome typically associated with increased task motivation. We find no support for the popular claim that enthusiastic instruction will improve learning, although it may still improve engagement. This dissociation between motivation and learning is dis- cussed, as well as its implications for education and future research on student learning.
Download PDF of article
Goldstone, R. L., Kersten, A., & Carvalho, P. F. (2017). Categorization and Concepts. In J. Wixted (Ed.) Stevens’ Handbook of Experimental Psychology and Cognitive neuroscience, Fourth Edition, Volume Three: Language & Thought. New Jersey: Wiley. (pp. 275-317).
Concepts are the building blocks of thought. They are critically involved when we reason, make inferences, and try to generalize our previous experiences to new situations. Behind every word in every language lies a concept, although there are concepts, like the small plastic tubes attached to the ends of shoelaces, that we are familiar with and can think about even if we do not know that they are called aglets . Concepts are indispensable to human cognition because they take the “blooming, buzzing confusion” (James, 1890, p. 488) of disorganized sensory experiences and establish order through mental categories. These mental categories allow us to make sense of the world and predict how worldly entities will behave. We see, hear, interpret, remember, understand, and talk about our world through our concepts, and so it is worthy of reflection time to establish where concepts come from, how they work, and how they can best be learned and deployed to suit our cognitive needs.
download PDF of article
Marghetis, T., Goldstone, R. L., & Landy, D. (2017). Even when people are manipulating algebraic equations, they still associate numerical magnitude with space. Proceedings of the 39th Annual Conference of the Cognitive Science Society. (pp. 2675-2680). London, England: Cognitive Science Society.
The development of symbolic algebra transformed civilization. Since algebra is a recent cultural invention, however, algebraic reasoning must build on a foundation of more basic capacities. Past work suggests that spatial representations of number may be part of that foundation, but recent studies have failed to find relations between spatial-numerical associations and higher mathematical skills. One possible explanation of this failure is that spatial representations of number are not activated during complex mathematics. We tested this possibility by collecting dense behavioral recordings while participants manipulated equations. When interacting with an equation’s greatest [/least] number, participants’ movements were deflected upward [/downward] and rightward [/leftward]. This occurred even when the task was purely algebraic and could thus be solved without attending to magnitude (although the deflection was reduced). This is the first evidence that spatial representations of number are activated during algebra. Algebraic reasoning may require coordinating a variety of spatial processes.
Download PDF of article
Carvalho, P. F., & Goldstone, R. L. (2017). The most efficient sequence of study depends on the type of test. Proceedings of the 39th Annual Conference of the Cognitive Science Society. (pp. 198-203). London, England: Cognitive Science Society.
Previous research has shown that the sequence in which concepts are studied changes how well they are learned. In a series of experiments featuring naturalistic concepts (psychology concepts) and naïve learners, we extend previous research by showing that the sequence of study changes the representation the learner creates of the study materials. Interleaved study leads to the creation of relatively interrelated concepts that are represented by contrast to each other and based on discriminating properties. Blocked study, instead, leads to the creation of relatively isolated concepts that are represented in terms of their central and characteristic properties. The relative benefits of these representations depend on whether the test of conceptual knowledge requires contrastive or characteristic information. These results argue for the integrated investigation of the benefits of different sequences of study as depending on the characteristics of the study and testing situation as a whole.
Download PDF of paper
Lara-Dammer, F., Hofstadter, D. R., & Goldstone, R. L. (2017). A computer model of context dependent perception in a very simple world. Journal of Experimental & Theoretical Artificial Intelligence, 29:6, 1247-1282. DOI: 10.1080/0952813X.2017.1328463
We propose the foundations of a computer model of scientic discovery that takes into account certain psychological aspects of human observation of the world. To this end, we simulate two main components of such a system. The first is a dynamic microworld in which physical events take place, and the second is an observer that visually perceives entities and events in the microworld. For reason of space, this paper focuses only on the starting phase of discovery, which is the relatively simple visual inputs of objects and collisions.
Download PDF of article
Tullis, J. G., & Goldstone, R. L. (2017). Instruction in computer modeling can support broad application of complex systems knowledge. Frontiers in Education, 2:4, 1-18. doi: 10.3389/feduc.2017.00004
Learners often struggle to grasp the important, central principles of complex systems, which describe how interactions between individual agents can produce complex, aggre-gate-level patterns. Learners have even more difficulty transferring their understanding of these principles across superficially dissimilar instantiations of the principles. Here, we provide evidence that teaching high school students an agent-based modeling language can enable students to apply complex system principles across superficially different domains. We measured student performance on a complex systems assessment before and after 1 week training in how to program models using NetLogo (Wilensky, 1999a). Instruction in NetLogo helped two classes of high school students apply complex sys-tems principles to a broad array of phenomena not previously encountered. We argue that teaching an agent-based computational modeling language effectively combines the benefits of explicitly defining the abstract principles underlying agent-level interac-tions with the advantages of concretely grounding knowledge through interactions with agent-based models.
Download PDF of article
Meagher, B. J., Carvalho, P. F., Goldstone, R. L., & Nosofsky, R. M. (2017). Organized simultaneous displays facilitate learning of complex natural science categories. Psychonomic Bulletin & Review, DOI 10.3758/s13423-017-1251-6.
Subjects learned to classify images of rocks into the categories igneous, metamorphic, and sedimentary. In accord with the real-world structure of these categories, the to-beclassified rocks in the experiments had a dispersed similarity structure. Our central hypothesis was that learning of these complex categories would be improved through observational study of organized, simultaneous displays of the multiple rock tokens. In support of this hypothesis, a technique that included the presentation of the simultaneous displays during phases of the learning process yielded improved acquisition (Experiment 1) and generalization (Experiment 2) compared to methods that relied solely on sequential forms of study and testing. The technique appears to provide a good starting point for application of cognitive-psychology principles of effective category learning to the science classroom.
Download a PDF of this article
Carvalho, P. F., & Goldstone, R. L. (2017). The sequence of study changes what information is attended to, encoded and remembered during category learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 43, 1699-1719.
The sequence of study influences how we learn. Previous research has identified different sequences as potentially beneficial for learning in different contexts and with different materials. Here we investigate the mechanisms involved in inductive category learning that give rise to these sequencing effects. Across 3 experiments we show evidence that the sequence of study changes what information learners attend to during learning, what is encoded from the materials studied and, consequently, what is remembered from study. Interleaved study (alternating between presentation of 2 categories) leads to an attentional focus on properties that differ between successive items, leading to relatively better encoding and memory for item properties that discriminate between categories. Conversely, when learners study each category in a separate block (blocked study), learners encode relatively more strongly the characteristic features of the items, which may be the result of a strong attentional focus on sequential similarities. These results provide support for the sequential attention theory proposing that inductive category learning takes place through a process of sequential comparisons between the current and previous items. Different sequences of items change how attention is deployed depending on this basic process. Which sequence results in better or worse learning depends on the match between what is encoded and what is required at test.
Download a PDF of this article
Goldstone, R. L., & Day, S. (2017). Transfer of knowledge. In K. Peppler (Ed.) The SAGE Encyclopedia of Out-of-school Learning. Thousand Oaks, CA. (pp. 784-787).
Transfer of knowledge is the application of knowledge learned in one context to new, dissimilar problems or situations where the knowledge would be useful. Teachers, coaches, camp counselors, parents, and learners often have the experience of a learner showing apparent understanding when questioned about a topic in a way that closely matches how it was initially presented but showing almost no understanding when queried in a new context or with novel examples. This entry further explains the concept of knowledge transfer. It then discusses several different strategies used to support knowledge transfer.
Download a PDF of this article
Goldstone, R. L., Weitnauer, E., Ottmar, E., Marghetis, T., & Landy, D. H. (2016). Modeling Mathematical Reasoning as Trained Perception-Action Procedures. In R. Sottilare, A. Graesser, X. Hu, A. Olney, B. Nye, and A. Sinatra (Eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 4 – Domain Modeling. Orlando, FL: U.S. Army Research Laboratory. (pp. 213-223).
We have observed that when people engage in algebraic reasoning, they often perceptually and spatially transform algebraic notations directly rather than first converting the notation to an internal, non spatial representation. We describe empirical evidence for spatial transformations, such as spatially compact grouping, transposition, spatially overlaid intermediate results, cancelling out, swapping, and splitting. This research has led us to understand domain models in mathematics as the deployment of trained and strategically crafted perceptual-motor processes working on grounded and strategically crafted notations. This approach to domain modeling has also motivated us to develop and assess an algebra tutoring system focused on helping students train their perception and action systems to coordinate with each other and formal mathematics. Overall, our laboratory and classroom investigations emphasize the interplay between explicit mathematical understandings and implicit perception action training as having a high potential payoff for making learning more efficient, robust, and broadly applicable.
Download PDF of paper
Tullis, J. G. & Goldstone, R. L. (2016). Comparison versus reminding. Cognitive Research: Principles and Implications, 1, 1-20, DOI 10.1186/s41235-016-0028-1.
Comparison and reminding have both been shown to support learning and transfer. Comparison is thought to support transfer because it allows learners to disregard non-matching features of superficially different episodes in order to abstract the essential structure of concepts. Remindings promote memory for the individual episodes and generalization because they prompt learners to retrieve earlier episodes during the encoding of later related episodes and to compare across episodes. Across three experiments, we compared the consequences of comparison and reminding on memory and transfer. Participants studied a sequence of related, but superficially different, proverb pairs. In the comparison condition, participants saw proverb pairs presented together and compared their meaning. In the reminding condition, participants viewed proverbs one at a time and retrieved any prior studied proverb that shared the same deep meaning as the current proverb. Experiment 1 revealed that participants in the reminding condition recalled more proverbs than those in the comparison condition. Experiment 2 showed that the mnemonic benefits of reminding persisted over a one-week retention interval. Finally, in Experiment 3, we examined the ability of participants to generalize their remembered information to new items in a task that required participants to identify unstudied proverbs that shared the samemeaning as studied proverbs. Comparison led to worse discrimination between proverbs related to studied proverbs and proverbs unrelated to studied proverbs than reminding. Reminding supported better memory for individual instances and transfer to new situations than comparison.
Download PDF version of paper
Marghetis, T., Landy, D., & Goldstone, R. L. (2016). Mastering algebra retrains the visual system to perceive hierarchical structure in equations. Cognitive Research: Principles and Implications, 1(25), 1-10, DOI 10.1186/s41235-016-0020-9.
Formal mathematics is a paragon of abstractness. It thus seems natural to assume that the mathematical expert should rely more on symbolic or conceptual processes, and less on perception and action. We argue instead that mathematical proficiency relies on perceptual systems that have been retrained to implement mathematical skills. Specifically, we investigated whether the visual system—in particular, object-based attention—is retrained so that parsing algebraic expressions and evaluating algebraic validity are accomplished by visual processing. Object-based attention occurs when the visual system organizes the world into discrete objects, which then guide the deployment of attention. One classic signature of object-based attention is better perceptual discrimination within, rather than between, visual objects. The current study reports that object-based attention occurs not only for simple shapes but also for symbolic mathematical elements within algebraic expressions—but only among individuals who have mastered the hierarchical syntax of algebra. Moreover, among these individuals, increased object-based attention within algebraic expressions is associated with a better ability to evaluate algebraic validity. These results suggest that, in mastering the rules of algebra, people retrain their visual system to represent and evaluate abstract mathematical structure. We thus argue that algebraic expertise involves the regimentation and reuse of evolutionarily ancient perceptual processes. Our findings implicate the visual system as central to learning and reasoning in mathematics, leading us to favor educational approaches to mathematics and related STEM fields that encourage students to adapt, not abandon, their use of perception.
Download a PDF version of this paper
Carvalho, P.F., Braithwaite, D.W., de Leeuw, J.R., Motz, B.A., & Goldstone, R.L. (2016). An in vivo study of self-regulated study sequencing in introductory psychology courses. PLoS ONE 11(3): e0152115.
Study sequence can have a profound influence on learning. In this study we investigated how students decide to sequence their study in a naturalistic context and whether their choices result in improved learning. In the study reported here, 2061 undergraduate students enrolled in an Introductory Psychology course completed an online homework tutorial on measures of central tendency, a topic relevant to an exam that counted towards their grades. One group of students was enabled to choose their own study sequence during the tutorial (Self-Regulated group), while the other group of students studied the same materials in sequences chosen by other students (Yoked group). Students who chose their sequence of study showed a clear tendency to block their study by concept, and this tendency was positively associated with subsequent exam performance. In the Yoked group, study sequence had no effect on exam performance. These results suggest that despite findings that blocked study is maladaptive when assigned by an experimenter, it may actually be adaptive when chosen by the learner in a naturalistic context.
Download PDF of article here