3.1. Research Question 1
In research question 1, we asked, “How do students perform on an SHI systems thinking modeling and writing assignment?” Statistical analyses were conducted using mean scores on students’ drawn models and newspaper articles across all 3 years. For students’ drawn model scores, there was a significant effect of model category on overall model score at the
p < 0.05 level (
F(2, 384) = 91.67,
p < 0.05). Post hoc comparisons using Tukey’s honestly significant difference (HSD) test indicated that the mean score for components was significantly higher than the mean score for mechanisms, which was also higher than the mean score for phenomenon/patterns (
Table 4) (see
Appendix B). These results suggest that students included more components than mechanisms or patterns in their drawn models of the system. The model category, mechanisms, correlates with, components (
r(127) = 0.24,
p < 0.05), but not phenomenon/patterns. This observation indicates that as students included more mechanisms in their models, the quantity of components increased in their drawn models as well.
Statistical analyses were also conducted using the written systems thinking newspaper article scores. There was a significant effect of article category on overall model score (
F(5, 768) = 401.6,
p < 0.05). Results show that students scored the highest on problem identification from their written newspaper article and scored the lowest on their description of unintended consequences. Post hoc comparisons using Tukey’s HSD test indicated that the mean score for problem identification was significantly higher than all of the other categories (
Table 5) (see
Appendix B). Although the category of implementation challenges is not significantly different from limitations or stakeholder awareness, students scored higher on it than on unintended consequences, indicating that students were best at articulating the problem within the system and least proficient in describing the unintended consequences of the system. Although stakeholder awareness and model limitations also represented areas of improvement for students, model limitations was distinct because it was correlated with all of the categories (stakeholder awareness,
r(127) = 0.178,
p < 0.05; unintended consequences,
r(127) = 0.422,
p < 0.05; implementation challenges,
r(127) = 0.0543,
p < 0.05) except problem identification. Overall, these findings indicated that as students incorporate more ideas about model limitations, their overall article score increases.
3.2. Research Question 2
For research question 2, we asked, “To what extent is the systems thinking model score predictive of the writing assignment score on a sociohydrologic issue?” Written article, model scores, and cumulative systems thinking assignment scores for each year were also compared to one another to gain further insight into the relationships between the two systems thinking tasks. A regression analysis and analysis of variance (ANOVA) were performed, results of which suggest that students who score better on the drawn model also perform better on the written article (
t(125) = 6.60,
p = 0.01,
η2 = 0.88) (
Figure 3). We also analyzed the effect of year on total systems thinking score, which is the drawn model and written article combined, through a regression and an ANOVA analysis (
r(125) = 3.19,
p = 0.04,
η2 = 0.57;
F(2, 126) = 19.8,
p < 0.001). Both analyses indicate that there were statistical differences between total systems thinking scores for each year of the course (see
Appendix C). An ANOVA of the effect of year on the total systems thinking score revealed that regression lines of expected scores overlain with observed scores for each year demonstrate the slope remaining constant for varying intercepts for each year (
β = 0.058). The way we approached the year was taking this as a blocking effect. This allows us to assume and model that the years are acting differently.
The systems thinking scores across years were significantly different from each other. The higher total model and total article scores were all from 2019, whereas the lower total model and total article scores were from both 2017 and 2018. These outstanding points could have resulted from changes made to other course components and overall differences between student populations from year to year. However, the overall regression for model effect was greater than that for year effect on the systems thinking score. This allowed us to end up with a model including a year effect. Where the intercept starts was different because some years were naturally more variable, and the slope remained the same for the total systems thinking score for each year. Overall scores differed between years, but the relationship between the drawn model and written article scores did not. The fundamental relationship was the same no matter where they started or ended.
3.3. Research Question 3
For research question 3, we asked, “How do students evaluate their own systems thinking models of a real-world sociohydrologic issue?” This qualitative data served to augment the quantitative results from research questions 1 and 2. A positive correlation existed between the limitations score and the overall written assignment score (r(127) = 0.71, p < 0.001; F(1, 128) = 7.51, p < 0.05). Correlations were neither found between the limitations written assignment score and overall model score, nor the individual scores for mechanisms, components, and phenomenon/patterns. Students who included a more robust discussion of limitations also performed better on the overall written assignment. Out of the 129 students who completed the systems thinking assignment, 22% failed to include a discussion of any limitations of their drawn model. Of the students who did discuss a drawn model limitation, following analyses, three themes emerged: scope/scale limitations; temporal limitations; and specific components, mechanisms, or phenomena excluded.
3.3.1. Scope and Scale Limitations
First, analyses revealed responses categorized as those having to do with the limitations of the capacity to deal with concepts such the limits to the assignment itself, limited available information, or a limited level of specificity. Students commented on the limitations inherent within the assignment itself, including ideas such as the physical space the assignment uses, the quantity of factors, and the ability to effectively communicate their ideas about a “wicked problem”. For example, one student responded about these types of limitations, writing, “Part of the issue of showing all data is that there can never be enough space to show connections without it becoming incredibly confusing to understand and intricate” (ST_55). Other students echoed this message of scope and sale limitations by writing, “It does not show all aspects of this issue, it only shows the ones that are easy to portray” (ST_9). Similarly, a student wrote that, “The model would have to be expanded tenfold to be able to incorporate all of the human interactions in this system” (ST_6). Students felt that they were not able to effectively discuss all of the influences and aspects of the Raccoon River Water Crisis without compromising the intelligibility of their drawn models. Sometimes students combined multiple ideas into one response such as, “The limit of the model is that there are so many components involved and the model does not clearly explain the how much each party contribute” (ST_129). This response demonstrates both the concepts of scope and scale—the idea of scope as a nearly infinite quantity of components that they would need to include in their model for it to be accurate. The idea of scale is also alluded to; some components had larger impacts than others within the system, which this student noted was not defined within the model. For the model, students were not specifically asked to prioritize components, mechanisms, or phenomena. Similarly, some components of the system remained unmeasured or undocumented (e.g., microplastics), further limiting the overall scope and scale of the model.
However, some felt that they did not have all of the information they needed to effectively convey the scope and scale of the Raccoon River Water Crisis system. For example, one student responded to model limitations by stating, “I think the systems thinking model is limited just because of all the ‘hidden’ things that haven’t been in the news articles” (ST_130). This acknowledges that there are components that are missing from their available information sources, which could have contributed to their model’s accuracy. Another student described a lack of quantitative data as a limiting factor of their model, “I was limited due the fact that there are no numbers that shows how one component affect the other” (ST_124). This response indicates that the level of precision of their model was hampered by the lack of quantitative data available. This level of specificity as a scope and scale limitation was less common in student responses. However, several students commented on scope and scale specificity limitations in reference to names and overall dynamics.
Some students explored the idea of scope and scale specificity through their discussion of limitations related to grain size. One student listed a generalized statement of limited scope and scale by writing, “Broadly, farmers, wildlife, government and environmental groups are not specific. They are listed as large groups although there are probably many different opinions and perspectives within these groups” (ST_96). This type of limitation demonstrates that although the student chose not to break down groups into subgroups, they acknowledged that in doing so, their model may be misleading. A student spoke to this idea as using the model for approximating the scenario without including every specific detail available. They wrote,
The model we use to estimate what is going on is likely to be limited to not putting into consideration every little factor that is involved in this process and it is likely to make assumptions about some processes involved but it is going to help us with estimating what is going on with the river and its system.
(ST_61).
This type of response indicates that even though the models were limited in scope and scale, as well as the fact that some of the details were glossed over, the models were still valuable as proxies for the scenario overall.
3.3.2. Temporal Limitations
Temporal limitations were primarily described as those having to do with not knowing what will happen in the future with the system. In a written response containing a temporal limitation, one student said,
I think that the system model gives more of a past and present description instead of the future description and although that’s good, I think it would be even better if the future was also deeply analyzed because it would help in determining the rate at which the problem needs solved.
(ST_63)
This response indicated that students were aware of the past, present, and future dimensions of a system and acknowledge that their models are limited without the future possibilities. A few students spoke to future possibilities as limiting factors within their models. A student with this type of response wrote, “It may take years of research to learn what species got affected by the algae in the river, and what health effects it had on people” (ST_45). Responses like this one demonstrate that without the ability to either know or predict future effects of the Raccoon River water crisis on different parts of the system, models will be limited to past and present data, which may not encompass all of the system changes, including specific components, mechanisms, or phenomena.
3.3.3. Specific Components, Mechanisms, or Phenomena Excluded
Most often, in their discussion of model limitations, students listed a specific component/mechanism/phenomenon that was missing from their drawn model. The most common of these three categories was specific components that were excluded from the model. For example, one student wrote, “I find limitation with the way that there is not shown part of the city population in contaminating the rivers, it seems like all blame is for the farmers who use fertilizers on their farms” (ST_139). Student responses such as this indicate that they realized their models were limited in the specific perspectives included. Other students shared similar sentiments, stating that their models were limited in the lack of farmer perspectives included. Another specific component students cited as missing from their interpretation was monetary values. A student responded to the model limitations by writing, “My model does not show economic struggles of the area and how the money in this city is currently being used” (ST_8). This student demonstrated awareness of the importance of money in finding a solution, but also the effect that lack of money can have on different stakeholders. Similarly, a student wrote that, “It doesn’t include all the possible solutions, or the specific amount of money that’s been put towards fixing the crisis” (ST_90). Responses such as this indicate that students were aware of prior solutions and expenses and that there could be other solutions that have not been tried. Often, student responses had a dimension of more than one type of limitation.
Overall, students described fewer mechanisms as missing from their systems thinking models. The students that did include a mechanism as missing from their model largely focused on two processes—economics and environmental processes. One student writing about economic processes missing from their model wrote,
It also doesn’t show the complex economic processes. Companies in Des Moines help farmers with tractors and agribusiness and sales and this causes a growth in the population of Des Moines. People work on large farms that contribute to the Des Moines economy and grows Des Moines further. This kind of large scale economic and industrial feedback is very intricate…
(ST_6)
Students writing about detailed processes such as this exhibited a robust understanding of the problem’s social and scientific components. Students who wrote about environmental processes as a limitation of their model also included ideas about socioscientific components, “The graph also doesn’t specify how the water may flow, even through the ground, reaching other areas that aren’t polluting or receiving benefits from the state” (ST_113). Students incorporating knowledge from across the semester of hydrologic and human interactions demonstrated their depth of learning and attainment of course learning goals.
Phenomena or patterns were also identified as specific model limitations that were discussed in the written newspaper articles. The majority of responses in this category of limitation surrounded the idea of polluted water flowing from the Raccoon River to the dead zone in the Gulf of Mexico and harming wildlife. One student wrote about all of these ideas in summary by stating,
The model is missing the dead zone and the environmental portion of the issue. To make the model better, it would have to include these environmental effects. Including the animal species and the systems that function in that environment. Another way to make this model stronger, would be to add the communities that would also be affected in the Gulf.
(ST_121)
Students demonstrate their ability to view the contribution of one geographic area to the degradation of another. Another student wrote, “… but it does little to show the far-reaching effects of this problem as a whole. Nitrates from these and other fields around the United States pollute the Gulf of Mexico, and countless other waterways” (ST_25). This response took the idea of phenomena generalizability to a higher level by describing how the model was limited by leaving out this aspect and including the idea that this is happening in other parts of the country and affecting other waterways.