6/13/10

How Should Complex Systems Be Tested?

This post extends a conversation I began on 19 May 2010, titled testing the validity of complex systems. This is the fifth post on this topic since then.

The argument I am making is that researchers need to do some sort of complete (holistic) test of their topic, to: (1) make sure that the definition of a complex system they are using applies; and(2) make sure that their topic fits this definition.

The question I want to address here is how should such holistic testing be done?

Again, this will take a bit of blogging, but it seems to me that testing can be thought of at two basic levels.

1. Deep/Thorough Testing: The first and most rigorous level would require one or more studies devoted to a sort of deep or thorough testing to determine if one's definition of a complex system applies to a give topic and, related, if that topic can be validly and reliably called a complex system.

This first type of testing is the focus of the community health science study I am doing with my colleague, Galen Buckwlater. For the last couple years, researchers have been explicitly or implicitly treating communities and their health as if these things are complex social systems. Our research question is: is such an assumption valid and reliable? In other words, can one assume that the commonly used definition of a complex system applies to the study of communities and their health and, conversely, can communities and their health be called a complex system?

To conduct this type of test, we did the following. (A) First, we reviewed the literature to determine what the common definition of a complex system is that researchers use. (B) Next, we found a case study that represented the average community researchers typically study and collected data on it. (C) Then, we took each descriptor from the common definition of health and ran a series of tests. For example, a commonly held assumption is that communities are self-organizing. To determine if this is true, we examined if the conception of self-organizing used by these researchers to determine exactly what they mean by this concept. Then, we empirically tested this concept of self-organization to see if our community actually engaged in this behavior. In total, we ran ten individual testsn on the commonly used definition of complex system used in the community health science literature. It was a tremendous amount of work. And, in the process we used a wide arsenal of techniques, including hierarchical regression, curvilinear regression, correlation, k-means cluter analysis, the self-organizing map neural net algorithm, network analysis, qualitative case-based comparative method and computational (agent-based) modeling.

One can think of this first type of testing as helping a field along by increasing the rigor of its concepts and its knowledge of the type of complex system it it studying.


2. Shallow/Preliminary Testing. The second type of testing is what we might expect all researchers to do before and during the process of modeling a particular topic as a complex system. In this case, one would begin by explicitly outlining the particular definition of a complex system one is using. Then, one would conduct some type of preliminary tests to determine if one's topic is, indeed, a complex system.

The testing in this second case is likewise rigorous but it is more background work. Also, it is something that takes place before and during the model building process. The quality of one's results is something that is reported in the methods section of a study.

I have used this type of testing in a couple studies we have done. The first one was my research with Fred Hafferty on medical professionalism and the second was the book on sociology and complexity science that I wrote with Fred as well. In both instances we articulated the definition of a complex system we were using and tested to see if our topic fit it reasonably well.

This second type of testing involves the development of what we call a meta-model, and it is one of the first steps in the SACS Toolkit modeling process--this is the new method Fred and I developed for studying complex systems. SACS stands for sociology and complexity science. For more about our method, see our BOOK

Developing a meta-model (a model of one's model) allows researchers to determine, right from the beginning, if their definition of a complex system is rigorous and if their topic is (empirically speaking) a complex system. In addition to the development of a meta-model, the SACS Toolkit has a total of nine built-in procedures that researchers are expected to use to explore their definition and topic in complex systems terms. My brother John and I are writing a paper on how the SACS Toolkit does this and will be presenting it this summer in Sweden at the International Sociological Association Meetings. I should be done with the paper in the next couple weeks and will post it on here. I also plan to blog more about the SACS Toolkit so that readers can get a better sense of the method.

No comments:

Post a Comment