Breaking Down Research Designs

After a less-than-stellar set of test grades on chapters 7 and 8, I decided to re-explain the characteristics of research design. The test question was:

What is a design? What are its components?

In chapter 2, we state (p. 44): "The design of a research project includes the number of groups, how they are treated, and how the behavior is measured. The dependent variable, the independent variable, levels of the independent variable, and how extraneous variables are controlled are all aspects of research design."

In chapter 7, we introduce between-subjects designs and the extraneous variables of selection, differential attrition, and diffusion of treatment. In chapter 8, we introduce within-subjects designs and the extraneous variables of testing, instrument change, history, maturation, and regression (to the mean).

Let's break down design even further.

RESEARCH Type

• Experiment
• Quasi-experiment
• Field Study
• Naturalistic Observation
• Participant Observation
• Case Study
• Interview
• Focus Group
• Oral History
• Archival Study
• Small N Study
• Other
  

NUMBER of Groups (N)

• One
• Two
• Three
• Four
• More

GROUPS

• Between Subjects
• Within Subjects
• Mixed

VARIABLES

• Independent (number, levels)
• Dependent (number, quantitative or qualitative, continuous or discrete)
• Extraneous (how controlled or not)

EXPLORATORY DATA ANALYSIS & STATISTICS

• Graphs
• Descriptive Statistics
• Correlations
  
• Confidence Intervals
• NHST Tests and Significance (e.g., t-test, ANOVA, chi-square)
  
• Nonparametric Tests (e.g., Mann-Whitney, Wilcoxon Matched-Pairs Signed Ranks, Spearman correlation coefficient)
  
• Effect Sizes

CONTROLLING EXTRANEOUS VARIABLES

• Via design
• Via procedures

ETHICS

As we note in chapter 3, any design that fails to properly follow the current ethical standards is, by definition, a bad design.

So, there are many aspects to research design. Careful researchers devote much time toward perfecting their design and then pilot testing it before collecting data for real.

Self-selection?

In chapter 7 we introduce the extraneous variable of selection. Self-selection is a form of selection that occurs when people choose to do one thing and not another. A good example is when high school students choose to drop out of school.

An uniformed or naive researcher might draw the wrong conclusion when presented standardized test scores of 9th graders and 10th graders if a large proportion of the students dropped out of school when they turned 16 years of age. Assume that the 16 year olds who dropped out had lower standardized test scores and also assume that the law required them to attend school until they turned 16. In that scenario, the 10th grade AVERAGE scores would rise because the scores of the students who dropped out would no longer be included.

However, if a researcher did not know that the 9th and 10th grades were different to start with because of self-selection, he or she might look for another explanation. Using this example as an introduction, look at this recent news story from the Boston Globe about self-selection by gender in scientific, engineering, and technical fields.

This article examines a controversial area in contemporary culture, the relative imbalance of males over females in scientific, engineering, and technical fields. This imbalance may have many causes. As a recent New York Times article noted, gender discrimination and sexual harassment (both quid pro quo and hostile environment) are much more common in scientific, engineering, and technical fields.

The Boston Globe story, however, highlights the research of an economist, Joshua Rosenbloom of the University of Kansas. He decided to study men and women in the area of information technology. He and his co-authors found that men and women who enjoyed working with tools or machines were the most likely to choose a career in information technology. Also, people who enjoyed working with people turned out to be less likely to choose a career in information technology. His research, with co-authors Ronald A. Ash, Brandon Dupont, and LeAnne Coder was published in the Journal of Economic Psychology. They concluded that self-selection was the main reason for the paucity of women in information technology positions.

Of course, self-selection is not the only factor operating here. The interesting new finding is that self-selection may play a large role than previously imagined in the career choices of men and women. Also mentioned in the Boston Globe article is the research of Susan Pinker. She has found that in societies with the greatest amount of personal freedom, women opt for jobs not typically held by men, thus contributing to the disparity in occupational choice. Paradoxically, the proportions of women working such jobs is highest in countries with lower levels of individual freedom, and yes, in those societies the disparity in the jobs between men and women is less. Here is a link to a New York Times review of her book: The Sexual Paradox: Men, Women, and the Real Gender Gap.

Gender studies are a fertile field for social scientists and present them with many opportunities for conducting research and for teasing out the interactions between multiple sources of causation. However, some of those sources may be due to internal factors such as personal preference.

Epidemiology's Methods

Several recent articles have highlighted how epidemiologists conduct research and some have criticized their results.

An LA Times article discusses some of these results, methods, and criticisms. Some of the past results have been 180 degrees apart. Look at the results they cite about coffee and its effects. In 1981, a study concluded that two or three cups of coffee a day tripled the risk of pancreatic cancer. In 2001, another, larger study concluded that the earlier results were not true. Similarly, a 1981 study found that drinking coffee reduced the risk of colorectal cancer. Predictably, a later study in 2005 contradicted it.

Most epidemiological studies are observational. As we discuss in chapter 10, observational studies (e.g., naturalistic observation and participant observation) do not alter the situation under observation. Other ways to conduct epidemiological research is through cohort studies, case control studies (or retrospective), and cross-sectional studies.

The big advantage of epidemiological studies is that they are relatively cheap to conduct. However, they do not allow researchers to make cause-and-effect conclusions about the variables under investigation.

Much better results are provided by randomized clinical trials (or what we call true experiments). Randomized clinical trials are the medical version of true experiments. In a clinical trial, two (or more) groups are created by a random procedure.

Another LA Times article highlights some of the recent research discrepancies discovered by observational methods vs. clinical trials.

A long New York Times article also covers the issues inherent in observational research. That article concludes:

• "All of this suggests that the best advice is to keep in mind the law of unintended consequences. The reason clinicians test drugs with randomized trials is to establish whether the hoped-for benefits are real and, if so, whether there are unforeseen side effects that may outweigh the benefits. If the implication of an epidemiologist's study is that some drug or diet will bring us improved prosperity and health, then wonder about the unforeseen consequences. In these cases, it's never a bad idea to remain skeptical until someone spends the time and money to do randomized trial and, contrary to much of the history of the endeavor to date, fails to refute it."

Our final word echoes our advice in chapter 1, learning about methods is a good thing.

Rattus norvegicus

I have fond memories of running rats nearly 35 years ago as an undergraduate. Back then, I helped maintain the rat and pigeon colony, even though the rats made me sneeze. I had to wear a mask to work in there. That space is no more and is now an administrator's office. Today's undergraduates are not likely to remove a laboratory rat from its cage, "gentle" it every day, design animal experiments, and run rats.

When I first came to Southern Arkansas University, there was a small, temporary, rat lab in the department. It was set up every spring semester and dismantled after graduation. One day I noticed small beetles flying in the hall, Tribolium confusum, or confused flour beetles is what they were. They were feeding on the rat chow. Since then, we have had no rats. Can't say I miss them however.

However, rats are still in the picture, research-wise. A New York Times article reviewed some recent rat research including research in dreaming, drug use, and sexual behavior. Some rats even show evidence of meta-cognition (see Jonathon D. Crystal's home page), or knowing what they know.

So the domestic Norway rat, Rattus norvegicus, is still around. They are just harder to find than they used to be.