Data Collection and Analysis

In order to determine whether a treatment or prevention program or policy is effective, program staff and evaluators collect and manage data during each step of planning and implementation, and after the program is over.

Different kinds of data are useful at different stages. For instance, process measures describe how the program was implemented, while outcome measures describe the effects of program efforts. All of these measures can help administrators strengthen, improve, and showcase the program.  Data collection and management are important steps to finding out what works and why in campus and community alcohol, other drug, and violence prevention.

There are a number of other considerations as you prepare to collect your data:

What should you measure?

Data needs to indicate what the problems are, what programming has been implemented, and what was accomplished. At each stage, your evaluator can provide important resources to help in this process.

For example, in the needs assessment stage before program implementation, evaluators may review:

  • Survey data about current levels of alcohol and other drug use and violent incidents on campus and in the surrounding community in order to determine needs and issues that are unique to that environment
  • Observational assessments of specific campus or community environments to determine contexts in which problems occur
  • Incident reports from campus and community sources to determine magnitude of problems, as well as where and when the problems occur

These data sources can assist you in determining which specific alcohol, other drug, and violence-related outcomes should be the focus of efforts. In addition, they can help you develop a logic model to outline the processes by which problems occur and the mechanisms through which these problems can be reduced.

During implementation, evaluators collect process measures such as:

  • Utilized resources (staff, volunteers, funds) to see how they compare with what was initially planned
  • Activity levels and the number of people or groups served by a program to determine if it reached the intended population

After implementation, evaluators analyze outcome measures including:

  • Changes in behavior, both intermediate (e.g., wait staff refusing to serve underage patrons) and long term (fewer students driving while intoxicated)
  • Environmental outcomes, such as observations of rowdy behavior by students at bars, parties, or on the street

Planning for human subject protection.

Institutions of higher education and funders may have specific requirements that prohibit data collection unless specific procedures are followed. Prior to collecting data, determine whether your institution has an institutional review board (IRB) that governs human subject protections. Nearly all universities have an IRB; state and community agencies may have their own IRB or may have an arrangement with another institution. In addition to following IRB guidelines, review general considerations for protection of human subjects and ensure that the study protocol is designed in accordance with these guidelines.

Getting help.

Evaluation is especially vital in an era of fiscal constraint. Long-term financial support for prevention work, whether it comes from outside funding sources or is part of a college's regular budget, will be available only if evaluation results warrant it. Evaluation is also important to help program planners revise their programs and policies so that they can focus their efforts on implementing the most effective strategies. Unfortunately, very few campus-based prevention programs have undertaken an evaluation that meets rigorous methodological standards, leaving them vulnerable to attack and fiscal cutbacks.

The difficulty is not that prevention coordinators are unaware of the need for evaluation, or that they are worried about their program failing to measure up. Rather, most coordinators do not feel equipped to conduct a rigorous evaluation.

The Center web site includes many resources that can assist coordinators in the process of designing and implementing evaluation efforts. Your campus may also have resources to assist you such as institutional research offices or faculty with expertise in evaluation issues. In some cases you may have resources to hire an evaluator. The National Center for Safe and Supportive Learning Environments provides technical assistance and training in areas including evaluation.

References

Methods for Assessing College Student Use of Alcohol and Other Drugs, Higher Education Center for Alcohol, Other Drug Abuse, and Violence Prevention, 2008.

 

Featured Resources

cover photo of the report - Methods for Assessing College Student Use of Alcohol and Other Drugs

Offers a straightforward method for gathering and reporting student survey data on substance use-related problems. Administrators must understand the nature and extent of these problems at their institutions in order to develop effective programs and policies to reduce alcohol- and other drug-related (AOD) problems on campus.

The Higher Education Center for Alcohol and Other Drug Abuse and Violence Prevention

Focuses on the growing problems of income and wealth disparity and social mobility, and the role higher education can play in economic prosperity. The article emphasizes the need for board members to understand what the data say about their institution's contributions to restoring opportunity in America and creating a culture that prioritizes student success.

JOSÉ LUIS SANTOS AND KATI HAYCOCK
Thumbnail cover image - Expectations Meet Reality: The Underprepared Student and Community Colleges

Explores the experiences of community college students in the areas of assessment, placement, and developmental education. The report critically examines the "broken state" of developmental education and the implications it has on the academic outcomes of underprepared students. As a recommendation to resolve this issue, the report encourages colleges to review their programs, measurement systems, and student data when developing strategies to help underprepared students successfully complete college-level work.

Center for Community College Student Engagement
Report cover - College Costs in Context A State-by-State Look at College (Un)Affordability

Analyzes state-level college affordability at two- and four-year public colleges, by focusing on the share of family income required to cover the net price paid by students at each income level. The report finds striking inequities in public college affordability, both within and across states. While college costs are high relative to family incomes for most students in most states, the lowest income students face the most extreme and unrealistic financial expectations. Sortable data by state and sector are also available for download.

Presents key data that show the continuing educational inequities and opportunity gaps for students of color and low-income students and highlights promising practices that many colleges are taking to advance success for students of all backgrounds.