Evaluation
What is program evaluation?
Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their organizational environments which are designed to inform social action in ways that improve social conditions (Rossi, Lipsey, & Freeman, 2004).
Why perform program evaluation?
- To ensure program goals & objectives are being met.
- To determine if the allocated resources are yielding the greatest benefit.
- To identify what works well, what does not, and why.
- To identify the areas of Afternoons R.O.C.K. in Indiana that need improvement.
- To inform management decisions regarding the program.
- To help develop future policies related to Afternoons R.O.C.K. in Indiana.
- To report on progress and communicate Afternoons R.O.C.K. in Indiana's value.
- To comply with Government Performance and Results Act.
How do we measure the success of Afternoons R.O.C.K. in Indiana?
The success of Afternoons R.O.C.K. in Indiana is measured by the program objectives. There are three types of objectives that are used to assess program effectiveness, including process, impact, and outcome objectives.
The process objectives specify the manner in which the program is intended to be carried out. Process objectives measure whether the program was carried out as intended and is indicated by the number of youth the program intends to serve. The number of youth that were actually served by the program is gathered from the rosters submitted to the IPRC.
It is expected that by June 30, 2010, the collaborating member agencies of this coalition shall:
- implement at least one DPSP in each county in the DSA.
- spend down all funds available for serving youth.
- at least 75% of the participating youth will attend all program sessions.
- at least 80% of all participants will complete all program requirements.
The impact objectives measure the effect of the program on drug-use among participants. Impact objectives indicate the projected prevalence of alcohol, tobacco, marijuana, and inhalant use after participation in the Afternoons R.O.C.K. in Indiana program.
At the conclusion of the program, it is expected that use will be lower than in previous years. Specially, the prevalence of reported use in the previous month:
- cigarette use by participating youth shall not exceed 5%.
- smokeless tobacco use by participating youth shall not exceed 2%.
- alcohol use by participating youth shall not exceed 8%.
- binge drinking use by participating youth shall not exceed 2%.
- marijuana use by participating youth shall not exceed 3%.
- cocaine use by participating youth shall not exceed 1%.
- stimulant use by participating youth shall not exceed 1%.
The outcome objectives describe the expected skills/knowledge or behavior that the youth will gain from participation in the program. Outcome objectives measure the change in expected skills, knowledge, beliefs, or perceptions among program participants. Outcomes of the program relate to the perceived risk and peer disapproval of drug use. Both the impact and outcome objectives included in the Program Profile specify the intended effect of the program on participants.
At the conclusion of the program, it is expected that the perceived risk associated with:
- cigarette use shall exceed 82%.
- monthly marijuana use shall exceed 81%.
- regular marijuana use shall exceed 82%.
- occasional cocaine use shall exceed 81%.
- regular cocaine use shall exceed 81%.
- alcohol use shall exceed 75%.
- weekend binge drinking use shall exceed 81%.
- daily binge drinking use shall exceed 82%.
Outcome information is gathered from anonymous survey forms administered to program participants upon enrollment (pre-survey) and completion (post-survey).
What information does Afternoons R.O.C.K. in Indiana collect?
All Substance Abuse and Mental Health Services Administration (SAMHSA) grantees are required to collect and report certain data so that SAMHSA can meet its obligations under the Government Performance and Results Act (GPRA). GPRA, a public law passed by the U.S. Congress in 1993, was enacted to improve stewardship in the Federal government and to link resources and management decisions with program performance. The National Outcome Measures (NOMs) were developed by SAMHSA as a set of domains and measures that focus specifically on its programs. The NOMS require domain-specific information that is collected on the Afternoons R.O.C.K. survey data forms.
The Afternoons R.O.C.K. survey data forms (pre-survey & post-survey) collects specific information from youth in the following categories:
- Age of First Use
- 30-day prevalence
- Perceived risk
- Perception of peer approval
- Frequency of gambling behavior
- Risky behavior (riding with someone under the influence)
- Intention to use alcohol, tobacco, and other drugs
How is fidelity assured in Afternoons R.O.C.K. in Indiana programs?
In order to assure the effectiveness of the implementation of evidence-based prevention curricula in Afternoons R.O.C.K. in Indiana programs, the Division of Mental Health and Addiction has added several items from the evidence-based programs’ fidelity instruments to the program monitoring instruments. Program directors and program supervisors must complete the fidelity instruments which have program monitoring questions for each program cohort.
