National Evaluation
Team for the Comprehensive Community Mental Health Services Program for
Children and Their Families
(ORC Macro)
Background
ORC Macro is a research, management consulting, and information technology firm
supporting business and governments worldwide. It provides survey and market
research, policy analysis and evaluation, performance improvement, training,
and technology support.
In addition to providing funding to communities, through grants and cooperative
agreements for children’s mental health services, the Center for Mental Health
Services (CMHS) contracted with ORC Macro to conduct a national multi-site
evaluation of the implementation of systems of care in these communities.
The evaluation has several goals:
-
Describe the children and families served by the CMHS-funded systems of care.
-
Determine the nature and extent of clinical and functional outcomes for
children and families served in systems of care.
-
Examine how children and families experience the service system and how they
use services and supports (i.e., utilization patterns).
-
Estimate the cost of serving children in systems of care.
-
Examine the development of systems of care as they move toward offering
integrated and comprehensive services.
-
Assess the effectiveness of the system-of-care approach as compared to usual
service delivery approaches.
-
Assess the effectiveness of evidence-based treatments within systems of care.
-
Support technical assistance activities to help CMHS best meet program goals.
Description of Activities
Study Evaluation
The budget of each funded community includes positions for two full-time local
evaluators. These evaluators collect data describing children and families,
services these families receive, and the outcomes of those services. The data
are collected as specified by the guidelines of the National Evaluation.
As the National Evaluation Team, ORC Macro is responsible for training local
evaluators in national evaluation data collection and submission procedures;
analyzing the data; interpreting the results; producing annual reports to
Congress and other written reports and products on the progress, results, and
lessons learned from the implementation of system-of-care programs. In
addition, continuous updates are given to CMHS on the status of the project,
and funded communities are given individual reports with analyses of their
data. Phase I of the evaluation began in 1994 and included the 22 communities
that began their grant funding in 1993 and 1994. Phase II of the evaluation
began in 1997 and included 9 communities that began their funding in 1997 and
the 14 communities that received initial grant funding in 1998. Phase III of
the evaluation includes the 22 communities that began their grant funding in
1999–2000. Phase IV of the evaluation includes the 18 communities that were
awarded cooperative agreements in 2002, 7 communities that were awarded
cooperative agreements in 2003, and four communities that were awarded
cooperative agreements in 2004.
In addition, the National Evaluation Team collects system-level data that
includes information on the development of interagency infrastructure,
strength-based services, family involvement, and culturally competent services
in each funded community. National Evaluation staff work with local communities
to obtain service and cost data from billing and management information systems
to examine expenditures for services and shifts in service costs. The Staff
also conducts a number of special studies to assess thing such as the
sustainability of systems of care, effectiveness of evidence-based practices,
and the role primary care physicians play in systems of care.
Evaluation Design
The core evaluation design includes five components:
Descriptive Data Collection
The descriptive data profile children and families who enter systems of care.
Descriptive data are collected on all children and include demographic
characteristics, social/functional characteristics, and mental health
diagnoses. The demographic data include characteristics such as age, ethnicity,
sex, family income, and family living situation. The social/functional data
include information such as educational performance, juvenile justice history,
previous psychiatric problems, and abuse history. The distinctions and
similarities among children based on their descriptive data may help explain
variations in the treatment experiences of children and families.
Outcome Data Collection
The outcome data trace changes in children’s mental health across time. Outcome
data are collected on a sample of children in each funded community at intake
into the system of care and every 6 months thereafter. Outcome data include
assessments of children’s emotional and behavioral status, strengths,
educational performance, children’s living environments, families’ involvement
in their children’s treatment, and child and family satisfaction with the
services.
Cost and Service Data Collection
Cost and service data describe the types of services used by children and
families in the system of care and the costs associated with these services.
The evaluation collects individual data from community management information
systems on children’s use of services and their costs in the system of care.
The analyses of cost and service data highlight the mixture of services
delivered by systems of care and the estimated average costs of treating a
child.
System-of-Care Assessment Data Collection
System-of-care assessment data describe the organization of the child-serving
system, how it delivers services, and how it affects the mental health outcomes
of children and families. Every 18 months, site visits are conducted to each
funded community to determine whether any developmental changes have taken
place at the organizational and service delivery levels. A system-of-care
community’s level of development is determined by measuring system attributes
during every site visit through record reviews, interviews, and observations.
Examples of the attributes include service availability, cultural competence of
services, family involvement, and interagency structure.
Comparison and Treatment Effectiveness Studies
A significant aspect of the national evaluation is that it includes comparisons
between those children and families served by CMHS-funded systems of care and
those served by usual service delivery systems. The comparison study component
of the national evaluation was initiated to address a critical question: “Can
greater improvements in children’s behavior and functioning be attributed to
the system-of-care approach?” To answer this question, three CMHS-funded
system-of-care communities were matched in Phase I with three non-CMHS-funded
communities that appeared to have made less progress toward adopting a
system-of-care approach. Two additional CMHS-funded system-of-care communities
from Phase II of the evaluation are being compared to matched communities
without CMHS funding. Children are followed for up to 3 years, both during and
after services are completed. Data are collected from caregivers, youth,
providers, and service agencies.
A more recent addition to the evaluation is treatment effectiveness studies,
which are designed to evaluate the effectiveness of evidence-based
interventions within systems of care. In two CMHS-funded system-of-care
communities from Phase II of the evaluation, some children receive Parent–Child
Interaction Therapy, and some receive system-of-care services as usual.
Children and families are followed for up to 18 months, both during and after
services are completed. In two Phase III communities, some caregivers attend
6-week Common Sense Parenting® training, and children and caregivers are
followed for up to 2 years. The treatment effectiveness study to be conducted
in Phase IV communities is currently being designed and will be implemented
over the next year.
The National Evaluation Team also coordinates or supports the implementation of
the Family-driven Study, the Sustainability Study, the Culturally Competent
Practices Study, the Primary Care Study, and an evaluation training
curriculum for family members.
Partners in the National Evaluation
ORC Macro works closely with several partners to conduct the national evaluation
of the CMHS-funded system-of-care programs. The Research and Training Center
for Children’s Mental Health at the Florida Mental Health Institute, University
of South Florida, collaborates actively by conducting System of Care Practice
Review in selected communities, providing logic modeling supports to
the program, analyzing national evaluation data, and consulting on other
program evaluation activities. The Federation of Families for Children’s Mental
Health is also an active partner who provides input on the process and outcomes
of the program evaluation with a specific emphasis on how the evaluation
results can be utilized at the individual community level to strengthen
families. In addition, the Federation and ORC Macro are collaborating on the
implementation of the Family-driven Study. The Research and
Training Center on Family Support and Children’s Mental Health at Portland
State University will evaluate and disseminate the Family-Evaluator
Collaboration Training, a curriculum developed by the Federation of
Families to help evaluators and family members improve their
collaboration. Faculty members from Duke University are assisting with
implementation of the treatment effectiveness study that will be conducted in
Phase IV communities. A Services Evaluation Committee composed of experts in
children’s mental health services helps to guide the overall process of the
evaluation by addressing key components within the context of regularly
scheduled consultation meetings.
Evaluation Products
The national evaluation team is continuously involved in the dissemination of
information regarding the program and its associated outcomes. Regular
evaluation products include an annual report to Congress; monthly System-of-Care
Evaluation Briefs on selected national evaluation topics; presentations
at conferences and briefings to key policy-making audiences; journal articles
and book chapters; and data reports to funded system-of-care communities that
can be used for local-level dissemination and program development.
National Evaluation Team
ORC Macro
3 Corporate Square, Suite 370
Atlanta, GA 30329
Phone: 404-321-3211
Fax: 404-321-3688
Website: http://www.macroint.com
|
Project Management
|
Wayne Holden
Officer-in-Charge
E-mail: Emery.W.Holden@orcmacro.com
|
Brigitte Manteuffel
Principal Investigator
E-mail: Brigitte.A.Manteuffel@orcmacro.com
|
Freda Brashears
System-of-Care Assessment
E-mail: Freda.L.Brashears@orcmacro.com
|
Bob Stephens
Data Analysis/Treatment Effectiveness Study
E-mail: Robert.L.Stephens@orcmacro.com
|
Christine Walrath
Dissemination/Treatment Effectiveness Study
/Special Studies
E-mail: Christine.A.Walrath@orcmacro.com
|
|
System-of-Care Assessment Staff
|
LaQuinda Gentry
System-of-Care Assessment
/Culturally Competence Practices Study
E-mail: LaQuinda.M.Gentry@orcmacro.com
|
Phyllis Gyamfi
System-of-Care Assessment
E-mail: Phyllis.Gyamfi@orcmacro.com
|
Natalie Henrich
Qualitative Data Analyst
/Culturally Competent Practices Study/Site Management Liaison
E-mail: Natalie.J.Henrich@orcmacro.com
|
Gina Sgro
System-of-Care Assessment/Site Management Liaison
E-mail: Gina.M.Sgro@orcmacro.com
|
Matosha Harris
System-of-Care Assessment
E-mail: Matosha.J.Harris@orcmacro.com
|
Anika Keens-Douglas
Research Assistant
E-mail: Anika.I.Keens-Douglas@orcmacro.com
|
Site Management Staff
|
Dimple Desai
Site Management Liaison
E-mail: Dimple.V.Desai@orcmacro.com
|
Eileen Franco
Site Management Coordinator and Liaison
/Treatment Effectiveness Study
E-mail: Eileen.Franco@orcmacro.com
|
Stacey Willocks
Site Management Liaison
E-mail: Stacey.A.Willocks@orcmacro.com
|
Joe Walker
Tribal Site Management Liaison
E-mail: Joseph.J.Walker@orcmacro.com
|
Data Management Staff
|
Brooke Brimm
Data Management/Site Management Liaison
E-mail: Brooke.H.Brimm@orcmacro.com
|
Ranilo Laygo
Data Management Coordinator
/Primary Care Study/Site Management Liaison
E-mail: Ranilo.M.Laygo@orcmacro.com
|
Bhuvana Sukumar
Data Management
/Treatment Effectiveness Study/Site Management Liaison
E-mail: Bhuvana.Sukumar@orcmacro.com
|
Elana Light
Data Management/Site Management Liaison
E-mail: Elana.R.Light@orcmacro.com
|
Jun Wan
Data Management
E-mail: Jun.Wan@orcmacro.com
|
Andrea Strahan
Research Assistant
E-mail: Andrea.E.Strahan@orcmacro.com
|
Services & Costs Study
|
Anna Kriveloya
Services and Costs Analyst
E-mail: Anna.Krivelyoa@orcmacro.com
|
Ebony McDuffie
Research Associate
E-mail: Ebony.R.McDuffie@orcmacro.com
|
Becky Wood
Research Assistant
E-mail: Rebecca.F.Wood@orcmacro.com
|
|
Other Project Staff
|
Leza Young
Publications and Presentations/Special Projects
E-mail: Katherine.E.Young@orcmacro.com
|
Angela Sheehan
Treatment Effectiveness Study
/Evidence-based Practice Study/Site Management Liaison
E-mail: Angela.K.Sheehan@orcmacro.com
|
Qualandria Bell
Project Assistant
E-mail: Qualandria.A.Bell@orcmacro.com
|
Sheniqua Williams
Project Assistant
E-mail: Sheniqua.L.Williams@orcmacro.com
|
Susan Zaro
Project Consultant
E-mail: Susan.M.Zaro@orcmacro.com
|
|
A liaison from the national team acts as the primary resource and point of
contact for all system-of-care communities funded since 1997. When
specialized expertise in one of the components of the national evaluation is
needed, however, liaisons work with the core staff of the component to provide
technical assistance to the communities.
|