National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
BACKGROUND
The National Evaluation Team is a partnership comprised of Macro International Inc. (Macro) [http://www.orcmacro.com/] and Walter R. McDonald & Associates, Inc. [http://www.wrma.com/]. This partnership is supported by evaluation partners of the Federation of Families for Children’s Mental Health [http://www.ffcmh.org/] and the Research and training Center for Children’s Mental Health (Louis de la Parte Florida Mental Health Institute, University of South Florida) [http://rtckids.fmhi.usf.edu/].
Macro International Inc. (Macro) is a research, management consulting, and information technology firm supporting business and governments worldwide. It provides survey and market research, policy analysis and evaluation, performance improvement, training, and technology support.
Walter R. McDonald & Associates, Inc. (WRMA) is a leading provider of consulting services to human services agencies and organizations, WRMA provides managerial and technical consulting in the education, juvenile justice, health, and human services fields to improve the lives of children and families.
The Center for Mental Health Services (CMHS) contracts with Macro International Inc. (Macro) and WRMA to conduct a national multisite evaluation of the implementation of systems of care in these communities.
The evaluation has several goals:
< Describe the children and families served by the CMHS-funded systems of care.
< Determine the nature and extent of clinical and functional outcomes for children and families served in systems of care.
< Examine how children and families experience services within systems of care and how they use services and supports (i.e., utilization patterns).
- Estimate the cost of serving children in systems of care.
- Examine the development of systems of care as they move toward offering integrated and comprehensive services.
- Assess the extent to which funded systems of care are sustainable, and sustained post-funding.
< Assess the effectiveness of the system of care approach as compared to usual service delivery approaches.
< Assess the effectiveness of evidence-based treatments within systems of care.
- Provide technical assistance to grantee communities to enhance community-level evaluation capacity and support data collection for the national evaluation.
- Provide feedback to CMHS about program progress at the grantee level to support targeted technical assistance and continuous quality improvement of the program.
DESCRIPTION OF ACTIVITIES
Study Evaluation
The budget of each funded community includes positions for two full-time local evaluators. These evaluators collect data describing children and families, services these families receive, and the outcomes of those services. The data are collected as specified by the guidelines of the National Evaluation.
The National Evaluation Team is responsible for training local evaluators in national evaluation data collection and submission procedures; analyzing the data; interpreting the results; producing annual reports to Congress and other written reports and products on the progress, results, and lessons learned from the implementation of system of care programs. In addition, continuous updates are given to CMHS on the status of the project, and funded communities are given individual reports with analyses of their data. Phase I of the evaluation began in 1994 and included the 22 communities that began their grant funding in 1993 and 1994. Phase II of the evaluation began in 1997 and included 9 communities that began their funding in 1997 and the 14 communities that received initial grant funding in 1998. Phase III of the evaluation includes the 22 communities that began their grant funding in 1999–2000. Phase IV of the evaluation includes the 18 communities funded in 2002, 7 communities funded in 2003, and 4 communities funded in 2004. Phase V of the evaluation includes 25 communities funded in 2005, and 5 communities funded in 2006.
In addition, the National Evaluation Team collects system-level data that includes information on the development of interagency infrastructure, strength-based services, family involvement, and culturally competent services in each funded community. National Evaluation staff work with local communities to obtain service and cost data from billing and management information systems to examine expenditures for services and shifts in service costs.
Evaluation Design
The core evaluation design includes five components:
Descriptive Data Collection
The descriptive data profile children and families who enter systems of care. Descriptive data are collected on all children and include demographic characteristics, social/functional characteristics, and mental health diagnoses. The demographic data include characteristics such as age, ethnicity, sex, family income, and family living situation. The social/functional data include information such as educational performance, juvenile justice history, previous psychiatric problems, and abuse history. The distinctions and similarities among children based on their descriptive data may help explain variations in the treatment experiences of children and families.
Outcomes Data Collection
The outcomes data trace changes in children’s mental health across time. Outcomes data are collected on a sample of children in each funded community at intake into the system of care and every 6 months thereafter. Outcomes data include assessments of children’s emotional and behavioral status, strengths, educational performance, involvement with the criminal justice system, children’s living environments, strains experienced by caregivers of children with mental health needs, family functioning, and child and family satisfaction with the services.
Services and Costs Data Collection
Services and costs data describe the types of services used by children and families in the system of care and the costs associated with these services. The evaluation collects individual data from community management information systems (MIS) on children’s use of services and their costs in the system of care. The analyses of services and costs data highlight the mixture of services delivered by systems of care and the estimated average costs of treating a child.
system of care Assessment Data Collection
system of care assessment data describe the organization of each child-serving system, how it delivers services, and how it affects the mental health outcomes of children and families. Every 18 months, site visits are conducted to each funded community to determine the types of developmental changes taking place at the organizational and service delivery levels. A system of care community’s level of development is determined by measuring system attributes during each site visit through record reviews, interviews, and observations. Examples of the attributes include service availability, cultural competence of services, family involvement, and interagency structure.
Program Sustainability Data Collection
Sustainability data describe the extent to which program infrastructure, philosophy, services, and funding are maintained, as well as factors considered to influence their sustainability. Key individuals in each community complete a survey during years associated with transitions in program match requirements, and during the last year of funding. Graduated communities are assessed five years post-funding.
Comparison and treatment Effectiveness Studies
A significant aspect of the national evaluation is that it includes comparisons between those children and families served by CMHS-funded systems of care and those served by usual service delivery systems. The comparison study component of the national evaluation was initiated to address a critical question: “Can greater improvements in children’s behavior and functioning be attributed to the system of care approach?” To answer this question, three CMHS-funded system of care communities were matched in Phase I with three non-CMHS-funded communities that appeared to have made less progress toward adopting a system of care approach. Two additional CMHS-funded system of care communities from Phase II of the evaluation were compared to matched communities without CMHS funding. Children are followed for up to 3 years, both during and after services are completed. Data are collected from caregivers, youth, providers, and service agencies.
A more recent addition to the evaluation is treatment effectiveness studies, which are designed to evaluate the effectiveness of evidence-based interventions within systems of care. In two CMHS-funded system of care communities from Phase II of the evaluation, some children receive Parent–Child Interaction Therapy, and some receive system of care services as usual. In two Phase III communities, some caregivers attend 6-week Common Sense Parenting® training. In the treatment effectiveness study being conducted in two Phase IV communities some families receive Brief Strategies Family Therapy.
Partners in the National Evaluation
Macro International Inc. (Macro) works closely with several partners to conduct the national evaluation of the CMHS-funded system of care programs. The Research and training Center for Children’s Mental Health at the Florida Mental Health Institute, University of South Florida, provides logic modeling [http://logicmodel.fmhi.usf.edu/#]supports to the program and funded communities, supports System of Care Practice Reviews, analyzes national evaluation data, and consults on program evaluation. The Federation of Families for Children’s Mental Health conducts evaluation training for family members, provides input on the process and outcomes of the program evaluation with a specific emphasis on how the evaluation results can be utilized at the individual community level to strengthen families. In addition, the Federation and Macro International Inc. (Macro) collaborate on the implementation of a family-driven research study. A Services Evaluation Committee composed of experts in children’s mental health services provides guidance to the national evaluation by addressing key components through annual and ad hoc meetings.
Evaluation Products
The national evaluation team is continuously involved in the dissemination of information regarding the program and its associated outcomes. Evaluation products include an annual report to Congress [http://www.mentalhealth.samhsa.gov/publications/allpubs/CB-E198/default.asp, http://www.mentalhealth.samhsa.gov/publications/allpubs/CB-E199/default.asp, http://www.mentalhealth.samhsa.gov/publications/allpubs/CB-E200/default.asp, http://www.mentalhealth.samhsa.gov/publications/allpubs/CB-E201/default.asp]; Special Issues (Children Services: Social Policy, Research, and Practice Journal [http://www.leaonline.com/toc/cs/5/1?part&cookieSet=1]; Journal of Emotional and Behavioral Disorders [http://www.questia.com/browselibrary/journalArticles.jsp?r=5002065999&o=2001|03|22|009|001|Spring]); monthly system of care Evaluation Briefs on selected national evaluation topics; presentations at conferences and briefings to key policy making audiences; journal articles and book chapters; and data reports to funded system of care communities that can be used for local-level dissemination and program development.
CONTACTS
National Evaluation Team
| |
Atlanta |
New York City |
Macro International Inc. (Macro)
3 Corporate Square, Suite 370
Atlanta, Georgia 30329 |
Phone: (404) 321-3211
Fax: (404) 321-3688
http://www.orcmacro.com |
Phone: (212) 941-5555
Fax: (212) 941-7031
http://www.orcmacro.com |
| |
Rockville |
Denver |
Walter R. McDonald and Associates, Inc.
12300 Twinbrook Parkway, Suite 310
Rockville, MD 20852 |
Phone: 301-881-2590
Fax: 301-881-0096
http://www.wrma.com |
Phone: 303-488-3437
Fax: 303-488-3421 |
Project Management Staff
Brigitte Manteuffel
Principal Investigator
Brigitte.A.Manteuffel@orcmacro.com
John Fluke (Colorado)
Principal Investigator/National Evaluation Liaison
jfluke@wrma.com
Freda Brashears
system of care Assessment
Freda.L.Brashears@orcmacro.com
Jennifer Dewey
Site Management/National Evaluation Liaison
Jennifer.D.Dewey@orcmacro.com
Phyllis Gyamfi
system of care Assessment/Data User’s Group
Phyllis.Gyamfi@orcmacro.com
Anna Krivelyova
Services and Costs Study
mailto:Anna.Krivelyova@orcmacro.com
Angela Sheehan (New York City)
Evidence Based treatment Studies/Continuous Quality Improvement
National Evaluation Liaison
mailto:Angela.K.Sheehan@orcmacro.com
Bob Stephens
Data Analysis/treatment Effectiveness Studies
Robert.L.Stephens@orcmacro.com
Christine Walrath-Greene (New York City)
Dissemination/treatment Effectiveness Studies
Christine.M.Walrath-Greene@orcmacro.com
Susan Zaro
Special Studies
Susan.M.Zaro@orcmacro.com
Carol Lichenstein
Site Management/National Evaluation Liaison
clichenstein@wrma.com
Kara Reihman
Treatment Effectiveness Study/Primary Care Provider Study
Kara.S.Riehman@orcmacro.com
system of care Assessment Staff
Matosha Glover
system of care Assessment
Matosha.J.Glover@orcmacro.com
Site Management Staff
Stacy Johnson
National Evaluation Liaison
Stacy.F.Johnson@orcmacro.com
Sheniqua Miles
Site Management/system of care Assessment
Sheniqua.L.Miles@orcmacro.com
Michelle Schurig
National Evaluation Liaison/Spanish translations
Michelle.L.Schurig@orcmacro.com
Laura Whalen
National Evaluation Liaison
Laura.G.Whalen@orcmacro.com
Tisha Tucker
National Evaluation Liaison
Alyce.L.Tucker@orcmacro.com
Connie Maples
National Evaluation Liaison
Connie.J.Maples@orcmacro.com
Data Management Staff
Ye Xu
Data Management
Ye.Xu@orcmacro.com
Services & Costs Study Staff
Ebony Montgomery
Services and Costs Study
Ebony.R.McDuffie@orcmacro.com
Megan Brooks
Data Management
Megan.A.M.Brooks@orcmacro.com
Special Studies Staff
Qualandria Bell
Special Studies/treatment Effectiveness Studies
Qualandria.A.Bell@orcmacro.com
Wanda Hackett
Special Studies
mailto:WHEcare@aol.com
Leza Young
Publications and Productions/Sustainability Study
Katherine.E.Young@orcmacro.com
Project Assistant Staff
L. Anja Maguire
Project Assistant
lmaguire@wrma.com
Tamar Gunby
Program Assistant
Tamar.H.Gunby@orcmacro.com
In Phases II, III, IV, and V of the national evaluation (communities funded in 1997–2005), a liaison from the national team acts as the primary resource and point of contact for the grant community. When specialized expertise in one of the components is needed, liaisons work with the core staff of that component to provide technical assistance to the grant communities.
|