1	The Matrix An optimal community evaluation is one that collects and analyzes data for decision making.
2	Evidence Based Programs Evaluation to Strengthen Programs. Evaluation for Accountability and Results. Evaluation for empowerment.
3	Participatory Outcomes Stakeholders are involved from the beginning; its all about people creating change. Responds to the demands for accountability by all parties. Connected to cultural relevance especially with families and communities. Investigates reality in order to change it.
4	Simplistic Evaluation Are we addressing family and community needs? How are we doing over time? Are we meeting standards or outcome indicators? What are we accomplishing?
5	Simplistic Evaluation What are we doing right or wrong? What is the evidence? Can we share what we have learned? What should we do next?
6	What is the Matrix? The Matrix is an outcome-based approach that focuses on the changes of a population. Outcomes tell us how the population is doing at any one time in their actual situation. The Matrix uses indicators that measures the population’s status situation over a period of time.
7	The Matrix Outcomes Model History – Structure Using Tool with Populations Validity – Reliability Matrix Creator
8	History Government Performance and Results Act (GPRA) of 1993. Scales Committee, HHS developed Agency, Family and Community Matrices. Results Oriented Management and Accountability (ROMA). ICCS began examining the California Community Action model in 1996 by providing workshops locally.
9	History In 1999, ICCS & NRCFCP conducted testing for validity & reliability. Since 1997, ICCS has assisted 100+ programs design their indicators and train staff to use the Matrix. Today, ICCS Website provides online design & data base capability.
10
11	Agency Matrix Model Governance Relationships and Linkages Compliance Cultural Sensitivity Workforce Environment Planning Measurement and Evaluation External Communication Information Management
12	Community Matrix Public Policy Equity Civic Capital Service and Support Systems Economic Opportunity Collaboration and Partnerships Leadership
13	Family Development Matrix Provides reliable information directly from family members. Assists staff to address problems, builds on strengths and tracks change as experienced by a family. Families recognize their strengths as well as areas in need of further assistance. The data assists agencies and their funders to continue to improve community-based services.
14	Family Development Matrix Key Point: The purpose of an outcome-based approach is to focus on the well being of the family and ultimately the ability of the community to prepare children for success. The status levels within each category assess the changes that take place in each family situation.
15	Status Level Change (4) Safe/Self-Sufficient represents a strong and secure position for a person or family. (3) Stable represents a current plan and readiness to change. (2) At-Risk represents threat but not yet committed to strategies for change. (1) In-Crisis represents inability to meet needs, near collapse. Requires immediate intervention.
16	Child Development (Child’s Behavior) (4) Child’s behavior is satisfactory without adult supervision. (3) Child’s behavior is responsive supervision. (2) Child has some behaviors requiring adult intervention. (1) Child has intensive behavioral issues and needs immediate adult intervention.
17	Youth Development (Adolescent Job Skills and Work History) (4) Has developed job skills and can keep a job. (3) Can look for work with assistance, has some job skills and/or work history. (2) Has limited job skills and/or work history. (1) Does not have any job skills and/or work history.
18	Family Health (Family Access to Health Care) (4) Has health insurance for entire family and adequate resources to pay for it. (3) Has insurance coverage for major health care needs for entire family or can pay for health care through a combination of sources. (2) Inadequate coverage for entire family. (1) No health insurance or unable to qualify for health insurance.
19	Family Assessment Listen to the Family’s situation. Restate what you have heard using Matrix indicators. Ask more questions to cover the Matrix categories. Score each category together.
20	Subsequent Assessments Baseline Assessment: The worker meets with a family member to determine initial scores for each of the outcome categories contained in the Matrix. Additional Assessments: Subsequent meetings take place depending upon the duration of services where the Matrix "scores" established at the previous meeting are revisited, and another assessment with new scores is completed.
21	Subsequent Assessments A Family Services Plan: Is established with the family using the baseline assessment, reviewed again with the family in subsequent assessments, and adjusted as long as the family is receiving services.
22	Family Services Plan (Based on Principles of Family Support) Step one: Identify areas of improvement and areas of strength. Step Two: Ask the family which category they would like work on at this time. Step Three: Work with the family to identify strengths that can be applied to assist with goal achievement. Step Four: Complete a Family Services Plan.
23	Case Planning
24	Summary Key point: The Matrix not only tracks family needs from the beginning of a worker relationship, but functions as an empowerment tool to accomplish the overarching goal of the family--to become or remain self-sufficient. Key Point: The Matrix is a tool built on the Principles of Family Support. It gives the family a central, active role in determining their goals and steps they will take toward achieving their goals.
25	Outcome Indicator Tests (1) Validity - We ask the population to tell us that the status levels accurately reflect that which is real in their lives. (2) Reliability - Looking for the same outcome result when repeated measurements are taken by various individuals under the same conditions. (3) Pilot Test - The Matrix user experiences the outcomes model in assessment situations just before data collection. Revisions to the Matrix Model are made after each test.
26	"Jerry Endres" Jerry Endres Brad Richardson Judi Sherman Funding by the Packard Foundation
27	"Brad Richardson" Brad Richardson, Ph.D. University of Iowa School of Social Work National Resource Center for Family Centered Practice brad-richardson@uiowa.edu Office: (319) 335-4924 Mobile: (515) 771-3589 www.uiowa.edu/~nrcfcp
28
29	"OCS Sponsored" OCS Sponsored: Network Guide to Measuring Family Development Outcomes (Richardson, Theisen and Spears, 2003) Packard Foundation Organizational Effectiveness Program Sponsored: Testing the Reliability/Validity of the California Matrix (Endres, Richardson and Sherman, 1999) The Prevention Report: Outcomes Consultation: Lessons from the field (Part II): The Automated Assessment (Richardson and Landsman, 1999 #1) Using Outcomes in Decision-Making (Richardson and Huff, 2001 #2)
30
31	"Demonstrate effectiveness" Demonstrate effectiveness Promote informed decision making Help define how we go forward Formative and Summative Evaluation
32
33	"Reliability" Reliability: Intra-Rater Inter-Rater Parallel Forms Internal Consistency Validity: Face Construct Criterion Convergent Discriminant
34
35	"Narratives based on actual case..." Narratives based on actual case histories. Workers from agencies volunteered as subjects. Workers completed: Rater Profile & Score Sheet. Score Sheet = 11 categories of the Matrix. Rater Profile = last six social security digits, age, agency years in present position, years social service experience, whether or not the participant used the matrix before the study, whether or not the rater received training on the Matrix model.
36	"Workers scored domains using" Workers scored domains using: (1) in crisis (2) at risk (3) stable (4) safe/self-sufficient (5) thriving Half-points allowed Re-test conducted 2-3 weeks later
37	"Paired comparisons (t-tests and..." Paired comparisons (t-tests and Wilcoxon signed rank tests). Inter-rater reliability - which domains produced the most consistent, or discrepant, ratings. Focus group discussions provided for improvements in definition and sensitivity. Exogenous factors explored (training, age, experience). Consistency of ratings examined (~ item analysis used in test construction).
38	"Paired Comparisons showed consistent results" Paired Comparisons showed consistent results. Of 110 paired comparisons, 10 differed between the two points in time. Significant differences accounted for by only 2 cases.
39	"Only 10 items were significantly..." Only 10 items were significantly different between raters. 7 of the significant differences accounted for by only 2 cases.
40	Inter-Rater & Internal Consistency Reliability Obtained more than 60 percent agreement. Internal consistency (items across domains) obtained Cronbach’s alpha of .77 to .93 (T1) and .64 to .93 (T2).
41	"Mean age of raters =..." Mean age of raters = 41. Years in social service = 10. Years in current position = 3 to 4 years. Trained and had experience using Matrix = 78%. 44% of participants were employed in two agencies; not more than two participated from any of the other agencies. Significant correlations between ratings and rater characteristics were not found.
42	"Strength-based approach" Strength-based approach Builds Collaboration Outcome evaluation Family Centered/Partnership Cultural competence
43	"Creating a healthy family doesn’t..." Creating a healthy family doesn’t mean: Working toward some master list of “strengths.” What strengthens one family may not strengthen another family. Family strengths have to be discovered. Matrix works because measures are locally determined. Requires local assessment of reliability and validity.
44
45	"NEW" NEW Your program is now able to design outcomes and organize data using the Institute’s Web based system. The “Matrix Creator” guides the agency using an automated tool to: Customize a Matrix model to fit a specific program population. Conduct reliability and validity tests for a Matrix Outcomes tool. Train staff to use the Matrix model with populations. Conduct ongoing assessments and outcomes measurement. Enter and retrieve instant data and chart findings.
46
47	"Adolescent Development" Adolescent Development Adoption Adult Education & Employment Agency/Organization Standards Basic Needs Children’s Care & Safety Children’s Education Children’s Development Collaborative Leadership Community Development Community Engagement Community Environment Family Environment Family Relations Financial Stability Health & Safety Immigration Intergenerational Programs Juvenile Impact Program Long Term Care Culture Change Parents/Child Relationships Potential for Adolescent Transition Policy Leadership Seniors/Elders Support Program Seniors Employment and Service Shelter Social & Emotional Health Social Family Support Student Development Substance Abuse Transportation Volunteers
48	"Categories and Indicators" Categories and Indicators
49
50
51
52
53
54
55
56	"The Matrix model is an..." The Matrix model is an outcome tool for measuring performance in establishing results accountability. Conduct ongoing assessments, case management, and outcomes measurement. Online procedures for data collection and data entry Automated data entry using your program’s Matrix model Retrieve data immediately and chart findings Print charts and include in reports
57	"For information contact" For information contact: Jerry Endres, M.S.W. Director, Institute for Community Collaborative Studies Department of Health, Human Services and Public Policy California State University, Monterey Bay (831) 582-3624 (831) 582-3899 fax jerry_endres@csumb.edu http://hhspp.csumb.edu/community/matrix