Using Data to Drive Program Improvements

[serialposts]

PART SIX: Using Data to Drive Program Improvements

Data. I know it is a four-letter word. It makes policy wonks salivate lustfully and makes many front-line practitioners run for the hills (or the bottle).

Truth is, data doesn’t have to be scary or cumbersome or a nuisance. Done right, data is the ace up your sleeve to make your program transition from good to great.

As a starting point, know that there are resources out there that can help you if you are unfamiliar or uncomfortable with data. The National Alliance to End Homelessness has a range of nifty resources. I especially like What Gets Measured Gets Done. Data and performance measurement is also a subject matter I get asked to speak about a lot. So, if you want to check out some ofthat – littered with “Iain-isms” – feel free. Plus there are a few previous blogs (not part of this current series) where I have talked about performance measurement, data and organizing information in the context of functioning like a system instead of a collection of projects. This one in particular is short and the feedback we’ve received suggests it is my most entertaining blog entry (fire alarms, vibrating bed, strobe lights, knocks on the door in the middle of the night – how can you go wrong?). A couple of other articles may be a useful read if you are unfamiliar with some of the core concepts of data and performance management, or want to better understand how measurement improves organizational learning.

Now onto the matter at hand – driving program improvements through the use of data.

Collect the Right Data at Intake and Assessment to Help the Person/Family Get to the Right Program to Meet their Needs

I think a lot of intake and assessment processes can use refinement. Too often there is a “deep dive” into information that is completely unrelated to determining which housing program is best going to meet an individual’s/family’s needs. It can also be problematic to not have effective screening tools prior to going deeply into the intake and assessment.

Remember that an intake and assessment process should be about the client. It isn’t about the organization. It is about meeting their needs. It is about offering the right housing program choices based upon the needs they present.

We strongly recommend the use of the Service Prioritization Decision Assistance Tool to improve intake and assessment. To be transparent, we created it. But, it was created through a very thorough process with lots of vetting and research. And most of all we can PROVE that housing retention, client satisfaction and case manager satisfaction all increase through the use of the tool when compared to other tools, self-sufficiency matrices or no tool at all. The tool is now in use with over 70 communities across the world, been endorsed by various government entities and been supported by psychiatric consumer survivor groups.

Collect only What you NEED to Collect

More data is not better data. Twice as much is not necessarily twice as good.

So what is NEEDED?

You need basic (stress ‘basic’) demographic data and information that will help inform what the best housing program will be. Other information may be collected during a case management or support function; collecting that during intake and assessment is not necessary. In fact, I would argue it is unnecessarily intrusive.

High-functioning non-profits (and not just in the housing and homeless sector) have learned the lesson that less is more when it comes to data collection. Take for example Strive in Cincinnati. They went from about 150 data points down to about 10 that they felt were most important for their work. Dramatic decrease with remarkable increased performance (and buy-in).

Ask Yourself the “So What?” Question

I tell people repeatedly that we need to see our work in housing and homeless service delivery as QUALITY work, not QUANTITY work. I realize getting this message across to funders and politicians can be especially difficult. On a recent speaking tour throughout Minnesota I was bombarded with questions about the tension between what funders ask for and what organizations think they can deliver in a meaningful way. Many organizations feel pressured to serve more and more people, rather than focusing on a smaller number of people and serving them really, really well (so well in fact that they don’t become homeless again).

Which leads me to the “So What?” question.

Every organization needs to ask themselves the question of what difference they are actually making. For example: Organization X boasts that they housed 100 people last year. I ask, so what? Did they remain housed? Did the quality of their lives improve? Was there a positive impact on the community at large? What did it take (from a resource perspective) to achieve the work, and how much is it going to require on an ongoing basis? Why those 100 people and not a different 100 people – what selection process and prioritization process did you use and why?

Listen to Your Entire Staff Team & Build the Data Collection Requirements Across Organizations

There is some pretty interesting research that has explored the effectiveness of performance measurement systems when they are imposed down through a hierarchy as opposed to generated collaboratively (see for example Eckhart-Queenan’s work). There has been some other research that has explored the notion of trust as an important ingredient for successful performance management systems.

I highly recommend that on each staff team, everyone should be involved in the creation of the program logic model. It should not be something that happens in a back-room function solely as part of a funding application. Make it transparent and operational.

When building data collection requirements across a system, I strongly recommend the involvement of multiple organizations in the development of the approach and metrics. As we have done in our recent work in Detroit with the Homeless Action Network of Detroit, this is a three pronged process: conduct a survey to understand how people currently feel about data and how they use it; map out the existing array of services from the perspective of the client, from opportunities for diversion/prevention right through to how we support and monitor housing retention after they successfully exit the program; and, then have the community draft out the metrics that they think are important for each program area within the service system.

Your HMIS is NOT Your Performance Measurement System

Your HMIS is a place to store data and run reports on your data. That is awesome… if data entry is complete and there is a thoughtful data analysis plan. It also helps when it is an open system as opposed to a closed system.

Truth is, the HMIS presents information. It does not interpret information. That is up to you. The question you need to be able to answer from that which is included in the HMIS is What does this mean?

Don’t Just Do What your Funder Demands

Better data allows organizations to better influence the sort of information that funders look at as well as how they interpret it. For this reason – but not this reason alone – I am an advocate for organizations collecting not just what funders demand as a condition of receiving funds. Organizations should also collect the data they need to internally reflect on their program, what is working/not working (and why) and what they may want to consider doing differently.

Use The Data All Over the Place & Present it in Different Ways

Want buy in on data? Use it all over the place…staff meetings, newsletters, website, community meetings, board meetings, Facebook, that annoying spiel you here when you are waiting for someone to pick up their phone, Twitter, bulletin boards, plaques by the reception area, etc.

Too often, data is seen as something that drifts into a black hole never to be seen again. Or when it does emerge seems to be months or years after the fact. Bad idea. Bring the data to life. The way to do that is to use it in a timely way and plaster it all over the place. Everyone will know your organization and who visits your organization will know you are serious about data and performance if you do these things.

Also, remember that people learn in different ways and will respond to your data depending on how you present it. Consider different approaches – graphs, charts, infographics, trend analysis, narratives, etc. Don’t just hand out spreadsheets and expect people to do cartwheels.

Inputting Data is Part of the Real Work

If you want data to drive program improvements, the data has to actually exist, right? How many organizations reading this have one or more staff person who is behind in their data entry, despite frequent reminders/requests?

Every high performing housing program I have been a part of or evaluated sets aside work time within each and every day that is solely related to entering data and case notes. Nothing else. Not general admin time. It is data and information time. These organizations tend to have up to date data within their systems within 24 hours of client interactions.

Data and information entry is part of the real work done in housing programs. It is not something that happens when time allows or only when there are no direct service demands from clients. It can be a huge cultural change, but timely data entry is critical for housing program success.

Having a Meaningful Data Typology & Data Analysis Plan

If you create the right pieces of information in the right ways you will never need to hire a high-priced consultant, academic or analyst to make sense of your data (many of whom would just look at your watch and tell you what time it is anyway).

This means, however, that you need to invest time and energy thinking in advance about what progress reports and analysis you want to create at what time intervals, and what data manipulations or calculations will be required for that to happen. The more you think about this in advance and plan in advance, the more consistent you will become in the use of data (reports get run when they are supposed to get run) and your have focused attention not on all possible types of analysis that are possible, but rather the pieces of information that are most important to understanding if your housing program is working/not working, and for which populations and why.

The data typology simply refers to how you organize your data. Common ways are things like gender, age of clients, length of time homeless, veteran or non-veteran, ethno-racial identity, service entry point and the like.

The data analysis plan tells you how you find the answers to the questions you are asking. For example, if it is important to your organization to know how female veterans under the age of 30 are doing in housing compared to female non-veterans within the same age cohort you’d set up the queries necessary in advance to answer that very question on an ongoing basis.

Keep it Simple

The more simple you keep the data collection, analysis and dissemination of the data, the more buy-in there is going to be to data overall. If people feel that PhD’s need to muck about to make sense of the information, the day to day operational importance of the data has likely been lost. Everyone in your housing program should know exactly what the data being collected is, why it is important and what it is intended to measure. If they don’t all understand then you need to keep breaking it down until it is simply understood by one and all.

Don’t Ignore the Data if it Tells You Something You Didn’t Want to Know

Over the past 20 years working on various social justice projects it would seem to me that organizations love and celebrate their data when it seems to demonstrate that they are doing a good job and supports the narrative of what it is they say they are doing. BUT, many of these same organizations have a tendency to distance themselves from data if they think it presents any sort of picture that would somehow diminish their feeling of awesomeness.

Truth is, awesome organizations embrace continuous improvement not as a management buzz-phrase, but as something integral to their organizational DNA. Those organizations look at all data, but particularly like the data that suggests maybe their programs are not performing well. Then they can use data to reflect on practice and make substantial improvements, rather than continuing to deliver the same programs in the same ways with the same poor – or even just average – results.

Set Meaningful Indicators and Targets

Targets should never be an aspiration. They should be what you think the housing program can reasonably achieve with the resources available and within the specific operational climate. Unrealistic targets are a recipe for alienating people from wanting to collect and use data.

Indicators tell us the information we need relative to the targets. If we aren’t looking at the right things we’ll never know if the targets are being achieved. The two (indicators and targets) require a strong marriage.

I am a fan of creating indicators and targets within sectors of service as opposed to for specific projects. For example, I would suggest a minimum threshold in each of the following: outreach; emergency shelter; drop-in centers; employment & income; prevention & diversion; interim housing; permanent supportive housing; and, rapid re-housing. This allows for greater consistency within sectors of service, helps structure the services into a system model, but most importantly in the context of this blog allows us to track data that demonstrates that each of these sectors of service plays a role in ending homelessness. (And by the way, housing is the only known cure to homelessness, so surely to God  – or the deity of your choosing – indicators and targets should have a housing orientation.)

Focusing on these areas of data will improve performance and drive the right performance changes within your housing programs. Ignoring information or working solely from intuition is not a recipe for success. If ignorance is bliss, then there are far too many organizations that are orgasmic. We need dedicated and purposeful attention paid to the importance of data and how it makes us all better practitioners. That which we think can be different from that which we know – and data helps us figure out the difference.

[serialposts]

Iain De Jong is a data nerd to the nth degree. He is always looking for kindred spirits who share his passion for making data and performance management cool. More than a third of his professional work is related to data and performance systems, both re-constructing them as well as keynote speaking and seminars to get people pumped up about the opportunities that data truly presents. If you share his passion for data or want to explore specific pieces of information that should be collected and analyzed relative to your specific program, let him know atidejong@orgcode.com