Why I Don’t Believe Your Annual Report

February 22, 2012 • Blog, Development, nonprofit, Research • by
0 Comments

I received your annual report today. You claimed that 80% of your students go to college while only 40% of those in the community do. You told me its more likely your students go to college than become incarcerated, while it’s the opposite for the neighborhood you’re in. The same for teen pregnancy.

But the thing is, I don’t believe you.

It’s not that I don’t believe your numbers. I’m sure you are reporting them as honestly as you can. I just don’t believe that it was you who caused this difference in students’ lives. You’re suffering from the selection bias.

See, the students in your program are not a random sample from the community. While you’re open to everyone, not everyone shows up, just those that choose to. So you can’t compare yourself to the average of the community. Let’s think of it this way.

In year 1 the community has 10 students, 4 graduate high school. So the number you’re comparing too is 40%.

You start your program in year 2 but can only accept 5 students. Of the 5 students who show up to your program you notice that 4 graduate high school, a full 80%. Twice as much as the previous year. What you don’t realize is that no one else graduates high school (because the students with the will to succeed will seek out opportunities like yours to help them succeed). So for the community, 40% still graduate, you haven’t made any difference.

The selection bias is everywhere. There’s really no way around it other than to force people randomly in and out of your program. So be careful when comparing your statistics to the general statistics of your community.

Tomorrow I’ll talk about how to get me to believe your numbers.

Neighborhood Affects on Obesity & Diabetes

October 27, 2011 • Blog, Measurement & Evaluation, Public Policy • by
0 Comments

A professor from my former school, Jens Ludwig, had the opportunity to be play a key role in the Move to Opportunity study and has recently been looking at its affects on obesity and diabetes. Move to Opportunity (MTO) gave low-income women with children the opportunity to move from their current neighborhoods to more affluent communities. This opportunity was randomized across the group of families that signed up to participate in the study. What Professor Ludwig found was that the women that moved to more affluent communities reduced their likelihood of obesity or diabetes by about 1/5.

The most likely causes of this reduction are:

  1. -Diet (access to healthy foods)
  2. -Opportunity for physical activity
  3. -Access to medical care
  4. -Psychological stress
The study really highlights why randomized control studies in the social sciences and public policy are important. To learn more watch this video with Professor Ludwig discussing his findings.

foodbank

Activities vs Outcomes

July 19, 2011 • Blog, Development, Philanthropy & Giving • by
0 Comments

Activities are the things we do. Outcomes are the things we produce. Outcomes are more necessary than activities and today’s donors don’t want to just fund activities, they want to purchase outcomes.

An activity is handing out food at a food pantry. An outcome is helping people move from food dependence to food independence.

An activity is running an after-school basketball league. An outcome is increasing the odds a student ends up in college.

Activities make up the day to day life of social entrepreneurs and nonprofit leaders. Outcomes are why they started the organization to begin with. Read More…