Why I Don’t Believe Your Annual Report

February 22, 2012 • Blog, Development, nonprofit, Research • by
0 Comments

I received your annual report today. You claimed that 80% of your students go to college while only 40% of those in the community do. You told me its more likely your students go to college than become incarcerated, while it’s the opposite for the neighborhood you’re in. The same for teen pregnancy.

But the thing is, I don’t believe you.

It’s not that I don’t believe your numbers. I’m sure you are reporting them as honestly as you can. I just don’t believe that it was you who caused this difference in students’ lives. You’re suffering from the selection bias.

See, the students in your program are not a random sample from the community. While you’re open to everyone, not everyone shows up, just those that choose to. So you can’t compare yourself to the average of the community. Let’s think of it this way.

In year 1 the community has 10 students, 4 graduate high school. So the number you’re comparing too is 40%.

You start your program in year 2 but can only accept 5 students. Of the 5 students who show up to your program you notice that 4 graduate high school, a full 80%. Twice as much as the previous year. What you don’t realize is that no one else graduates high school (because the students with the will to succeed will seek out opportunities like yours to help them succeed). So for the community, 40% still graduate, you haven’t made any difference.

The selection bias is everywhere. There’s really no way around it other than to force people randomly in and out of your program. So be careful when comparing your statistics to the general statistics of your community.

Tomorrow I’ll talk about how to get me to believe your numbers.

Trials & Errors

January 23, 2012 • Blog, Research, Science • by
0 Comments

When we go to the doctor and they prescribe us some medication, we take it with confidence. Why? Because it has gone through a rigorous evaluation process to get from the lab to the pharmacy. That might not be so true though.

Wired has a great article entitled Trials and Errors: Why Science is Failing Us that walks through the failed story of Pfizer’s next blockbuster drug, torcetrapib. It was going to be a breakthrough medication in treating problems with cholesterol, like it’s cousin Lipitor, and when it was announced by Pfizer CEO during its Phase III trial, he was predicting it to be the company’s next big cash cow.

Two weeks later Pfizer announced they were suspending the Phase III trial and the company’s value dropped $21 billion.

The article is really a fascinating discussion of our experimental design methodology and our ability to truly understand complex systems. The author of the article, Jonah Lehrer, essentially states that we are overconfident in our abilities to determine causality within complex living systems. How can we really expect to understand the human body and how it works?, he asks.

While I agree with Lehrer’s conclusion, that we are overconfident in our understanding of causality, I disagree with some of his advice. It seemed at points in the article that his prescription was to cease trying to understand, to which I couldn’t disagree more. Just because an experimental drug that many scientists were confident would work failed Phase III trials does not mean that it wa a failure. We learned something in that study.

By discovering we didn’t know as much as thought, we learned a little bit more than we did.

Scientific understanding in this country is abysmal. I myself acknowledge that I know significantly less about biology, chemistry, and physics than I think a well-educated 20-something should. Prior to graduate school I had not taken a course in calculus or linear algebra. Our education system is failing at teaching our young people the STEM (science, technology, engineering, and math) curriculum and yet it is on those 4 subjects that the future will be built.

Our country has long valued entrepreneurialism, free thinking, and creativity. I myself believe those to be highly important characteristics that we must retain. In fact, I believe that if we can augment our training in STEM while continuing to push our children to discover artistic endeavors like music, literature, and drama, we will actually create better scientists. STEM needs creative people who understand both the laws of the world and have the hutzpah to challenge them. Those are the people who will solve our world’s most pressing problems and create our most lasting institutions.

You should definitely take a few minutes and read Lehrer’s article, Trials and Errors, but I hope you walk away with the belief that we don’t need less scientific inquiry, but more. We don’t need to retreat because we have been overconfident but must push understanding to match our beliefs. And if you have children, make sure they take as many math and science courses as possible in between music lessons and a the reading of a good book.

A Tale of Two Tacos

January 2, 2012 • Blog, Economics, Research • by
0 Comments

Down the street from my apartment is not one but two 24 hour taco places. They literally share a building and yet are not affiliated in any other way. How could it possibly be good for business to have two 24 hour taco places right next door to one another?

It doesn’t make intuitive sense. One would imagine that it would be best for similar businesses to put some space between themselves and yet, like with the two 24 hour taco joints, we see this kind of behavior all the time. Two (three or even four) gas stations share an intersection. Home Depot and Lowes build right next door to one another. Target is often across the street from Walmart.

A guy named Harold Hotelling noticed this kind of behavior among firms back in the 1920s and created a model to describe what was happening. Picture a town with one main street down the middle where two guys want to open identical 24 hour taco places. The people of the town don’t have a preference among the restaurants and will go to whichever one is closest.

Where should the two entrepreneurs open up their business?

Assume they both set up shop on opposite ends of the town, at the far ends of Main street. Since they are equally far apart half of the town goes to Taco Place A and half to Taco Place B. This seems fair but Taco Place A could do better by moving slightly more towards the middle, picking up more customers who are now closer to Taco Place A than Taco Place B.

Imagine this happens again and again. With the stores constantly moving closer and closer towards the middle of town. Eventually they will end up like the two taco places in my neighborhood, right next door to one another in the middle of town.

This result doesn’t just occur in business, it has very important applications for politics. In fact Hotelling’s model has been one of the most important for modern political science.

This theory tells us that in a two-party political system we should see candidates converging towards the middle. Of course our primary system messes with that some but in this recent election we saw more and more independent and third-party candidates running and influencing the debate. We saw this most prominently with tea-party candidates who were making sure that their Republican nominees wouldn’t move too much towards the center of town.

While the center of town might be the most lucrative place, its not where movements happen. The tea partyers on the outskirts of town have fundamentally shaped conservative politics. Those occupying wall street, while still struggling to exert their power don’t spend much time shopping in the middle of town.

If your only goal is to win, then move to the middle of town. There really is no better place (and there’s math to back that up). But if you want to shape the debate, exert your influence, and change the world the outskirts might fit you better.

Experiments in Charitable Giving

December 19, 2011 • Blog, Development, Research • by
0 Comments

The Freakonomics blog had a post last week entitled, To Ask or Not to Ask: Experiments in Charitable Giving. It was a brief follow-up to their What Makes a Donor Donate? podcast episode.

In the post they talk about the research of James Andreoni, Justin M. Rao, and Hannah Trachtman. In their experiment they positioned bell ringers outside of a grocery story in suburban Boston. They told a portion of the bell ringers to not say anything, to just stand there ringing the bell. They told the rest of the bell ringers to solicit customers as they were going in and out, asking directly for a donation.

Their findings are both interesting and illuminate unintended consequences.

While hardly anyone avoided the silent bell-ringers a full 30% purposefully avoided the ones making solicitations. Among those that did give, the donation increased by 75% for those that gave to the bell ringer who solicited the gift. The conclusion: that asking for the gift drives some people away but the gift size might increase.

Check out the whole post here and read their study here.

Evidence Based Philanthropy

December 19, 2011 • Blog, Philanthropy & Giving • by
0 Comments

The Philanthropy 2173 named “Evidence Based” one of their buzzwords of 2011. (Read the post here) I think this is a good choice. I believe that a level of academic rigor can and should be applied to the social sector, especially given the fundamental distortion found in the sector. It plays itself out in that those who receive the service and those who fund them are two people who will most likely never meet. You can read more of my thoughts on that here.

As nonprofit practitioners, we must work to ensure that our organizations are offering programming and services that are backed by research, measured for effectiveness, and creating the impacts we set out to create while minimizing unintended consequences. Of course, numbers and data isn’t everything. There are some things that will be impossible to apply the evidence based methodology to. It is important then, to work with people who know how to decipher what can and cannot be measured.

If your organization is interested in addressing these issues contact me and I’ll be happy to discuss what Means Well Does Good can do for you.

How To Spot A Bargain in Philanthropy

December 8, 2011 • Blog, Philanthropy & Giving • by
0 Comments

What makes something a bargain in the philanthropic world? Last week, Dean Karlan wrote a post on the Freakonomics blog entitled, Bargain Hunting for Charities. He wrote,

Gosh that sounds so stingy. When we are charitable, we don’t want to be cheap. This is our moment of giving, of generosity, not bah-humbugness. Alas, that is exactly what we should be. If we go to a restaurant for chicken wings, what would you think of the following prices:

  • 4 chicken wings: $8
  • 6 chicken wings: $8
  • 8 chicken wings: $8

Which would you opt for (assuming more is always better)? Naturally, it shouldn’t require much thought. So why not apply this to charity?

Karlan then goes on to highlight GiveWell, a great organization that does some very innovative work in studying nonprofit institutions and makes recommendations about excellent charities. There are a couple of similar organizations, but GiveWell seems to be the most robust. Unfortunately, many of these resources are vastly under-utilized when it comes to an individual’s giving, with only 1 in 10 donors utilizing such resources at all.

I love Karlan’s premise but I’d like to take it one step further, because the issue isn’t really that we go to the same restaurant and are shown the same price for different quantity wings. It’s more like different restaurants offering a burger at different prices. In that case we don’t always go for the cheapest. I’m not going to eat a burger at McDonald’s when I can go to Kuma’s Corner.

As donors we shouldn’t be focused solely on quantity but quality as well. We trade off those things in our consumer purchases all the time, and the same thing can occur in our philanthropic choices as well. A focus solely on quantity leads to ever decreasing overhead which does not always lead to the best quality (think low overhead, large transaction chains like McDonald’s). Quality and quantity are not positively correlated, but they don’t have a negative correlation either. Just because something is more expensive does not make it better.

As we approach our philanthropic decisions let’s think about quantity and quality. Also, check out resources like GivingWell. They help you make good giving decisions.

Forks Over Knives: Food & Causality

November 30, 2011 • Blog, Public Policy, Research, video • by
1 Comment

I’ve written some about causality on the blog recently. I think people are often searching for causal stories to explain why things happen. There is one area however, where I don’t think that is true; the negative outcomes of our own behavior. This is heightened when we feel as though we have not done anything wrong.

I’ve been thinking about all this because of a film I saw over the weekend, Forks Over Knives (trailer below). It’s a very interesting documentary on food and diet based largely on the work of Dr. T. Colin Campbell, author of The China Study. The basic idea is that we eat way too much meat and that we really should be focusing our eating on whole foods, i.e.; fruits, vegetables, and grains and that our diet is causing many of our problems with diabetes, cancers, and other diseases.

I read The China Study several years ago and highly recommend it for skeptics of vegetarianism and veganism. As someone who is very skeptical of most research, I found The China Study to be very well done and rigorous in its approach to studying how what we eat affects health outcomes. It is also by far the largest study of its kind and plainly an impressive undertaking.

Forks Over Knives follows several people as they move towards a whole food, vegan diet and the results are quite remarkable. Beyond weight loss and greater health, the participants saw diabetes disappear, cancer overcome, and more.

We don’t want to feel like the food we put into our bodies causes the diseases we suffer from but I would encourage you to take an evening and watch Forks Over Knives or read something like The China Study, Diet for a New America or Eating Animals and stretch yourself.

 

I’m Not Really Biased, Am I?

November 29, 2011 • Blog, Research • by
0 Comments

If there is one thing I have learned from Daniel Kahneman’s book, Thinking Fast and Slow, it is that I do not know as much about myself as I think I do. The Implicit test from Harvard (take them here) can prove this pretty clearly. They are simple word association tests but they can pull out their hidden biases.

For example, it might be easier for you to associate positive words with heterosexuality and negative words with homosexuality. Or it might be more difficult for your brain to organize positive words or images with women than with men.

The tests are quite compelling. As I took one I was surprised how difficult it was to change my instincts. What is important to note though, is that becoming aware of our biases can help us change our conscious actions even as we seek to rewire our unconscious biases. Just because the test shows you are biased towards one group or another does not mean you have to act upon that biases consciously. For example, the test I took showed that I had a bias against a particular group that was marginalized by my religion and while I have taken great strides to change those biases it is important to be aware that unconsciously they still exist.

The brain is a remarkable thing and can take years to rewire. The Implicit test just shows you that you might be further from rewiring than you thought.