• Evidence-based decision making depends upon decision makers having access to, and understanding evidence
• Our goal is to ensure the results of our projects inform their decision making.

• Inference = coming to a conclusion through a process of reasoning
• Inference also speaks about the quality of our evidence

Deduction

• Only birds have feathers. [Premise 1]
• A chicken has feathers. [Premise 2]
• Therefore a chicken is a bird. [Conclusion]

Induction

• All of the cows for sale I have ever seen have been brown. [Premise 1]
• My mother is going to buy a cow. [Premise 2]
• Therefore the new cow will probably be brown. [Conclusion]

• Induction allows us to make probable conclusions from samples.
• Statistics is the science of inductive reasoning.

• Our ability to remove the possibility that the same outcome could be produced by a different explanation
• The more explanations you can successfully eliminate the stronger the inference

• Its the opinion of Aunt Elaine.
• There is no measurement.
• The change could be explained many alternate pathways.

• Fish production changed between years in Aunt Elaine’s pond.
• We also know that this coincided with the change to a high rice bran fish feed.
• Results could be one-off (no replication), a result of local conditions or due to a difference unrelated to fish feed.

• Fish production coincided with the change to a high rice bran fish feed
• Consistent across all ponds so not likely due to local pond conditions
• Could still be due to a difference between years unrelated to the type of fish feed

• Fish production increased in the second year.
• There was an inter-year increase unrelated to fish feed.
• There may or may not be an additional effect of fish feed (going in the direction opposite of that expected)

• How can we prove that an intervention is working?
• and we wouldn’t get the same result without it?

• Sometimes referred to as “Counterfactual approaches / thinking”

• Advantage: a starting point
• Disadvantage: just a lonely piece of information, no inference

• Advantage: we start to see differences through time
• Disadvantage: low sample size - can’t distinguish between signal and noise

• Advantage: more measurements = more confidence it isn’t noise
• Disadvantage: more measurements = more work

• Advantage: we can see inherent differences between sites
• Disadvantage: doubled the amount of monitoring

• Advantage: a counterfactual is measured
• Disadvantage: sites may not be replicates (A > B: incorrect), we don’t know the “before” state

• Advantage: the site is its own “control”
• Disadvantage: could be just a temporal effect coinciding with the intervention, or just noise, absence of a temporal “control” impacts inference

• Advantage: we know how sites inherently differ, there is a temporal control, strong inference.
• Disadvantage: we need to have the social license for non-intervention in the temporal control

For experiments

• Use a control group (easier said than done)
• Avoid bias by random assignment to treatment (e.g intervention vs control)
• Include “before” monitoring to understand site based differences

• More sites give us confidence to apply our results more generally
• A greater sample size (number of observations) allows us to better separate signal from noise and ask more sophisticated questions
• Simulate your study design before hand to ensure sample size will be adequate

• BE PRACTICAL: easier to simplify your question rather than increase sample size

• Consider staggered entry (controls are created through time lags)
• Every community gets the exactly the same interventions but the timing of when they receive it differs

{width = “70%”}

Staggered entry: consider deploying 2 mutually exclusive projects

• 1st year: baseline monitoring
• 2nd year: communities randomly receive project A or B
• 3rd year: communities receive the missing project

• Advantage: BACI inference
• Advantage: communities do not see themselves as penalised

Organisational capacity is an issue

• CBOs: qualititative methods e.g. testimonials (narrative)
• small NGOs: before-after perception surveys (tables & excel graphs)
• national NGOs/ government departments: field monitoring: before-after, less often control-impact (basic statistical methods e.g. t-tests, chi-squared tests)
• INGOs: potential for some advanced statistical methods (e.g. linear mixed-effect models, use of remote sensing data)
• international collaborations with researchers: novel statistical methods and innovative project design

• You now understand the effectiveness of your project or now know how to exploit a pattern in the data e.g. new anti-malaria practice
• How do you ensure the decision makers are aware of this information?
• How do you ensure the decision makers use this information in their decision making?

• Ideally:
  • you developed a plan early on
  • decision makers have been actively involved all along

• Decision makers need to be involved early
• Focus monitoring on inference not numbers
• Aim for the best inference within budget constraints & organisational ability
• Be realistic
  • If it will be a challenge for a CBO to collect and evaluate data, accept qualitative data such as testimonials
  • A well financed, multi-year project run by a large INGO should be held to a much higher level of inference

• Most organisations don’t understand how data and strategy fit together.
• Data is not just for experiments…