On September 22, 2016 Dr. Rebecca Kilburn, Senior Economist at the RAND Corporation and Professor at the Pardee RAND Graduate School, visited as a guest lecturer in the Evaluation Lab. Dr. Kilburn shared some of her experiences as an evaluator and researcher in the field of early childhood interventions.

Kilburn spoke about the promise and pitfalls of the focus on evidence-based programs.  Of course, we all want programs that work.  But how do we know if a program that claims it is evidence-based really is?  And what if there are no evidence-based programs for a particular population, such as those living in rural and Native American communities?

For those looking to implement an evidence-based program, Kilburn poses four questions. 

First, how reliable is the source claiming that the program is evidence based?  Programs may advertise that they are evidence-based even if the evidence is weak.  Look for studies that used a randomized controlled trial (RCT), which randomly assigns potential clients to program and control groups, and then compares outcomes for the two groups.  Then look for programs that created a big difference for participants, compared with the control group.

Second, was the program tested in the context and for the population you will be serving?  A program that was successful for an African-American population in an urban setting might not translate well to a Native American community in rural New Mexico.

Third, can you implement the program with fidelity?  When a program has a great evidence base, it’s still a “black box.”  You know it worked, but you don’t know which elements made it work.  Was it the curriculum?  Or that it was conducted in the home?  Or that the provider was from the same community as the client?  If you adapt the program by changing the content, location or delivery, you really aren't’t implementing the same evidence-based program.

Fourth, can you meet the program implementation guidelines?  Some programs require that you use Registered Nurses and that you enroll some minimum number of families.

Sometimes it is hard to find an evidence-based program that fits the bill.  Kilburn mentioned the best-known Nurse-Family Partnership home visiting program (NFP).  As its name suggests, a key program element of NFP is the use of nurses.  But New Mexico has a nursing shortage.

Luckily, Kilburn was able to conduct an RCT for a new home visiting program in Santa Fe that uses both nurses and also trained paraprofessionals to deliver content.  The program, called First Born, was developed and has been used for many years in Silver City, NM.  Nurses are in charge of the health content and the paraprofessionals deliver the other topics.  Kilburn and her co-author found that children in the program were less likely to go the emergency room and needed fewer visits to the pediatrician in the first year of life.  Whether other long term advantages accrue to these children must await the passage of time.  But now a program more suitable for the New Mexican context may make its way on to the evidence-based list maintained by the federal home visiting grant program.  States must spend 75% of these grants on the handful of programs that are on this list.  According to Kilburn, this policy may be stifling truly promising and possibly very effective homegrown programs that don’t have the resources to conduct an RCT.

Finally, Kilburn emphasized the importance of the work that students are doing with organization partners, since developing logic models and documenting program activities and client progress are necessary preliminary steps to undergoing a full-scale RCT.