What’s So Hard About Evidence-Based Practice? Step 2: Engaging with Research Print E-mail
By Pru Mitchell   
There are various models of evidence-based practice (EBP) that incorporate a varying number of steps. Table 1 shows one six-stage process of EBP (University of Sydney, 2015) and Todd’s (2008) EBP model of evidence for practice, evidence in practice and evidence of practice, which may be familiar to school library staff.
 

Step

What is EBP
(University of Sydney, 2015)

Holistic model of evidence-based practice
(Todd, 2008, p.40)

1

Formulate a clinical question

Evidence for practice... examining and using empirical research to form practices and inform actions, and to identify best practices

2

Search the literature

3

Sort, read and critique the literature

4

Come to a ‘bottom line’, ie recommendations for practice, based only on the best available literature

5

Implement the recommendations, documenting them... apply the evidence to the situation in consultation with those who it will affect

Evidence in practice... integrating the available research evidence with the deep knowledge and understanding derived from professional experience, as well as using local evidence to identify learning dilemmas and needs, and achievement gaps

6

Share the results of your EBP with others, ideally through writing them up

Evidence of practice...measured outcomes and impacts, going beyond process and activities as outputs. It establishes what has changed for learners as a result of inputs, interventions, activities, and processes

 
Table 1 Models of evidence-based practice
 

 

 
Part 1 of this article highlighted the increasing emphasis on evidence-based practice in education, and considered the difficulties educators encounter in searching for and accessing research literature. This corresponds to steps one and two from the EBP model, and to part of Todd’s evidence for practice.
 

. . . "finding is easy . . . but reading is hard" . . .

While many teachers would agree with Bailey (2015) in her rant about how publishers and libraries could improve access to research literature, when Stanford researchers were interviewed about the challenges of searching, they declared that "finding is easy . . . but reading is hard" (Sack, 2014). This article looks at how school libraries might raise a culture of reading research as described in step 3 above. What are the challenges educators face as they ‘sort, read and critique’ the literature they need to inform evidence-based practice?

Getting sorted

Research traditionally comes as a report, often presented as a densely typed two-column pdf – a throwback to an era of paper journals with their limited page space. However, evidence does not just come in journals, and the range of formats and platforms for publishing research is growing. For example, Jove, the Journal of Visualized Experiments, is a peer reviewed video journal of science research. There is also a move to publish the datasets that inform research reports so they can be checked, re-used and mashed up with other data to create new research. Available education data sources include:
The task of sorting research is an obvious area where the information management expertise of the library can assist colleagues to devise a system for managing research as it is collected, especially when it comes from many different sources. Just as we teach students in the ‘organising’ stage of the Information Process some form of tagging and storage is needed. While a card file still works for some people, online bibliographic tools such as Zotero, RefWorks or Endnote allow the import of citations with full-text files attached, categorisation into themes or folders, and export of formatted references.

Curation

There are research information services that help educators by pre-sorting, selecting and organising the mass of published research into forms that are easier to deal with.
 
Curated lists, whereby research experts and content experts provide alerts to ‘good’ research, save both time and the cost of ‘just in case’ library subscriptions - provided the scope of the selection matches your needs and the curator has the right credentials. 
  • The Australian Institute for Teaching and School Leadership (AITSL) maintains a Leadership Research Repository
  • The Australian Council for Educational Research (ACER) has a number of lists curated from the Australian Education Index by topic. For example Learning Ground is updated with evidence from Indigenous researchers, educators and organisations working to improve outcomes for Indigenous learners. For those who prefer Twitter as their curation tool, updates are tweeted @LearningGround.
Of course, merely collecting and organising research does not ensure we are ‘reading’ and learning from it. That is another challenge entirely.

Reading research

Website analytics are a great source of evidence, and what they reveal about reading online provides a challenging insight. An online education magazine editor shared her concern recently that while their research articles and interviews generate significant numbers of tweets and likes and mentions, the page analytics show that very few of these ‘readers’ actually read the article or clicked on the video interview. This trend is reported by other online publications (Mangoo, 2013). In effect, teachers are broadcasting, but not engaging with the research.
 

In effect, teachers are broadcasting, but not engaging with the research.

Most would agree that reading a headline or a tweet about a research paper is not sufficient to underpin evidence-based practice. Academics accuse teachers and education policy makers of ignoring research, and in return practitioners accuse researchers of being "too remote" (Hempenstall, 2006). There is plenty of rhetoric on this supposed lack of engagement with research that involves blaming either or all of the education community, the research community, publishers, lack of funding for research, copyright regimes, the media or social media. Recently, non-educationalists and not-for-profits have entered the fray, holding forth on what needs to be done to fix the lack of evidence in education. The attention and commitment to this issue is welcome, but it is important that the conversation comes from a partnership rather than deficit mindset.
 
At the personal and school level, the factor that we can influence is making research reading a priority. Recognising the many priorities teachers and school leaders face, what can school libraries do that is conducive to building a culture of informed and evidence-based practice? 

Abstracts and summaries

When alerting staff to research, rather than simply passing on a link and a headline, consider including the abstract. Abstracts are a crucial element of research publishing, providing up front details of the research design, description of the study, a summary of the findings and suggestions for further research. The goal of a good abstract is for the reader to know within 100-200 words whether this research contains relevant evidence. For longer reports, an executive summary of up to ten percent of the length of the full report serves the same purpose.
  • MESH Guides is a developing platform where academics and teachers work together to produce summaries of research-based specialist knowledge to support teacher professional judgement, for example Acoustics – Listening and Learning

Reviews and digests

Research reviews provide more in-depth summary of the methodology and findings of a piece of research and may include commentary by a reviewer who has relevant content knowledge. The review format is often more engaging to read than a traditional research report.

A literature review, or digest, reviews selected research on a specific topic, research within a defined time period, or research produced by a particular organisation. They tend to feature recent research, are rarely comprehensive and are structured into a narrative.

Critiquing research

The critiquing of research is an essential component of engaging with research. With students we often use the acronym CRAP (Currency, Reliability, Authority, and Purpose) to teach evaluation of sources. The same requirement for critical thinking, and a certain amount of scepticism, applies when reviewing research into professional practice. Research is not perfect. There are examples of conflicting research, dubious research, unintelligible research or research not relevant to our context. There is particular concern currently over correlations that are read and accepted as causation, and on ‘neuromyths’ arising from misunderstanding of neuroscience (Howard-Jones, 2014). Grinnell & Unrau (2010, p.169) list three key criteria to consider when deciding whether evidence is relevant and robust:
  • validity (closeness to the truth), 
  • impact (size of the effect), and 
  • applicability (usefulness in our practice)
As with any field, the research world has specific ways of working. If understanding and evaluating research is not something covered in previous training, there is a series of accessible articles published in The Conversation, 2014, titled Understanding Research that outlines common traps in reading and understanding research.

Systematic reviews and meta studies

A systematic review seeks to locate and evaluate all available research evidence relevant to a research question. In response to concern over the selective nature of many literature reviews, systematic reviewers work in teams and use a strict and transparent methodology. They evaluate each study according to specific levels of evidence based on validity. This hierarchy indicates which studies should be given most weight in an evaluation (NHMRC 2000, p.100). An example of this hierarchy of evidence can be seen in Kysh (2013).
 
 
 
Figure 2 
Reference: Kysh, L 2013, Difference between a systematic review and a literature review, figshare, http://dx.doi.org/10.6084/m9.figshare.766364. Licence CC-by
 
Randomised controlled trials (RCTs) are experimental comparison studies in which participants are allocated randomly to intervention or control groups. Meta-analyses of randomised controlled trials that take the results from several studies identified in a systematic review, and combine and summarise these quantitatively, are often held as the pinnacle of the hierarchy of evidence (NHMRC, 2000, p. 101). Marzano and Hattie are two well-known names in the area of meta-studies in education. In 2011, Higgins et al. (2015) developed a simple tabular way of displaying the relative cost, evidence and impact of common educational interventions. This was published as "an accessible summary of educational research on teaching 5-16 year olds" by the UK Education Endowment Foundation

Engaging with evidence

In the same way that teaching does not equal learning, the act of conducting or reading research changes nothing for the learner. It requires some kind of response to the evidence, and taking appropriate action. For schools, this response can be effectively developed in local, collaborative teams.
  • The Teacher School Learning Community is one strategy where teachers in teams or at whole school staff meetings put aside some time to read short evidence-based articles and engage in discussion of their context.
  • ResearchED reports on schools in the UK which have appointed ‘research leads’, middle- to senior-level staff with a position of responsibility related to research in the school.
  • The Annual Excellence in Professional Practice Conference (EPPC) provides professional learning teams with an opportunity to share their collaborative action research projects.

Context

Context is a key consideration in considering research. Others can rate effectiveness and impact of evidence, but judging the applicability of that evidence against your school’s context requires local perspective. Education involves human beings who bring a range of variables to any situation. It is impossible to filter for all possible differences to ensure a closed environment where what is being measured is the only thing attributable to the result. Replication of studies in new contexts is a vital, though underfunded element of research.

Conflicting evidence

. . . what we know as the ‘best evidence’ may have been the ‘best available evidence’ at the time.

Conflicting evidence is an issue teachers have to grapple with. Research that leaves more questions than it answers is frustrating but is part of the process, in the same way that negative findings are important contributions to building up evidence. We know that things change, and the results of research conducted some years ago may no longer hold true. All professionals need to recognise that what we know as the ‘best evidence’ may have been the ‘best available evidence’ at the time.
 
It is our responsibility to ensure the currency of our knowledge of the best evidence. There are many studies on educational myths such as Adey & Dillon’s (2012) book Bad Education and McDermott & Hall’s (2007) Scientifically Debased Research on Learning. They raise in particular the question of why people refuse to unlearn something in the face of new evidence.

Curriculum

Engaging deeply with research brings other benefits for teachers and school library staff, including improved expertise and skills for teaching research throughout the curriculum. Lupton (2014) shows how widely these skills are embedded in the Australian Curriculum, and they are essential to the significant individual research activities students undertake as part of senior studies, for instance:

Conclusion

Returning to the models of evidence based practice in Figure 1, it is important to remember that the goal of the ‘sort, read and critique research’ step is to come to a bottom line. That is, to decide on exactly what is the best evidence based on all the information and critical techniques at your disposal. Then beings the real work of applying the learning from this evidence as you implement these recommendations, and move into Todd’s evidence in practice stage.

References

Adey, P. & Dillon, J. (2012) Bad Education: Debunking Myths in Education, Open University Press, Maidenhead.
 
Bailey, N. (2015) ‘Are We There Yet? No . . . and this is Why – An Appeal to Database Owners and Academic Libraries’, Informative flights, http://thinkspace.csu.edu.au/informativeflights/2015/09/20/are-we-there-yet-no-and-this-is-why-an-appeal-to-database-owners-and-academic-libraries.
 
Grinnell, R. & Unrau, Y. (2010) Social Work Research and Evaluation: Foundations of Evidence-Based Practice, 9th edition, Oxford University Press, New York.
 
Hempenstall, K. (2006) ‘What Does Evidence-Based Practice in Education Mean?’ in Australian Journal of Learning Disabilities, 11 (2), pp. 83-92.
 
Higgins, S. Katsipataki, M. Coleman, R. Henderson, P. Major, L. & Coe, R. (2015) The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit, Education Endowment Foundation, London, Accessed at: https://educationendowmentfoundation.org.uk/toolkit/toolkit-a-z/about-the-toolkit.
 
Howard-Jones, P. (2014) ‘Neuroscience and Education: Myths and Messages’ in Nature Reviews Neuroscience, 15, pp. 817-824, doi:10.1038/nrn3817.
 
Kysh, L. (2013) Difference Between a Systematic Review and a Literature Review, figshare, Accessed at: http://dx.doi.org/10.6084/m9.figshare.766364.
 
Lupton, M. (2014) ‘Inquiry Skills in the Australian Curriculum v6: A Birds-Eye View’ in, Access, 28 (4), pp. 8-29.
 
Mangoo, F. (2013) ‘You Won’t Finish This Article: Why People Online Don’t Read to the End’ in Slate, Accessed at: http://www.slate.com/articles/technology/technology/2013/06/how_people_read_online_why_you_won_t_finish_this_article.single.html.
 
McDermott, R. & Hall, K. (2007) ‘Scientifically Debased Research on Learning, 1854-2006’ in Anthropology & Education Quarterly, 38 (1), pp. 9-15.
 
National Health and Medical Research Council (NHMRC) (2000) How to Review the Evidence: Systematic Identification and Review of the Scientific Literature. Handbook series on preparing clinical practice guidelines, Commonwealth of Australia, Canberra, Accessed at: http://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/cp65.pdf.
 
NSW Department of Education (2007) ‘Information Skills in the School: Engaging Learners in Constructing Knowledge,’ NSW Department of Education and Training, Ryde, Accessed at: http://www.curriculumsupport.education.nsw.gov.au/schoollibraries/teachingideas/isp/docs/infoskills.pdf.
 
Perkins, K.  (2014) ‘Parents and Teachers: Working Together to Foster Children’s Learning’ in The Research Digest, No.10, Queensland College of Teachers, Brisbane, Accessed at: http://www.qct.edu.au/Publications/Periodical/QCTResearchDigest2014-10.pdf.
 
Sack, J. (2014) ‘Helping Researchers See Farther Faster’, Google Scholar blog, Accessed at: http://googlescholar.blogspot.com.au/2014/09/10th-anniversary-series-helping.html.
 
Todd, R. (2008) ‘The Evidence-Based Manifesto’ in School Library Journal, 54 (4), pp. 38-43, Accessed at: http://www.slj.com/2008/04/sljarchives/the-evidence-based-manifesto-for-school-librarians.
 
University of Sydney (2015) ‘What is EBP?’ in speechBITE, Accessed at: http://speechbite.com/ebp/what-is-it.
 
 
Pru Mitchell is the Manager of Information Services for the Australian Council for Educational Research. She can be contacted on:
 03 9277 5549
; This e-mail address is being protected from spambots. You need JavaScript enabled to view it ; www.acer.edu.au.