IJPLE 2 (1) 2018

Evaluating legal literacy programmes – aims,
challenges, models and a call to action

Richard Grimes1

Building ‘best practice’ 2

Generally speaking, if one asks most people whether improving public understanding

of the law and legal system is a good thing, most hands will go up. This is perhaps no

surprise. An improved level of legal literacy should not only raise awareness of rights

and responsibilities but may well provide people with more of an informed choice

about what to do if they encounter legal issues. It might give those concerned the tools

and confidence to address some of these problems themselves through self-help and

it may, address, at least to some degree, inequalities that otherwise exist. Overall,

improving levels of legal literacy could enhance access to justice more generally. The

generic term often ascribed to raising legal awareness amongst the wider population

is public legal education (PLE).

1 Formerly Director of Clinical Programmes, York Law School, University of York, UK and now a legal education
and access to justice consultant, and contactable at: richard.grimes@hotmail.co.uk

2 This contribution is based on a presentation and discussion that took place at the Ed O’Brien memorial
workshop on Street Law best practice in Durban, South Africa in April 2016. I am grateful for all who attended
that session and for their insights. Thanks must also go to David McQuoid-Mason and the other staff at the
School of Law, KwaZulu Natal University and the South African Street Law movement more generally for their
vision, organisation and commitment in setting up the event and in following through with the publication of
which this is a part. ‘Best practice’ is used here in the sense of not what is best per se but rather what models,
techniques and approaches can be devised, adapted and implemented through proven work elsewhere.

mailto:richard.grimes@hotmail.co.uk

IJPLE 2 (1) 2018

There is a wealth of anecdotal material suggesting that all of the above is highly

relevant and impactful3 but there is little by way of clear empirical evidence to

substantiate such claims.4

This article will look at the need for, and means of, developing such an evidence base

and ends with a plea for more research, sharing of ideas and collaboration in terms of

evaluating PLE. The article will be presented in the format of an introductory section

looking at the background and history of PLE (in particular Street Law) in terms of

aims, challenges and models from the perspective of evaluating impact.

This is followed by a ‘lesson plan’ in which a sample set of possible PLE options are

set out with the means of evaluating impact incorporated in the model used. It is

hoped that this might provide a guide for those wishing to devise (or revise) their legal

literacy programmes from an evaluative viewpoint.

As will be seen below the starting point is that those responsible for design and

delivery must identify what it is they are expecting to achieve from any planned

session and whether those outcomes are in fact reached. This type of evaluation,

formative evaluation, measures whether the objectives of any particular lesson are

3 There is a great deal of published work on the perceived benefits of PLE – see for example: the collection of
articles in J. Robins (Ed.), Waking up to PLE : Public legal education, access to justice & closing the justice gap, Justice
Gap series, Solicitors’ Journal, 2013 and, more jurisdictionally and community-specific: J. Krishnan, S. Kavadi, A. Girach,
D. Khupkar, K. Kokal, S. Mazumdar, Nupur, G. Panday, A. Sen, A. Sodhi, and B. Shukla, Grappling at the
Grassroots: Access to Justice in India’s Lower Tier, Harvard Human Rights Journal, Vol. 27, 2014, 151.

4 A recently published study in the USA does however highlight the impact of one prominent PLE approach –
Street Law – on school pupils although even this study highlights the need for further research. See: Sean G.
Arthurs, Street Law: Creating tomorrow’s citizens today, Lewis and Clark Law Review, 19:4, 925.

IJPLE 2 (1) 2018

achieved and provides sign posts for how to revise instruction to ensure objectives are

achieved.

The template should enable those using it to replicate and, as necessary, adapt an

evaluable model for use in the field.

This paper then has a sample form appended as an example of how PLE sessions

might be evaluated.

Aims

Before turning to the models and impact measurement what specifically are the

purposes of evaluation? Evaluation is important for several related reasons. First and

foremost if planned outcomes and actual achievements are not clear how can the value

of what is being carried out be assessed – with a view to monitoring progress,

awarding possible credit and making improvements in future delivery? More

strategically perhaps, and as will be seen, many PLE initiatives rely on either public

funding, the backing of foundations and other charitable bodies and/or the input of

voluntary and not for profit personnel and initiatives. The relative lack of impact

evidence is surely an obstacle to securing financial and other resource backing? The

aim therefore of this article is to raise the importance of evaluation and to identify the

means by which any assessment may be carried out.

In a nutshell, can we identify:

• whether outcomes set for a particular PLE session or event have been achieved?

IJPLE 2 (1) 2018

• if future planning and delivery can be improved?

• whether there is impact – what difference does an improvement in legal

awareness actually make in the immediate, short-term or longer-term contexts?

• if and how the evidence can be used for development purposes – for example

funding, curriculum design, policy change?

• if findings of an empirical nature match what instinct and anecdote suggests –

thus building a body of evidence that has consistency and resonance.5

Challenges

First, what is meant here by PLE? At its most general, it is the raising of awareness of

law and the legal system through a variety of techniques and methods including the

provision of information (hard copy, electronic and face to face) as well as through a

more formal education interface such as one-off presentations or structured courses

and programmes. This takes into account the Street Law approach where target

audiences are introduced to a range of legal rights and responsibility issues through

interactive learning and teaching techniques often lead by lawyers, judges, and

trained law students under professional supervision.6

5 For example the claims made by the Bethel Institute through their learning pyramid – as discussed in A.
Kumar, Personal, Academic and Career Development in Higher Education - SOARing to Success, Routledge
Taylor & Francis, 2007

6 Some Street Law programmes are delivered by in-house teams largely based in NGOs, the initiators and
probably the most prolific in the USA and internationally being Street Law Inc., Washington DC. Others are law-
school based. For a discussion on the history and current prevalence of Street law programmes see: R. Grimes,
E. O’Brien, D. McQuoid-Mason and J. Zimmer, Street Law and Social Justice Education, in The Global Clinical
Movement: Educating Lawyers for Social Justice, F. Bloch (ed.), OUP, 2010. I do not suggest here that students

IJPLE 2 (1) 2018

There seems to be a degree of consensus that PLE should include, but not be limited

to, the dissemination of information. A point often made is that in addition to the

acquisition of knowledge, PLE involves fostering understanding and the development

of skills. It is also argued that PLE should aim to influence attitudes, and build

confidence. Distinctions need to be drawn between information per se and education

more generally. The latter commonly involves custom made and subject specific

material, delivered more than likely in an interactive way, whereas the content of the

former tends to be generic, with the direction of flow largely being one way, from

‘expert’ to recipient. It also is inclined to treat the audience as a passive receiver of that

information. Street Law is probably the best example of the education model and one

that shapes part of the evaluation template examples given below.

The quest for robust evaluative evidence is, however problematic. This is on a number

of levels. First, what is being measured and secondly, how might that measurement

be reliably and consistently done? It should also be remembered that studies involving

sensitive data may need ethics approval from professional or other institutional

bodies (such as universities whose staff or students carry out research) and that there

may be legislative requirements over the handling of such data.

One way to measure social impact is to study widespread changes over time.

However, the literature suggests that neither the measures nor the tools appear to exist

are or necessarily make, good teachers, but their use in many PLE programmes is a model that can have
multiple beneficiaries including the students themselves.

IJPLE 2 (1) 2018

in the PLE context (although there have been a number of evaluative studies in other

disciplines using defined methodologies).7

In UK Ministry-led task force on PLE in 2007, PLE presented the following challenges

that affect both planning and evaluation:

• PLE sessions or materials are typically part of some larger curricula

• PLE is unlikely to be clearly recognised by practitioners

• PLE is frequently tailored to achieving goals for users

• target ‘audiences’ for PLE initiatives vary widely by age, background, ability and

needs

• participants in PLE-related work are unlikely to recognise its nature or scope

• the goals of PLE may focus on different outcomes including changes in

behaviours, skills and attitudes. 8

In addition, the lack of a ‘like with like’ comparison and the nature of learning as

process rather than product make evaluation challenging to say the least.9 By way of

7 For example see: C. Brennan and K. Gallagher, Consumer Support Networks: improving consumer advice
in the UK, International Journal of Consumer Studies, 26 (3), 227, 2002; and Financial Services Authority,
Measuring financial capability: an exploratory study, FSA, 2005

8 PLEAS Task Force, Developing capable citizens: the role of public legal education, Department for
Constitutional Affairs, Russell Press, 2007

9 For the product vs. process debate see: S.J. Lachman, Learning is a process: toward an improved definition of
learning, The Journal of Psychology: interdisciplinary and applied, 131 (5), 1997, 477

http://onlinelibrary.wiley.com/doi/10.1111/ijc.2002.26.issue-3/issuetoc

IJPLE 2 (1) 2018

contrast few ask for the learning legitimacy of other forms of education to be proven.

Judging by the number of students reportedly disinterested in lectures, a similar study

on impact and the value-added component of learning passively might be usefully

called for!10

All of these factors or characteristics have measurable dimensions and may be inter-

related. The measurement of impact is therefore complex and difficult.

Measuring how and when improved awareness rises to the level of impacting a

participant’s ability to implement the newly acquired knowledge or skill is also

difficult to establish – particularly in trying to identify cause and effect. This has

previously been shown to be the case in legal self-help situations.11

Despite these challenges, it is suggested here that impact can be measured at different

moments in time and with different techniques and approaches. A combination of

methods and, where relevant, an amalgamation of findings coupled with subsequent

analysis can reveal valuable insights on impact and outcomes. Let us turn to this now.

10 The evidence for this is largely anecdotal but supported by much of the literature on learning and teaching.
See for example; G. Gibbs Twenty terrible reasons for lecturing, SCED Occasional Paper No. 8, Birmingham.
1981.

11 See: J. Giddings, M. Lawler and M. Robertson, The Complexities of Legal Self-Help, in: J. Robins, op cit, 50

IJPLE 2 (1) 2018

Models

Space in this article does not permit a detailed discussion of evaluative research

techniques. For those interested there is a wealth of material available elsewhere.12

Suffice it to say for present purposes that useful evaluation might consist of a mix of

quantitative and qualitative methods. These might start with a simple record of the

number of those attending PLE sessions or accessing PLE material. Entry and exit

questionnaires might be used to see what the target audience expected and then made

of the materials and/or presentations. If a particular PLE project is targeted at a specific

issue it may be possible to monitor impact in terms of the resolution of disputes, take

up of benefits or incidence of unwarranted activity such as domestic violence,

unlawful eviction or anti-social behaviour (in each instance in terms of both number

and participant perception). Focus groups could be used to prompt feedback and

discussion. Evidence could be gathered from PLE participants (the audience, the

12 See for example: S. Halliday and P. Schmidt, (Eds.). Conducting Law and Society Research: Reflections on
Methods and Practices, Cambridge University Press, 2009. There are also important studies in other subject
fields that attempt to evaluate impact – for example in youth justice, numeracy and financial literacy and in
consumer protection – see: J. Kenrick, Young People’s Social Welfare Need and the Impact of Good Advice,
Youth Access, 2007; Financial Services Authority, Towards a National Strategy for Financial Capability,
Financial Services Authority, 2003, Financial Services Authority, Measuring Financial Capability: an Exploratory
Study, Financial Services Authority, 2005 and Financial Services Authority, Levels of Financial Capability in the
UK: Results of a Baseline Survey, Financial Services Authority, 2006; and, Individuals’ awareness, knowledge
and exercise of employment rights, 2007 and the annual Competition Act and Consumer Rights survey Office of
Fair Trading (UK), respectively. Citizenship education more generally was the subject of a longitudinal study in
the UK, which began in 2001 and ran until 2009. The report on the findings can be found in: A. Keating, D. Kerr,
T. Benton, E. Mundy and J. Lopes, Citizenship education in England 2001-2010: young people’s practices and
prospects for the future: the eighth and final report from the Citizenship Education Longitudinal Study, National
Foundation for Educational Research, DFE, 2010.For an interesting account of PLE programmes and evaluative
methods see: M. Sefton, Public Legal Education Strategy (PLES) Task Force Scoping Report, Paper 2/03a,
Research Unit, Department for Constitutional Affair, 2006 (from which the citations to some of the reports
referred to in this footnote were taken).

IJPLE 2 (1) 2018

presenters and other stakeholders) before, at, after and following the event of release

of material in question. The appendix to this paper contains an example of how

session delivery might be evaluated. Of course any evaluation will hinge on what is

expected and that might vary considerably, for example if a PLE session was a one-off

event looking at a particular issue or was more overtly educational in an incremental

sense such as a short course or semester-long module.

Hard evidence may be relatively easily obtained when impact can be measured at the

time of or soon after the PLE input. The much more difficult question as to lasting or

longer-term significance of an increase in legal awareness inevitably requires

longitudinal studies which are, by their nature, time-consuming to carry out,

relatively expensive to administer and difficult to firmly establish cause and effect.

Some notable examples of highly successful studies in the legal awareness field (rather

than impact of PLE as such) can be found and make for interesting reading.13 The

Pleasance et al Causes of Action studies, based on a rolling programme, for instance

show the very high price paid for unresolved legal disputes (some GBP £3.5 million a

year and not including the human cost). It begs the question of the extent to which a

greater awareness of legal rights and responsibilities might mitigate against such

waste and the negative impact on personal well-being.

13 For example: H. Genn, Paths to Justice: What People Do and Think About Going to Law Hart, 1999 and P.
Pleasance, N. Balmer, A. Patel, A. Cleary, T. Huskinson and T. Cotton, Civil Justice in England and Wales: Report
of Wave 1 of the England and Wales Social and Civil Justice Panel Survey Legal Services Commission and Ipsos
Mori, 2011

IJPLE 2 (1) 2018

A suggested template and examples of evaluation in a Street Law context

Let us therefore take the issues and principles identified above and put theory into

practice by designing lesson plans that address the need for evaluation.

In order to stress the importance of a clear structure for PLE preparation and delivery

and to act as a guide for those developing PLE programmes a common template is

used throughout this book. This uses the following format:

Who is the PLE directed towards – audience?

What is covered – topic/subject matter?

What is the purpose – learning outcomes?

Where and when will it happen – location and day/time?

Who is doing what – preparers, presenters and any relevant supervision?

Content – what knowledge, skills and/or values are to be covered?

How will it be done – mode of delivery with timings (focuser, small groups, report

back, wind up and evaluation)?

What is needed – resources (materials, equipment, room(s) and people)?

How was it – for you and them?

What next – future progress?

IJPLE 2 (1) 2018

For illustrative purposes we will work with two examples here following a similar

template; first, there is a plan for delivery of a typical Street Law session and secondly

a plan for the use of hard copy legal information (which could also/instead be

delivered as part of a web-based PLE project).

Example 1 - work plan for a ‘typical’ Street Law session on stop and search provisions

under domestic law14

Institution: a law school running a credit bearing module called ‘Law in the community’

Item Details Timings Comments
Target audience Young people in a ‘special’

school for pupils with a
record of prior poor school
attendance.

Check age and
capacity of
pupils. Do risk
assessment.

Topic Police stop and search – your
rights and responsibilities.

Teachers at
school have said
that some pupils
are often subject
to stop and
search as they
have a
‘reputation’
amongst local
police.

Learning outcomes
(LOs)

For pupils:
1. To be able to

specify when a
police officer can
insist that a
person stops and

The learning
outcomes must
be SMART 16 and
for evaluation
purposes need to
be linked to the

14 This example was provided by delegates at the Ed O’Brien Street Law conference, Durban, South Africa, April
2016
16 Specific, Measurable, Appropriate or Assignable, Relevant or Realistic and Timely or Time-bound – see: G.
Doran, There's a S.M.A.R.T. way to write management's goals and objectives, Management Review (AMA
Forum) 70(11), 35, 1981.

IJPLE 2 (1) 2018

answers
questions

2. To know when it
is appropriate to
co-operate with
the police even if
the police cannot
legally insist

3. To identify what
can be done if the
police exceed
their powers or if
a person is
arrested.

For the law student
presenters:15

assessment
methods.

Location/day/time The stop and search lesson is
one of a series of ‘know your
rights’ presentations to be
delivered at the school on a
set day per week for one
hour.

60 minutes plus
travel time.

Timings will need
to be agreed with
the school staff
and normally fit
into an allotted
timetable space.

Preparation/delivery/
supervision (who?)

A team of 5 law students will
research the law and
applicable procedures and
will deliver the session at the
school. One staff member to
supervise preparation and
delivery.

The content and
delivery format
will be checked
by the supervisor
(an ex-practising
lawyer in the law
school).

Content 1. Police powers to stop
an individual/group

2. Police powers to
consequently search
persons/property

3. Consequences of
refusing to stop or be
searched

4. Possible sources of
help if arrested

5. Possible action/ route
for complaint if police
powers exceeded or
person otherwise

15 A separate set out learning outcomes are required if the law students are expected to achieve certain
educational goals. These will vary according to the module studied and the level at which it is offered e.g. first
year or final year of study. For reasons of space limitation these details are not included in the template here.

IJPLE 2 (1) 2018

unhappy with
treatment exceeded.

Delivery (how, logistics
and timings)

60 minutes using interactive
techniques.

Arrival and set up
- Introduction

and ice-
breaker – 5
mins

- Pupils to
observe role
play of police
stopping and
searching an
individual – 10
mins

- Brainstorm
what has
happened and
why? 15 mins

- Replay role
play (using
pupils) using
Forum Theatre
technique17 15
mins

- Discussion on
what to do if
the police are
thought to be
at fault – 10
mins

- Recap through
a quiz what the
basic legal
position is – 5
mins.

Timings are
approximate and
flexible but need
to be monitored.
If the session is
likely to overrun
additional time
may be found in
a following
session.

Resources Paper, pens, flip chart, props
for role play, room large
enough for role-
play/discussion, possibly a
prize and/or refreshments.

Outcomes (check if LOs
are achieved)

Quiz with prize(s).

Other evaluation
(target audience and/or
presenters)

Questionnaire for students to
complete before and after the
event; follow up session

17 A useful devise in which observers can stop then action during a role play if they think what is happening is
incorrect or could be improved. The objector then assumes the role of the person who was interrupted. For a
discussion of this technique see: http://dramaresource.com/forum-theatre/, accessed 1 June 2016.

http://dramaresource.com/forum-theatre/

IJPLE 2 (1) 2018

recapping on subject matter
of this presentation; inclusion
of stop and search questions
in pupils’ course assessment.

Review for further
development

Discussion with school staff
on value of session; debrief
for law students post-event.

Link with future
projects/sessions

Part of on-going set of
presentations. Development
of similar programmes at
other venues.

Example 2 - work plan for a tool kit on self-representation before courts and tribunals18

Institution responsible: Advocacy rights – an NGO supporting unrepresented litigants19

Item Plan Comments
Target audience Advocacy groups and their

members.
Important to make
links with
local/regional/national
groups and any
relevant networks.

Topic Representing yourself in civil
courts and tribunals.

Learning outcomes (LOs) For users:
1. To know where the local

courts and tribunals are
located

2. To know how to start or
respond to the issue of
legal proceedings

3. To be able to recognise
what will happen next
once proceedings are
issued

4. To be able to gather and
present relevant evidence

5. To know how to address
the court or tribunal

6. To know where to go to
for help if required.

The purpose of the
toolkit is to provide an
overview of legal
proceedings so that a
non-lawyer can
recognise what
happens and why in
legal proceedings
before courts and
tribunals.

18 This example is included to contrast with the more typical Street Law presentation
19 This is a fictional NGO but based on various manifestations in the UK and other countries for example – the
Rotherham Advocacy Partnerships. More information on this can be found at:
http://www.rotherhamadvocacy.org.uk/, accessed 1 June 2016.

http://www.rotherhamadvocacy.org.uk/

IJPLE 2 (1) 2018

Location/day/time Not relevant

Preparation/delivery/supervision
(who?)

The preparation will
be supervised by an
experienced practising
lawyer.

Content 1. Overview of jurisdiction of
courts and tribunals

2. Issue of proceedings
3. Preliminary hearings and

orders
4. Evidence
5. Trial
6. Remedies and other court

orders
7. Settling cases out of court
8. Sources of help for

litigants in person.

Delivery (how, logistics and
timings)

Booklet available in hard copy or
in an e-version and available on
NGOs’ websites.

Resources Distribution points in publically
accessible places (e.g. libraries and
schools). Links to websites.

Outcomes (check if LOs are
achieved)

A questionnaire to be
incorporated in the booklet/on
website.

Other evaluation (target
audience and/or presenters)

Focus groups of users and
court/tribunal staff.

Review for further development Informed by focus group. A
suggested template for user/other
stakeholder feedback is appended
to this article. Possible link with an
academic institution/government
department to conduct empirical
and longitudinal ‘impact’ study.

Link with future
projects/sessions

Future plans depend on nature of
feedback.

IJPLE 2 (1) 2018

Summary of evaluation ‘best practice’

Potential roles for PLE include the raising of awareness of rights and responsibilities

generally, preventing problems from arising or escalating and assisting (directly

through improving self-help capacity or indirectly through enhancing the ability to

identify a problem) and then recognising routes through which further assistance can

be sought, such as referral to another agency (including a lawyer).

In any version in which PLE is targeted at individuals or groups with problems and

issues, PLE might also focus on those who assist others – for example a training of

trainers or development of para-legal model.20 In either case when working with the

target audience evaluation is still critical in order to monitor the educational process

as well as to allow for fine-tuning and other future development.

Whilst evaluation is problematic, particularly in terms of robust methodologies and

the inevitably long-term nature of impact studies, a careful review of expected

outcomes and utilisation of a range of research techniques may enable the impact of

PLE to be monitored and analysed for the long-term benefit of wider community.

Clearly there is much work to be done on developing, funding and conducting PLE

impact research. This brief paper is intended to help kick-start that process. If anyone

is interested in working on PLE evaluation please contact the editors of this journal.

20 See for example: P. Patel, Z. Douglas and K. Farley, Learning from a ‘paralegals’ intervention to support
women’s property rights in Uganda, international Centre for Research on Women, 2014, available at:
http://www.icrw.org/sites/default/files/publications/ICRW%20Uganda%20Paralegals_final.pdf, accessed 1
June 2016

http://www.icrw.org/sites/default/files/publications/ICRW%20Uganda%20Paralegals_final.pdf

IJPLE 2 (1) 2018

Appendix

Evaluation Template Form

Please complete the following set of questions in as much detail as possible for your PLE
project Please return completed forms, via e-mail, to XXXXX, by XXXXX.

Project Reference:
Project Title:
Lead Organisation:
Project Start Date:

a. Please provide a brief summary of the aims of the project and state how
these will be, or have been, achieved:

b. Which of the sub-themes does the project fall into? (please refer to
guidance attached but please amend if these are incorrect and add the
groupings by client and category.)

Client Group:
Category:
Project objective:

c. How has the project provided concrete and material help for targeted
clients (this should be based on real examples and should include the
number of clients helped and the level of help delivered)?

Example(s)

d. What have been the long-term impacts of the project, if any, and how has
the project secured these improvements?

e. Are there aspects of the project that could be replicated elsewhere, if so,
what are they and under what circumstances would it be appropriate to
replicate these (this could include information materials that have been
produced by the project, for example)?

f. What impact has any funding (including matched funding) had on the
project? Has it added value to the project, is so, how

IJPLE 2 (1) 2018

g. What links, if any, has the project had or made to other initiatives
(government/council/otherwise)?

h. What difficulties, if any, has the project experienced?

i. What plans are there for this or related projects in the future and what is
required to achieve these?