Evaluating the educational impact of information and communications technology in Irish schools
Jean McNiff
A paper presented at the 8th International Conference on Teacher Research, Richmond, BC, April, 2001
My work context
I have been working across the island of Ireland for ten years. I work as an independent researcher and consultant. My main work has been developing professional learning courses leading to the award of higher degrees in the Republic of Ireland, and for this I am in partnership with British universities who accredit the courses. To date, nearly 70 educators have been, or are about to be, awarded Masters degrees for their studies of their own educational practices, a figure that is significant for Irish education, given that there are only some three and a half million people in the south. I also supervise twelve PhD candidates. As well as my work with universities, I undertake independent consultancy work, and in this capacity I am active in Northern Ireland, where I am involved in the Education for Mutual Understanding initiative.
Background to the research
In autumn 2000 I was invited, in a private capacity, to conduct an evaluation for the National Centre for Technology in Education (NCTE), an agency of the Irish Department of Education and Science. The NCTE has been in existence since 1997 and has been the body responsible for overseeing the development of a coherent initiative to ensure that all Irish students would ‘have the opportunity to achieve computer literacy and to equip themselves for participation in the information society’ (Government of Ireland, 1997: 2).
In this paper I would like to explain how the evaluation developed, and how the NCTE aimed to ensure that its work could be judged in terms of its educational impact. I also want to explain how I see evaluation as part of the personal professional learning of the participants involved in the programme which is being evaluated, and to offer a view of evaluation that it is indeed an aspect of a practice which may be seen as research. Evaluation, for me, involves the presentation of evidence which enables the practice to be judged in terms of its educative influence in the lives of practitioners.
In presenting this account of work in progress, I am hoping to show that I can claim that I am contributing towards an improvement in the quality of educational experience for teachers, and that they in turn are contributing to the quality of educational experience for their students in classrooms; and I am inviting your critical scrutiny of my claim as part of my own self evaluation.
Information and communications technology in Ireland
Information and communications technology (ICT) was first introduced into Irish schools in the 1980s. In 1984, the Department of Education initiated a pilot project ‘Computers in Education’, in 34 schools representative of all types of primary schools over a two-year period (Healy, 2001). This project was generally viewed positively (see for example the Irish National Teachers’ Organisation, 1996).
However, the initiative was not followed through in a systematic way, some of the reasons being, according to the Irish National Teachers’ Organisation (1994), the lack of funding and lack of training for teachers. In its 1997 document ‘Schools IT 2000 – A Policy Framework for the New Millennium’ (Government of Ireland, 1997: 2), the Department of Education and Science admitted that ‘Ireland lags significantly behind its European partners in the integration of information and communication technologies (ICTs) into first and second level education.’
This realisation, particularly in the context of a booming economy and massive investment from overseas, led to an awareness of the need to promote ICT in schools to ensure a highly literate and competitive workforce for the new century. ‘The need to integrate technology into teaching and learning right across the curriculum is a major national challenge, which must be met in the interests of Ireland’s future economic wellbeing’ (Government of Ireland, 1997: 2) In November 1997, therefore, the Taoiseach and the Minister for Education and Science launched Schools IT 2000.
This initiative would put various structures in place at national and regional level:
· an Educational ICTs Co-ordination Unit in the Department of Education (now the Department of Education and Science);
· a National Centre for Technology in Education in Dublin City University to manage the implementation of Schools IT 2000 (established until 2001);
· A Schools IT 2000 base in at least ten Education Centres to provide support to schools on a regional basis (Government of Ireland, 1997: 5).
My work was with the NCTE.
The work of the NCTE
The work of the NCTE is organised in terms of strands. It aims to fulfil the recommendations of the 1997 report to establish a technology infrastructure, a skills infrastructure, and a support initiative. The immediate schools-focused aspect of the work is in relation to the strand Schools Integration Project (SIP), and this is also where I am involved.
SIP is an initiative that invited all first and second level schools to submit a proposal to the NCTE for funding and support to develop a particular ICT project within the school or a cluster of schools. In the event several hundred schools applied to be involved. An appointed selection committee identified 40 projects to be supported in the first instance and a further 41 projects have been supported.This brings the total to 81projects. Some of these projects therefore have been in existence for two years, and have produced impressive artefacts in the form of booklets, study guides, resources, information packs, and so on. Most schools have their own web-site. SIP on line aims to publish the work of the participating schools. Other schools did not make their application for support immediately, and others decided to implement their projects at various stages. Consequently projects around the country are all at different stages in their implementation and completion.
Part of the teacher education provision was to provide the technology, skills and support infrastructure mentioned above: for example, to offer inservice courses in ICT, as well as schools-based support for the development of web sites, schools-based computer suites, and other aspects in schools. A massive programme of inservice provision came into being, involving a team of inservice providers based at NCTE headquarters, regional education centres, and the national body of ICT advisers.
Setting up the evaluation process
In September 2000 I was invited by the Director of the NCTE to conduct an evaluation of the Schools Integrated Project. The Director’s concern was to provide evidence that the NCTE’s mandate to provide a technology infrastructure, a skills infrastructure, and a support infrastructure had been fulfilled. My own view is that such infrastructures do not count for much unless they can be shown to have influenced the quality of educational experience for children in classrooms. This also is my view of teacher education (similar to that of Michael Huberman, 1992), that unless teachers can show that their practice has influenced the learning experience of students, teacher education fails to fulfil its potential as a force for the kind of personal and social renewal which can lead to the development of good orders. Consequently, I suggested to the Director that we invite teachers in classrooms (in the immediate instance, the SIP coordinators in schools across the country) to produce accounts of their own self studies as they tried to improve their practice for their students’ educational benefit. He was in agreement, and we began to develop an evaluation strategy that would focus on supporting the 110 SIP coordinators to reconstruct their existing work in terms of an action research framework which focused on the production of evidence in support of the coordinators’ claims that they had influenced the quality of learning in classrooms.
Methodology
The kind of methodology I wanted to promote was one which values people rather than programmes (see also Kushner, 2000). I believe that the dominant kind of propositional theory which views organisations and organisational structures as abstract entities is not the most useful in helping people to understand their work and make practical decisions about how to improve their work. For me, organisations are people, and people need to make decisions themselves about what their lives and work are about (McNiff, 2000). Similarly, evaluation refers to a process of people monitoring and evaluating their own work. Education, in my opinion, is about helping people to see that they are the live agents in social change processes. Social change does not come about because a policy-maker mandated it; social change begins with people, and happens because individuals see the need for, and want to change (Rizvi, 1993). If social change begins in people’s minds (John Hume, leader of the SDLP in Northern Ireland frequently comments that decisions about decommissioning processes begin in people’s minds, not in board rooms), those people themselves need to accept the responsibility of being accountable for their own living. In terms of evaluation processes, this means that participants need to take responsibility for their own self evaluation and produce accounts to show that their life work counts for something in the education of their students. Nor can their claims to have influenced the educational experience of their students remain at the level of opinion or hearsay; those claims must be supported by the production of validated concrete evidence to show that what practitioners say can be taken as authentic and trustworthy.
I proposed therefore that all parties in the SIP initiative should undertake their own self evaluations. I saw my work as coordinating an effort to provide organisational support to enable the production of accounts by all participants to show that the SIP initiative was fulfilling the aims of the NCTE to provide a technology infrastructure, a skills infrastructure, and a support infrastructure, and to show that this provision was educationally beneficial. My commitment for full participation by all members of the organisation was animated by a vision of organisational learning for social change; but achieving a situation which can be called organisational learning means ‘tap[ping] people’s commitment and capacity to learn at all levels in an organization’ (Senge, 1990: 4; emphasis in original). From my experience (see above), I had found that classroom teachers really welcomed the opportunity of investigating their work and producing public accounts to show that their work was having educational impact; however, suggesting that people at all levels in an organisation should evaluate their own practice was more problematic, and this belief was soon to be strongly reinforced. In this paper, however, I do not intend to go into the political aspects of who agreed to undertake their self studies, and why; there is not space, and my own research is still in process. However, I do hope to produce a text around this project which will engage with the ideas of power and authority in organisations, and who decides who should be accountable for their practice.
Here I want to focus on what is being achieved for educational benefit, and to invite your critical comment as validation for my own claims that I am supporting organisational learning such that the quality of education for students in schools is being enhanced.
Implementing the evaluation process
I arranged with colleagues in the NCTE head office to set up an infrastructure to support schools-based SIP coordinators. This infrastructure took the form that:
· I met with a core team of 12 SIP coordinators who had been seconded from teaching to take up full-time posts as regional supporters;
· I arranged to meet with all schools-based SIP coordinators in a variety of settings: a plenary session of all 110 met during November 2000, and I met with all regional groups in January 2001;
· I supported the core team in supporting their regional groups of SIP coordinators;
· I provided materials in the form of handouts and books;
· I was available to support via telephone, email, postal correspondence, personal meetings.
(The initial plan was that this schedule would be implemented systematically through November 2000–May 2001. In the event, the schedule was severely disrupted in February 2001 because of stoppages of teacher meetings as part of the industrial action of the Association of Secondary Teachers, Ireland, in support of claims for improved pay and conditions; and the strict limitations put on teacher meetings because of the foot and mouth crisis. I had to be creative in finding ways to continue to support SIP coordinators, and I found myself working at a distance, sometimes from my home in England, and liaising closely with the two Project Officers based in NCTE head office.)
The evaluation process I adopted throughout was the one devised by Jack Whitehead of the University of Bath:
What is my concern?
Why am I concerned?
What do I think I can do about it?
What will I do about it?
How will I gather evidence to show that I am influencing the situation?
How will I ensure that any judgements I come to are reasonably fair and accurate?
What will I do then?
(see Whitehead, 1993, 2000; see also McNiff, 1988/92; McNiff et al., 1996).
I invited SIP coordinators to frame their work in terms of these questions. Many were delighted at the opportunity to place a theoretical framework around their practice, and evaluation comments about the usefulness of such provision included:
‘putting a design structure on our SIP projects.’
‘I feel I now have a clear idea on how to proceed.’
One colleague wrote:
‘The main thing for me was the fact that it put the ‘evaluator’ at ease and took away the terrible harsh interpretation that is associated with the term “evaluation”.’
(These comments are taken from evaluation feedback sheets at the end of regional presentations on the process of evaluation and how to write an evaluation report)
I reinforced these presentations with detailed guidelines on how to write up an action research report. These appear tohave been well received.
What claims am I making?
I am claiming that SIP coordinators are finding my support useful in framing their own work as research projects. The discourse seems to have changed nationally from ‘SIP project’ to ‘SIP action research project’. There is everywhere talk of identifying criteria whereby evidence can be produced to support claims to knowledge of improved learning.
I am claiming that SIP coordinators are experiencing new levels of confidence in a view of practice as research. Evaluation comments include:
‘I feel I know what I am doing now.’
‘I now see my work as research. I am aware that I need to produce evidence to show that I am influencing students’ learning.’
I am claiming that some organisational learning is taking place. Core team members have sent me reports to suggest that they are evaluating their practice as supporters. One colleague (who must at present remain anonymous) writes:
‘I am looking at my practice in terms of whether SIP coordinators can see the need to produce evidence to show that they also are identifying criteria to judge their practice.’
However, my hope to involve all members of the organisation remains only a hope. Not all members of the organisation have expressed a desire to undertake their own self evaluations (see above; I shall be exploring these issues in a comprehensive text). At least I am able to produce validated evidence that many members of the organisation are engaging in their self evaluations, and that they can show their educational influence in classrooms.
I am claiming the effectiveness of an action research approach to organisational evaluation. Further, I am claiming that when evaluation is conducted at the level of individual monitoring and critical engagement with practice, that this can then lead to whole school development and organisational learning, which can in turn lead to social improvement. This I acknowledge is a massive claim, but one which I aim to support by coordinating the compilation of a research archive of some 80 research reports which take the form of self studies, produced by SIP coordinators, to show how they improved the quality of learning for students in classrooms.
Where are we now?
When I return to England on 16th April I hope to collect from the post office a bundle of reports. The schedule agreed with the NCTE head office and SIP coordinators is that first drafts of reports should be written by 6th April. I will then read and respond to these first drafts, return them to their authors, and follow up that process with a further round of regional meetings in early May. Final reports will be produced by 21st May. My task then is to compile the reports into a coherent archive, and to produce a report to explain the process of evaluation and how it might be seen as supporting the NCTE’s claims to have supported teachers in improving the quality of learning for students through ICTs.
Implications
There are many implications arising from this research project, and I will detail them in my later text. Provisionally I would say that the research shows the value of practitioner-based research for school improvement, and, at a wider level, for social change. I do believe that social change has to happen initially in terms of individuals’ commitments to take the responsibility for improving their own situations, and that this begins by their reflecting critically on how they might improve their own work. I do believe that it is the responsibility of each one of us to put our own house in order.
I also hope to make a case for practitioner-based research as a powerful form of professional learning. Siobhán Ní Mhurchú (2000: 74), a participant whom I supported for her MA studies, wrote:
‘I believe that if you can give people hope when dealing with a particular problem, if you can show them a practical way, the problem can be overcome from within people’s own resources. … If action research were adopted as a form of professional development on a national scale, teachers’ own self perceptions would rise and we would be looking at an invigorated workforce that had the confidence to take a pro-active role in improving society. I believe that action research might be a more viable option for the Department of Education and Science in supporting professional development as it taps into a source of energy and goodwill that would enable people to innovate and manage change for themselves in their own educational environments’ (Siobhán’s work is reproduced in McNiff, in production).
At the level of what counts as educational knowledge and where might be the locus of that knowledge, I do believe that the research shows the implications for the development of a body of knowledge which is constituted of the descriptions and explanations that practitioners offer for their own educational practice (see Whitehead, 2000; see also actionresearch.net). While dominant propositional theories help us to understand what needs to be done, the propositional form of theory needs to be embedded within the living educational theories of practitioners as they begin to investigate their practice by asking questions of the kind, ‘How do I improve my work?’ (Whitehead, 1989).
I believe this is happening in the contexts where I work. I know that my own learning has been influenced by the ideas of my colleague Jack Whitehead, and I believe that such professional learning relationships are at the heart of how supporters themselves are able to support wider institutional and social change. I can make the claim that I am helping people in Ireland to reconceptualise their own professional knowledge base by citing the words of practitioners such as Siobhán Ní Mhurchú and Caitríona Mc Donagh, another MA course member:
‘I have come to understand that I can contribute to a much wider body of knowledge. [My dissertation] is part of that body of knowledge which is transforming what the research community understands as legitimate theory. As I have influenced the quality of my professional learning in my classroom and my workplace, so I hope also to influence the wider community of researchers in their understanding of how knowledge is produced and used within practitioners’ individual and collective practices. I have become aware of my own potential for influence, both in local and wider contexts, and I intend to take every opportunity to share my learning in my hope for a more democratic and caring approach to education in schools.’ (Ní Mhurchú, 2000: 73).
‘The awesome respect in which I had held educational research and theories prior to my engagement with [my MA studies] has given way to a new critical understanding of dilemmas of practice and theory. … Prior to this project I would not have considered my educational values or epistemology of practice worth sharing within the institution of the school. Living through the process of this research I have found a voice in the educational world. This teacher voice was seldom heard. The practising teacher tended to bow to academic educational theorists, to psychologists, to departmental inspectors, to parent bodies, yet where is the teacher’s voice heard? Teacher craft was not valued by institutions of education professionals. This form of research has given colleagues and me a voice and method to articulate our theories’ (Mc Donagh, 2000: 75).
In the case of these two colleagues I can show that I have influenced the quality of professional learning which judges its effectiveness in terms of whether the quality of student learning experience is improved. It now remains to be seen whether my influence has had an impact in the evaluation of the SIP projects. Time will tell. Like all good soap operas, I am leaving this presentation on a note of high closure. Next week I miss the episode in the soap Eastenders, where the answer ‘Who shot Phil Mitchell?’ is answered, and I have to wait until I get back to England to find out. Similarly, you will have to wait until the final documentation is available to make judgements about my claims; but I am hoping that the pile of reports waiting for me on my return will contain evidence to ensure that today’s presentation has been worthwhile for us all.
References
Government of Ireland (1997) Schools IT 2000: A Policy Framework for the New Millennium. Dublin, Department of Education and Science.
Healy, M. (2001) ‘How do I support the introduction of IT in my school?’ Unpublished MA dissertation, Thurles, University of the West of England, Bristol.
Huberman, M. (1992) ‘Critical Introduction’ in M. Fullan, Successful School Improvement. Buckingham, Open University Press.
Irish National Teachers’ Organisation (1994) Computers in Education: A Pilot Project. Dublin, INTO.
Irish National Teachers’ Organisation (1996) Information Technology in Irish Primary Education: Issues and Recommendations. Dublin, INTO.
Kushner, S. (2000) Personalizing Evaluation. London, Sage.
Mc Donagh, C. (2000) ‘Towards a Theory of a Professional Teacher Voice. How can I improve my teaching of pupils with specific learning difficulties in the area of language?’ Unpublished MA dissertation, Dublin, University of the West of England, Bristol.
McNiff, J. (1988/92) Action Research: Principles and Practice (first edition). Basingstoke, Macmillan / London, Routledge.
McNiff, J. (2000) Action Research in Organisations. London, Routledge.
McNiff, J. (in preparation) Action Research: Principles and Practice (second edition). London, RoutledgeFalmer.
McNiff, J., Lomax, P. and Whitehead, J. (1996) You and Your Action Research Project. London, Routledge.
Ní Mhurchú, S. (2000) ‘How can I improve my practice a teacher in the area of assessment through the use of portfolios?’ Unpublished MA dissertation, Cork, University of the West of England, Bristol.
Rizvi, F. (1989) ‘In defence of organizational democracy’ in J. Smyth (ed.) Critical Perspectives on Educational Ledership. London, Falmer.
Whitehead, J. (1989) ‘Creating a living educational theory from questions of the kind, “How do I improve my practice?”’, Cambridge Journal of Education 19(1): 137–153.
Whitehead, J. (2000) ‘How do I improve my practice? Creating and legitimating an epistemology of practice’, Reflective Practice 1(1): 91–104.
© Jean McNiff
e-mail: 106642.1700@compuserve.com
