Difference between revisions of "Evaluation"

From Competendo - Digital Toolbox
Jump to: navigation, search
m
m
 
(215 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
<div class="left-column-contentinner">
+
<div class="methodpage-content">
[[File:Created-by.png | 20px | Created By ]] [[User:Nils.zimmermann | N. Zimmermann]]/ H. Fahrun
+
<div class=teaser-text><div style="background: #eee; float: left; padding: 0px 5px 0px 5px; margin-right: 5px;"><i class="fas fa-arrow-left">&nbsp;</i> [[After]]</div>  
 +
[[File:Evaluation.png | right | 200px]]Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. Evaluation can be ''formative'' - that is, taking place during the development of a concept or during a workshop - or ''summative'', drawing lessons from a completed action or process.</div>
  
<div class=teaser-text>
+
<div class=left-box>
[[File:Evaluation.png | right | 200px]]Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. Evaluation can be ''formative'' - that is, taking place during the development of a concept or during a seminar, with the intention of improving the value or effectiveness of the proposal of the project. It can also be ''assumptive'', drawing lessons from a completed action of the project or a finished seminar.</div>
+
===Evaluation===
 +
Systematic collection, review, and use, of information and evidence to represent, evaluate and report learning, in different ways, for different purposes.
 +
</div>
 +
 
 +
<hr class=boldline>
 +
==Focuses of Reflection==
 +
Already during [[Planning]] facilitators were considering goals, contents, methods, topic, needs and the environment.
 +
 
 +
During a process or training it is necessary to check if the process is still inline with the plans or if priorities have changed or needs are different, or a process should take a general different direction. There are several purposes for evaluation:
 +
 
 +
<div class=left-box>
 +
 
 +
===Why are we assessing?===
 +
Assessment for '''measurement''' requiring a stable metric to identify starting
 +
points and distance travelled.
 +
 
 +
Assessment for '''selection''' requiring rank order or criterion-referenced
 +
 
 +
Assessment to '''diagnose''' next learning steps
 +
 
 +
Assessment to '''evidence''' impact or competence
 +
 
 +
Assessment to '''evaluate the learning''' approaches taken, educator or organisational performance.
 +
 
 +
</div>
 +
<div class=left-box>
 +
 
 +
===Why debriefing from a learner's perspective?===
 +
Debriefing has particularly valuable benefits in activities in the context of education for democratic citizenship and human rights education (EDC/HRE):
 +
* Debriefing is ''participatory learning''. It addresses learners as self-responsible subjects capable of reflecting, inquiring, comparing, connecting, and planning their own learning experiences.
 +
* Debriefing is necessary for ''sustainable learning''. When scrutinising their learning experience and dealing with different perspectives in a group discussion, learners gain additional information, deepen understanding, develop further competences, and gain self-confidence.
 +
* By becoming aware of their developmental needs and by recognising their own learning style, they gain knowledge and a ''critical understanding of themselves''.
 +
* Debriefing turns learners into ''owners and co-authors of their learning progress''. Games require learners to get involved actively, deeply, and at times personally. Debriefing guides and enables them to turn these, and probably also future, experiences into a fruitful learning outcome. Debriefing thus develops autonomous learning skills.
 +
 
 +
====Democracy Dimension====
 +
Debriefing can foster learning in all three dimensions of democracy education:
 +
*Learning ''about democracy'': Which were the topics of the game-based learning activity?
 +
* Debriefing fosters learning ''through democracy'': Learners see (practical) meanings and consequences of what happened, of how they acted and reacted. It offers participants a structured and safe environment to practice democratic and free exchange.
 +
* Debriefing fosters learning ''for democracy'': Learning about the relationship with their real-world experiences, insight in social developments and inspiration for social action.
 +
</div>
 +
 
 +
<hr class=boldline>
 +
 
 +
==Assessment - in Between or at the End?==
 +
While '''debriefing''' is usually integrated in each method implementation or program unit, '''summative and formative''' evaluation activities are defined in an evaluation and assessment strategy of a whole process. They aim to collect data beyond a single unit and to allow reflection on more fundamental aspects (overall progress, competence gain, lessons learned, group process...).
 +
 
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: Debriefing, summative and formative assessment" data-collapsetext="- Collapse">
 +
<div class=left-box>
 +
===Debriefing===
 +
Conducted at the end of a single learning unit with focus on reflection and assessment of what happened and was learned during this unit.  
 +
</div>
  
 
<div class=left-box>
 
<div class=left-box>
===Evaluation===
+
===Formative Assessment ===
Structured interpretation and giving of meaning to predicted or actual impacts of proposals or results.
+
Serves learning progression, supporting development over time, integral and repeating part of a learning process.
 +
 
 +
<div style="color: #ccc; font-size: 36px; text-align: center;">
 +
<i class="fas fa-arrow-right"> </i> <i class="fas fa-arrow-circle-down"> </i> <i class="fas fa-arrow-right"> </i> <i class="fas fa-arrow-circle-down"> </i> <i class="fas fa-arrow-right"> </i> ...</div>
 +
</div>
 +
 
 +
At the end of a phase or learning process, other reasons for evaluation may also come into play. You may want to demonstrate a person's learning progress or performance, or you may want to check whether the different goals have been achieved.
 +
 
 +
<div class=left-box>
 +
 
 +
===Summative Assessment ===
 +
Conducted at the end of a learning process.
 +
 
 +
<div style="color: #ccc; font-size: 36px; text-align: center;">
 +
<i class="fas fa-arrow-right"> </i> ... <i class="fas fa-arrow-right"> </i> ... <i class="fas fa-arrow-right"> </i> ... <i class="fas fa-arrow-right"> </i> ... <i class="fas fa-arrow-right"> </i> ... <i class="fas fa-arrow-circle-down"></i></div>
 +
 
 +
</div>
 
</div>
 
</div>
  
==Focuses of reflection==
+
 
During [[Planning]] facilitators were considering goals, contents, methods, topic, needs and the environment. From time to time it's necessary to check if the process is still inline with the plans or if priorities have changed or needs are different, or a process should take a general different direction.
+
<hr class=boldline>
<div class="mw-collapsible mw-expanded" data-expandtext="+ Read more" data-collapsetext="- Collapse">
+
 
 +
==What are We Assessing?==
 +
In process-oriented and non-formal learning processes we very often reflect how successful we facilitated a learning unit or to what extend the element of the process contributed to the intended goals. The '''aspects in assessment''' are similar to the criteria for planning. For instance, one could follow this logics:
 +
 
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: What are we assessing?" data-collapsetext="- Collapse">
  
 
[[File:Goal-content-method-space.png | 550px | center]]
 
[[File:Goal-content-method-space.png | 550px | center]]
  
  
<table>
+
<div class=left-box>
<tr>
+
 
<td>
 
 
===Goal Achievement===
 
===Goal Achievement===
 
* Group: Regarding the aimed interaction and cooperative learning in the group
 
* Group: Regarding the aimed interaction and cooperative learning in the group
 
*Topic: The coverage of the foreseen topical aspects
 
*Topic: The coverage of the foreseen topical aspects
* Individual: Learning/development goals (in particular competency development, see below)
+
* Individual: Learning goals and competence development (see below)
* Process: The proceeding of the collective learning process
+
* Process: The proceeding in the steps of the collective learning process
</td>
+
</div>
</tr>
+
<div class=left-box>
<tr>
 
<td>
 
  
 
===Methodology===
 
===Methodology===
 
*The choice of methods and their mix
 
*The choice of methods and their mix
*The ability of facilitators to implement their methodological concept
+
*The ability of facilitators to apply methodology
 
*Process moderation
 
*Process moderation
</td>
+
</div>
</tr>
+
<div class=left-box>
<tr>
 
<td>
 
 
===Content===
 
===Content===
 
* Coverage of the necessary thematical aspects at a satisfactory depth
 
* Coverage of the necessary thematical aspects at a satisfactory depth
* Inclusion of knowledge from the field participation/citizenship/democracy and how it was connected to the goal of the training
+
* Inclusion of knowledge from the field participation-citizenship-democracy and how it was connected to the goal of the training
 
+
</div>
</td>
+
<div class=left-box>
</tr>
 
<tr>
 
<td>
 
  
 
===Space/Context===
 
===Space/Context===
Line 54: Line 117:
 
* With special attention to special needs
 
* With special attention to special needs
 
*Inclusion of the local environment: in seminar work, in topical aspects
 
*Inclusion of the local environment: in seminar work, in topical aspects
</td>
+
</div>
</tr>
+
<div class=left-box>
<tr>
+
===Competence development and transfer===
<td>
+
* Participant's acquisition of competences
 
+
* Participant's development towards self-empowerment and democratic participation
===Competency development===
+
* Your acquisition of competences as a facilitator
* Participant's acquisition of competencies
+
* Transferability of learning aspects to the real life
* Participant's development towards feeling self-empowered and their ability to democratic participation
+
</div>
* Your acquisition of competencies as a facilitator
+
<div class=left-box>
</td>
 
</tr>
 
 
 
<tr>
 
<td>
 
  
 
===Management===
 
===Management===
 
*Event management, logistics, problem-solving
 
*Event management, logistics, problem-solving
 
*Cooperation within the facilitation team
 
*Cooperation within the facilitation team
</td>
+
*Were expectations met, needs respected?
</tr>
+
</div>
 +
 
 +
<hr class=boldline>
 +
 
 +
<div class=left-box>
 +
 
 +
===Reflection/Debriefing Scheme for a Single Method===
 +
While it is imperative that summative evaluation end up with enough data to cover all of these analytic areas, we also need to consider an approach to evaluation for each individual method we use that follows a basic standard. This or similar suggestions have proven workable for many educators in non-formal education.
 +
 
 +
 
 +
<hr class=simpleline>
 +
====Block 1: Facts and Feelings====
 +
What happened?<br>
 +
What did you feel about it?<br>
 +
(in regards to: yourself, group, topic, process..)
 +
 
 +
<hr class=simpleline>
 +
 
 +
====Block 2: Learning====
 +
What did you learn?<br>
 +
(in regards to: yourself; group & relationships developed during the activity; topics & concepts employed; the process..)
 +
<hr class=simpleline>
 +
 
 +
====Block 3: Transfer====
 +
What can you conclude for you daily life?<br>
 +
What can you do (differently) in regards to the problem?<br>
 +
(connection to the real life of the learners)
 +
 
 +
<hr class=simpleline>
 +
 
 +
<youtube center>gZVwykWYGcM</youtube><br>
 +
Source:  [https://demogames.eu Demogames]
 +
</div>
 +
 
 +
</div>
 +
 
 +
 
 +
<hr class=boldline>
 +
 
 +
==Evidence==
 +
To make a reasoned judgment, one needs a '''data base''' in addition to predetermined focus and purpose. Therefore, assessment is always dependent on good evaluation material.
 +
 
 +
In more traditional academic assessment, educational purposes have tended to focus on testing knowledge and understanding in one hit, examination-style papers, which are dominated by written evidence.
 +
 
 +
However, increasing awareness of the limitations, potential bias and ‘wash back’ of formal examinations has strengthened calls for more purposeful and meaningful assessment methods, drawing from richer and more diverse evidence, over time.
 +
 
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: Gaining evidence" data-collapsetext="- Collapse">
 +
<div class=left-box>
 +
===Within your process:===
 +
*Debate,
 +
*Declarative methods,
 +
* Interviews,
 +
* Observation,
 +
* Portfolio method,
 +
* Learning diary,
 +
* Autonomous reflection groups of learners,
 +
* Presentation,
 +
* Simulation
 +
* Evidence extracted from (project)work,
 +
* Tests
 +
After: CEDEFOP<ref name=CEDEFOP2>European Centre for the Development of Vocational Training (Cedefop): [https://www.cedefop.europa.eu/EN/Files/4054_en.pdf European guidelines for validating non‑formal and informal learning];; p. 59 ff.</ref>
  
</table>
+
===At the end of a learning process may stand....===
 +
 
 +
* Competence description along standards for programs or learning events
 +
* Certificates for participation
 +
* Diplomas
 +
* Qualified competence description
 +
* A completed portfolio
 +
* Learning diary<ref name=cedefop3>see as well: The European Centre for the Development of Vocational Training (Cedefop): [http://www.cedefop.europa.eu/EN/Files/4054_en.pdf European guidelines for validating non‑formal and informal learning]; p. 19</ref>
 +
</div>
 
</div>
 
</div>
  
Line 81: Line 207:
 
<hr class=boldline>
 
<hr class=boldline>
  
==Concept: The Method Mix==
+
==Method Mix==
 
The quality of results depends on the methodology of data collection for reflection. In particular, a range of different ways to assess and interpret the needed data should be included, according to the principle of [[Method Mix| method mix]].  
 
The quality of results depends on the methodology of data collection for reflection. In particular, a range of different ways to assess and interpret the needed data should be included, according to the principle of [[Method Mix| method mix]].  
  
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more" data-collapsetext="- Collapse">
+
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: The method mix" data-collapsetext="- Collapse">
  
 
<div class=left-box>
 
<div class=left-box>
===Method Mix:===
+
===Method Mix===
'''Style'''
+
Use a variety of methods according to:
 +
 
 +
====Style====
 
* Choose appropriate methods for your target group
 
* Choose appropriate methods for your target group
  
'''Confidentiality''':
+
====Confidentiality====
 
* Anonymous, half-public, or public
 
* Anonymous, half-public, or public
  
'''Group Relation'''
+
====Group Relation====
* In a plenum, in other collaborative ways, or in individual work
+
* In a plenum, in other collaborative ways, or self-assessment
  
'''Addresses different senses'''
+
====Addressing different senses====
* Individually speaking, dialogue, writing, or moving
+
* Speaking, dialogue, writing, or moving
 
</div>
 
</div>
 +
  
 
===Quantitative or qualitative?===
 
===Quantitative or qualitative?===
How deep should or must your evaluation go? When you want to know how your participants feel, you ask them to show you “thumb up/down”. Afterwards you know that ten persons feel well and three of them not so well. Or you ask detailed qualitative questions, with which you find out why they feel such a way or what they need to feel better. You use a ''quantitative'' and a ''qualitative'' method – both of them are effective based on the situation, and often complement each other.  
+
How deep should or must your evaluation go? When you want to know how your participants feel, you ask them to show you “thumb up/down”. Afterwards you know that ten persons feel well and three of them not so well.
 +
<div class=left-box>
 +
 
 +
====Quantitative Assessment====
 +
Uses '''numerical data''', intends answers on pre- described questions and sets focus on measurability (i.e. a group votes or facilitators prepare a test questionaire).
 +
</div>
 +
Or you ask detailed qualitative questions, with which you find out why they feel such a way or what they need to feel better. You use a ''quantitative'' and a ''qualitative'' method – both of them are effective based on the situation, and often complement each other.
 +
<div class=left-box>
 +
 
 +
====Qualitative Assessment====
 +
Focus on '''specific narrative information''', answers are not pre-structured by the question (i.e. facilitators conduct an interview with open questions, or an evaluator observes a learning process).
 +
</div>
 +
 
 +
===Cooperative or individual assessment?===
 +
<div class=left-box>
 +
 
 +
====Cooperative Assessment====
 +
[[File:Aquarium-evaluation.png | 600px]]<br>
 +
An example for cooperative evaluation in a training: Each participant marks his or her fish. Near the surface="+", at the bottom="-"
 +
 
 +
[[File:Bowls-evaluation.png | 550px]]<br>Participants place balls in different bowls, symbolising different aspects. They can also put liquid in one of the bowls or jars.
 +
</div>
 +
 
 +
<div class=left-box>
 +
 
 +
====Individual Assessment====
 +
[[File:Letter.jpg | 300px |right]]<br>
 +
Would all come to the front when participants would need to make their issues public? What can better be assessed individually or more discretely, could be addressed by individual methods. One example is the method [[A letter from the past]].
 +
</div>
  
 
===Documentation of the Results===
 
===Documentation of the Results===
Line 112: Line 269:
  
 
&nbsp;<br>
 
&nbsp;<br>
 +
 +
<hr class=boldline>
 +
 +
==Learner-Led Assessment Approaches==
 +
In order to enable people to be proactive, autonomous and creative, they need to be trained to articulate and claim competence, referencing multimodal evidence, for wide ranging purposes to warrant their claims. In this sense, evaluation is also a training of "learner assessment literacy" and a process which ideally is involving the learners and addressing their personal abilities.
 +
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: Learner-led assessment approaches" data-collapsetext="- Collapse">
 +
Of course, a democratic and participatory nonformal pedagogy must also proceed differently in judgment and evaluation than in school or university.
 +
 +
<div class=left-box>
 +
===Participatory Evaluation===
 +
The learners themselves should know in advance (transparency) and have a decisive say (co-determination) in regards to
 +
 +
* '''what aspects''' they want to evaluate
 +
* in '''what form''',
 +
* '''which meaning''' they assign to results,
 +
* '''with whom''' they want to share which results.
 +
</div>
 +
 +
<div class=left-box>
 +
 +
===Who assesses and evaluates?===
 +
 +
====Learners: Self and Peer====
 +
Learners draw from wide ranging evidence, data and reflection to evaluate. They might regularily assess performance, progress, organisational aspects, or the learning process and its dynamics. During longer learning events it's not uncommon that learners independently meet in small evaluation/reflection groups in absence of the facilitators.
 +
 +
====Facilitators====
 +
Facilitator assessment can be of learning to report outcomes and impact. It can support learning to identify further learning gains, effective learning approaches and further learning needs and goals.
 +
 +
Learners might also be invited to assess facilitators' approaches and performance.
 +
 +
====External actors====
 +
Civil society organisations, employers, field experts can involve a vakuable perspective. Inviting active contribution from others highlights the relevance and purpose of assessment to learners and builds clear links to the world of work and social engagement. For example, if learners create value for a specific group, it is this group who will be best placed to evaluate the success of the value created.
 +
</div>
 +
</div>
 +
 +
 +
<hr class=boldline>
 +
 +
==Quality of Evaluation and Assessment==
 +
More holistic learning designs need adequate evaluation, reflection and assessment approaches. The following table is showing typical progression paths. Of course, there is no one-size-fits-all approach, because reflection and evaluation must be proportionate to the other program components. Moreover, not every training, every intermediate step or every program can and must be evaluated in the same way. This makes it all the more important for facilitators not to stand still in terms of methodology and to continue developing their own evaluation skills.
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: Quality" data-collapsetext="- Collapse">
 +
[[File: Quality-evaluation-assessment.png | center|700px]]
 +
 +
<div class=left-box>
 +
* The more comprehensive the underlying competence concept is, the more precisely it can be evaluated or reflection can be stimulated.
 +
* The more steps are integrated into the learning process that generate evidence and enable reflection, the more meaningful and relevant the results of the evaluation will be.
 +
* The more diverse voices and perspectives are included, the more fair and relevant judgments and conclusions will be.
 +
* A prerequisite for quality development is that the work of teachers also becomes part of the evaluation culture and that they see themselves as learners.
 +
* A wealth of different observations and data supports the meaningfulness.
 +
</div>
 +
 +
</div>
 +
 +
 +
<hr class=boldline>
 +
 +
 +
<div class=left-box>
 +
===Assessment and recognition in formal and non-formal learning in entrepreneurship education===
 +
[[File:Assessment-E360.png | 180px | right | link=https://dare-network.eu/guide-assessment-and-recognition-in-formal-and-non-formal-learning-in-entrepreneurship-education/]]
 +
An introduction into competency-based assessment and  evaluation not only for entrepreneurship education by the project [https://entrecomp360.eu EntreComp 360], by Hazel Israel ([https://bantani.com Bantani Education]) with Svanborg Rannveig Jónsdóttir and Ramón Martínez.
 +
 +
*[https://dare-network.eu/guide-assessment-and-recognition-in-formal-and-non-formal-learning-in-entrepreneurship-education/ Download]
 +
 +
 +
 +
[[File: E360.png | 120px | link=https://entrecomp360.eu]]
 +
</div>
 +
 +
<hr class=boldline>
 +
 +
==Recognition==
 +
One crucial aspect of assessment in education and training is the recognition of learners’ gained competences. Or in other words – assessment and evaluation should lead to learners, and others, valuing their knowledge, skills, attitudes, values and learning progress.
 +
<div class=left-box>
 +
====Read more:====
 +
* [[Identification and Recognition of Learning Outcome |Identification, Recognition, Validation]]
 +
</div>
 +
 +
===Refer to Standards and Make Social Recognition Easier===
 +
Depending on where the learner needs recognition, it is worth to take care about words and aspects that can  be helpful. A lot of social contexts use specific language and have specific expectations toward the learning outcome. Conscious language and choice might give a description or certificate more value.
 +
 +
<div class="mw-collapsible mw-collapsed" data-expandtext="+ Read more: Referring to standards" data-collapsetext="- Collapse">
 +
<div class=left-box>
 +
 +
===European Skills/Competences, Qualifications and Occupations'' (ESCO)===
 +
ESCO offers a standardized terminology that includes skills, competences, qualifications and occupations. Its outcome for citizens and employees is to find the right terms for their formal and non-formal qualifications or for the job profiles they wish to develop. As well non-formal training provider may use the classifications for validation.<ref name=esco>European Skills, Competences, Qualifications and Occupations (ESCO): https://ec.europa.eu/esco/home</ref>
 +
* European Skills, Competences, Qualifications and Occupations | [https://ec.europa.eu/esco/home ESCO]
 +
</div>
 +
<div class=left-box>
 +
===Key Competences for Lifelong Learning===
 +
They are a good and recognized reference point, especially in the European context. Specific competence frameworks were until 2022 elaborated for digital competence (DigComp), entrepreneurship (EntreComp), learning-to-learn (LifeComp) and sustainability (GreenComp).
 +
 +
A short description of these can be found here:
 +
* [[Transversal or Key Competences]]
 +
</div>
 +
 +
<div class=left-box>
 +
===Citizenship, Human Rights===
 +
The EU was not yet elaborating a framework for citizenship competences. The entrepreneurship competence framework EntreComp includes some aspects - like proactivity, initiative and collaboration.
 +
 +
The ultimate European reference point is currently still the Council of Europe's Reference Framework of Competences for Democratic Culture (RFCDC). Originally created for democracy-related learning in schools, it is more and more applied in non-formal and lifelong learning.
 +
* Council of Europe: [https://www.coe.int/en/web/reference-framework-of-competences-for-democratic-culture/home Reference Framework of Competences for Democratic Culture]
 +
</div>
 +
 +
<div class=left-box>
 +
 +
===European Qualification Framework (EQR)===
 +
The EQF and the subordinated EU member state's national qualification frameworks are also worth paying attention to, in particular if the description is needed also for formal recognition. The EQF seeks to make the different national education systems more compatible and describes vocational profiles and educational outcome for a broad range of educational fields. Therefore the competence model is elaborated enough to offer a basis for certification and documentation of the gain of knowledge and skills. Methodologically member states agreed on the EQF in 2008, which is concretized through every member state with a national qualification framework (in example in Germany the [https://www.dqr.de/ Deutscher Qualifizierungsrahmen]). <ref name=EQF2>Europass: [https://europa.eu/europass/en/european-qualifications-framework-eqf European QualificationFramework]</ref> The framework provides 8 levels of proficiency from level one – “basic knowledge” – to level 8 –“knowledge at the most advanced frontier of a field of work or study and at the interface between fields”. The framework provides "benchmarks for qualification levels across Europe and encourage the embedding of validation systems with formal qualifications system"<ref name=cedefop>European Centre for the Development of Vocational Training (Cedefop) 2009: [http://www.cedefop.europa.eu/EN/Files/4054_en.pdf European guidelines for validating non‑formal and informal learning]; p. 30</ref>
 +
* European Qualification Framework | [https://europa.eu/europass/en/european-qualifications-framework-eqf EQR]
 +
</div>
 +
<div class=left-box>
 +
 +
===Other Competence-related Tools and Methods===
 +
Also other competence frameworks used in your organization or learning field can give orientation (like from OSCE, UNSECO, national curricula, or own frameworks).<ref name=CEDEFOP1>Following the discussion of: European Centre for the Development of Vocational Training (Cedefop): [http://www.cedefop.europa.eu/EN/Files/4054_en.pdf European guidelines for validating non‑formal and informal learning]; p. 44</ref>
 +
 +
* '''YouthPASS''' |  [https://www.youthpass.eu/en/youthpass/ YouthPASS]
 +
*''' Portfolio for youth leaders''' | [https://www.coe.int/en/web/youth-portfolio/home Portfolio]
 +
* '''European Skills Passport''' | [https://europass.cedefop.europa.eu/en/documents/curriculum-vitae EuroPASS]
 +
</div>
 +
</div>
 +
 +
 +
 
<hr class=boldline>
 
<hr class=boldline>
  
 
==Selected Methods==
 
==Selected Methods==
  
<div class=teaser-box>
+
<div class=teaser-box style="background: linear-gradient(to bottom, #affaf2, white);">
 
===Individual Reflection and Assessment===
 
===Individual Reflection and Assessment===
 
An overview over different methods for evaluation in between and at the end of a learning process.
 
An overview over different methods for evaluation in between and at the end of a learning process.
Line 123: Line 404:
 
<i class="fas fa-cog"></i> [[Learning Curve]]<br>
 
<i class="fas fa-cog"></i> [[Learning Curve]]<br>
 
<i class="fas fa-cog"></i> [[A letter from the past]]<br>
 
<i class="fas fa-cog"></i> [[A letter from the past]]<br>
 +
<i class="fas fa-cog"></i> [[Bridge to the Future]]<br>
 
<i class="fas fa-cog"></i> [[From now to tomorrow]]<br>
 
<i class="fas fa-cog"></i> [[From now to tomorrow]]<br>
 +
<i class="fas fa-cog"></i> [[Self-reflection flower]]<br>
 
<i class="fas fa-cog"></i> [[Personal Portfolio]]<br>
 
<i class="fas fa-cog"></i> [[Personal Portfolio]]<br>
 
<i class="fas fa-cog"></i> [[Bullet Journal]]<br>
 
<i class="fas fa-cog"></i> [[Bullet Journal]]<br>
Line 133: Line 416:
  
  
<div class=teaser-box>
+
<div class=teaser-box style="background: linear-gradient(to bottom, #affaf2, white);">
 +
 
 
===Selected Methods for Group Evaluation===
 
===Selected Methods for Group Evaluation===
 
These methods are facilitating a collective process of reasoning. Therefore, they are less confidential, allowing the group to exchange or discuss their observations or findings.
 
These methods are facilitating a collective process of reasoning. Therefore, they are less confidential, allowing the group to exchange or discuss their observations or findings.
Line 144: Line 428:
 
<i class="fas fa-cog"></i> [[Conflict tree]]<br>
 
<i class="fas fa-cog"></i> [[Conflict tree]]<br>
 
<i class="fas fa-cog"></i> [[Color Diary]]<br>
 
<i class="fas fa-cog"></i> [[Color Diary]]<br>
 +
<i class="fas fa-cog"></i> [[Decision and assessment with cards and dots]]<br>
 +
<i class="fas fa-cog"></i> [[Sociometry]]<br>
 
<i class="fas fa-cog"></i> [[Spoken Word Poetry]]<br>
 
<i class="fas fa-cog"></i> [[Spoken Word Poetry]]<br>
 
<i class="fas fa-cog"></i> [[Constructive Feedback]]<br>
 
<i class="fas fa-cog"></i> [[Constructive Feedback]]<br>
 
<i class="fas fa-cog"></i> [[Touch Somebody Who... ]]
 
<i class="fas fa-cog"></i> [[Touch Somebody Who... ]]
 +
 +
<i class="far fa-check-square"></i> [[Guidelines and Questions for Debriefing]]<br>
 
</div>
 
</div>
 
</div>
 
</div>
  
  
<div class=teaser-box>
+
<div class=teaser-box style="background: linear-gradient(to bottom, #affaf2, white);">
 +
 
 
===Self-Assessment of Facilitators===
 
===Self-Assessment of Facilitators===
 
Checklists for a team of educators or for individual self-assessment.
 
Checklists for a team of educators or for individual self-assessment.
Line 169: Line 458:
 
<hr class=boldline>
 
<hr class=boldline>
 
<div class=left-box>
 
<div class=left-box>
 +
 
==Inspiring Handbooks and Sources from the Community==
 
==Inspiring Handbooks and Sources from the Community==
 
<i class="fas fa-book" style="font-size: 54px; color: #ccc; float:right;"></i>
 
<i class="fas fa-book" style="font-size: 54px; color: #ccc; float:right;"></i>
Line 184: Line 474:
 
<hr class=boldline>
 
<hr class=boldline>
 
<div class=left-box>
 
<div class=left-box>
==Apps and Tools: Recognition, Assessment, Validation==
+
==Apps and Tools: Recognition, Competence Assessment==
 
<i class="fas fa-graduation-cap" style="font-size: 54px; color: #ccc; float:right;"></i>
 
<i class="fas fa-graduation-cap" style="font-size: 54px; color: #ccc; float:right;"></i>
  
 
<noinclude>{{:Apps-Recognition-Competence-Assessment}}</noinclude>
 
<noinclude>{{:Apps-Recognition-Competence-Assessment}}</noinclude>
 
</div>
 
</div>
 +
 +
<hr class=boldline>
 +
 +
==References==
 +
<references></references>
  
 
</div>
 
</div>
<div class="right-column-contentinner">
+
<div class="methodpage-infos">
 +
<noinclude>{{:Navi After}}</noinclude>
  
<noinclude>{{:Navi After}}</noinclude>
+
<div class=left-box>
 +
[[File:Assessment-E360.png | link=https://dare-network.eu/guide-assessment-and-recognition-in-formal-and-non-formal-learning-in-entrepreneurship-education/]]
 +
===Assessment & recognition in formal & non-formal learning in entrepreneur&shy;ship education===
 +
An introduction not only for entrepreneurship education by the project EntreComp 360<br>[https://dare-network.eu/guide-assessment-and-recognition-in-formal-and-non-formal-learning-in-entrepreneurship-education/ Download]
  
<div class=right-box> Evaluation is the structured inter&shy;pretation and giving of meaning to predicted or actual impacts of proposals or results.
 
 
</div>
 
</div>
 
  
 
</div>
 
</div>

Latest revision as of 07:24, 14 June 2023

Evaluation.png
Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. Evaluation can be formative - that is, taking place during the development of a concept or during a workshop - or summative, drawing lessons from a completed action or process.

Evaluation

Systematic collection, review, and use, of information and evidence to represent, evaluate and report learning, in different ways, for different purposes.


Focuses of Reflection

Already during Planning facilitators were considering goals, contents, methods, topic, needs and the environment.

During a process or training it is necessary to check if the process is still inline with the plans or if priorities have changed or needs are different, or a process should take a general different direction. There are several purposes for evaluation:

Why are we assessing?

Assessment for measurement requiring a stable metric to identify starting points and distance travelled.

Assessment for selection requiring rank order or criterion-referenced

Assessment to diagnose next learning steps

Assessment to evidence impact or competence

Assessment to evaluate the learning approaches taken, educator or organisational performance.

Why debriefing from a learner's perspective?

Debriefing has particularly valuable benefits in activities in the context of education for democratic citizenship and human rights education (EDC/HRE):

  • Debriefing is participatory learning. It addresses learners as self-responsible subjects capable of reflecting, inquiring, comparing, connecting, and planning their own learning experiences.
  • Debriefing is necessary for sustainable learning. When scrutinising their learning experience and dealing with different perspectives in a group discussion, learners gain additional information, deepen understanding, develop further competences, and gain self-confidence.
  • By becoming aware of their developmental needs and by recognising their own learning style, they gain knowledge and a critical understanding of themselves.
  • Debriefing turns learners into owners and co-authors of their learning progress. Games require learners to get involved actively, deeply, and at times personally. Debriefing guides and enables them to turn these, and probably also future, experiences into a fruitful learning outcome. Debriefing thus develops autonomous learning skills.

Democracy Dimension

Debriefing can foster learning in all three dimensions of democracy education:

  • Learning about democracy: Which were the topics of the game-based learning activity?
  • Debriefing fosters learning through democracy: Learners see (practical) meanings and consequences of what happened, of how they acted and reacted. It offers participants a structured and safe environment to practice democratic and free exchange.
  • Debriefing fosters learning for democracy: Learning about the relationship with their real-world experiences, insight in social developments and inspiration for social action.

Assessment - in Between or at the End?

While debriefing is usually integrated in each method implementation or program unit, summative and formative evaluation activities are defined in an evaluation and assessment strategy of a whole process. They aim to collect data beyond a single unit and to allow reflection on more fundamental aspects (overall progress, competence gain, lessons learned, group process...).

Debriefing

Conducted at the end of a single learning unit with focus on reflection and assessment of what happened and was learned during this unit.

Formative Assessment

Serves learning progression, supporting development over time, integral and repeating part of a learning process.

...

At the end of a phase or learning process, other reasons for evaluation may also come into play. You may want to demonstrate a person's learning progress or performance, or you may want to check whether the different goals have been achieved.

Summative Assessment

Conducted at the end of a learning process.

... ... ... ... ...



What are We Assessing?

In process-oriented and non-formal learning processes we very often reflect how successful we facilitated a learning unit or to what extend the element of the process contributed to the intended goals. The aspects in assessment are similar to the criteria for planning. For instance, one could follow this logics:

Goal-content-method-space.png


Goal Achievement

  • Group: Regarding the aimed interaction and cooperative learning in the group
  • Topic: The coverage of the foreseen topical aspects
  • Individual: Learning goals and competence development (see below)
  • Process: The proceeding in the steps of the collective learning process

Methodology

  • The choice of methods and their mix
  • The ability of facilitators to apply methodology
  • Process moderation

Content

  • Coverage of the necessary thematical aspects at a satisfactory depth
  • Inclusion of knowledge from the field participation-citizenship-democracy and how it was connected to the goal of the training

Space/Context

  • Facility: Opportunity to learn, to cooperate, to meet, to feel well-accommodated
  • Group: Opportunity and ability to involve, interact, relate to the other learners
  • Individual needs: Possibility to satisfy individual needs inside and outside the scheduled activities: social, cultural/spiritual, physical, intellectual, emotional
  • With special attention to special needs
  • Inclusion of the local environment: in seminar work, in topical aspects

Competence development and transfer

  • Participant's acquisition of competences
  • Participant's development towards self-empowerment and democratic participation
  • Your acquisition of competences as a facilitator
  • Transferability of learning aspects to the real life

Management

  • Event management, logistics, problem-solving
  • Cooperation within the facilitation team
  • Were expectations met, needs respected?

Reflection/Debriefing Scheme for a Single Method

While it is imperative that summative evaluation end up with enough data to cover all of these analytic areas, we also need to consider an approach to evaluation for each individual method we use that follows a basic standard. This or similar suggestions have proven workable for many educators in non-formal education.



Block 1: Facts and Feelings

What happened?
What did you feel about it?
(in regards to: yourself, group, topic, process..)


Block 2: Learning

What did you learn?
(in regards to: yourself; group & relationships developed during the activity; topics & concepts employed; the process..)


Block 3: Transfer

What can you conclude for you daily life?
What can you do (differently) in regards to the problem?
(connection to the real life of the learners)



Source: Demogames



Evidence

To make a reasoned judgment, one needs a data base in addition to predetermined focus and purpose. Therefore, assessment is always dependent on good evaluation material.

In more traditional academic assessment, educational purposes have tended to focus on testing knowledge and understanding in one hit, examination-style papers, which are dominated by written evidence.

However, increasing awareness of the limitations, potential bias and ‘wash back’ of formal examinations has strengthened calls for more purposeful and meaningful assessment methods, drawing from richer and more diverse evidence, over time.

Within your process:

  • Debate,
  • Declarative methods,
  • Interviews,
  • Observation,
  • Portfolio method,
  • Learning diary,
  • Autonomous reflection groups of learners,
  • Presentation,
  • Simulation
  • Evidence extracted from (project)work,
  • Tests

After: CEDEFOP[1]

At the end of a learning process may stand....

  • Competence description along standards for programs or learning events
  • Certificates for participation
  • Diplomas
  • Qualified competence description
  • A completed portfolio
  • Learning diary[2]



Method Mix

The quality of results depends on the methodology of data collection for reflection. In particular, a range of different ways to assess and interpret the needed data should be included, according to the principle of method mix.

Method Mix

Use a variety of methods according to:

Style

  • Choose appropriate methods for your target group

Confidentiality

  • Anonymous, half-public, or public

Group Relation

  • In a plenum, in other collaborative ways, or self-assessment

Addressing different senses

  • Speaking, dialogue, writing, or moving


Quantitative or qualitative?

How deep should or must your evaluation go? When you want to know how your participants feel, you ask them to show you “thumb up/down”. Afterwards you know that ten persons feel well and three of them not so well.

Quantitative Assessment

Uses numerical data, intends answers on pre- described questions and sets focus on measurability (i.e. a group votes or facilitators prepare a test questionaire).

Or you ask detailed qualitative questions, with which you find out why they feel such a way or what they need to feel better. You use a quantitative and a qualitative method – both of them are effective based on the situation, and often complement each other.

Qualitative Assessment

Focus on specific narrative information, answers are not pre-structured by the question (i.e. facilitators conduct an interview with open questions, or an evaluator observes a learning process).

Cooperative or individual assessment?

Cooperative Assessment

Aquarium-evaluation.png
An example for cooperative evaluation in a training: Each participant marks his or her fish. Near the surface="+", at the bottom="-"

Bowls-evaluation.png
Participants place balls in different bowls, symbolising different aspects. They can also put liquid in one of the bowls or jars.

Individual Assessment

Letter.jpg

Would all come to the front when participants would need to make their issues public? What can better be assessed individually or more discretely, could be addressed by individual methods. One example is the method A letter from the past.

Documentation of the Results

When choosing your method, you should also consider the form in which you need results. Language, pictures, photos… Many things are possible and they can complete a particular situation or also contrast it. It is important with most methods to formulate a question as concretely as possible.

As self-evaluation is crucial for independent learning, we also include methods her that help individuals to document learning outcomes, inspirations, and insights in an individual way.

 


Learner-Led Assessment Approaches

In order to enable people to be proactive, autonomous and creative, they need to be trained to articulate and claim competence, referencing multimodal evidence, for wide ranging purposes to warrant their claims. In this sense, evaluation is also a training of "learner assessment literacy" and a process which ideally is involving the learners and addressing their personal abilities.

Of course, a democratic and participatory nonformal pedagogy must also proceed differently in judgment and evaluation than in school or university.

Participatory Evaluation

The learners themselves should know in advance (transparency) and have a decisive say (co-determination) in regards to

  • what aspects they want to evaluate
  • in what form,
  • which meaning they assign to results,
  • with whom they want to share which results.

Who assesses and evaluates?

Learners: Self and Peer

Learners draw from wide ranging evidence, data and reflection to evaluate. They might regularily assess performance, progress, organisational aspects, or the learning process and its dynamics. During longer learning events it's not uncommon that learners independently meet in small evaluation/reflection groups in absence of the facilitators.

Facilitators

Facilitator assessment can be of learning to report outcomes and impact. It can support learning to identify further learning gains, effective learning approaches and further learning needs and goals.

Learners might also be invited to assess facilitators' approaches and performance.

External actors

Civil society organisations, employers, field experts can involve a vakuable perspective. Inviting active contribution from others highlights the relevance and purpose of assessment to learners and builds clear links to the world of work and social engagement. For example, if learners create value for a specific group, it is this group who will be best placed to evaluate the success of the value created.



Quality of Evaluation and Assessment

More holistic learning designs need adequate evaluation, reflection and assessment approaches. The following table is showing typical progression paths. Of course, there is no one-size-fits-all approach, because reflection and evaluation must be proportionate to the other program components. Moreover, not every training, every intermediate step or every program can and must be evaluated in the same way. This makes it all the more important for facilitators not to stand still in terms of methodology and to continue developing their own evaluation skills.

Quality-evaluation-assessment.png
  • The more comprehensive the underlying competence concept is, the more precisely it can be evaluated or reflection can be stimulated.
  • The more steps are integrated into the learning process that generate evidence and enable reflection, the more meaningful and relevant the results of the evaluation will be.
  • The more diverse voices and perspectives are included, the more fair and relevant judgments and conclusions will be.
  • A prerequisite for quality development is that the work of teachers also becomes part of the evaluation culture and that they see themselves as learners.
  • A wealth of different observations and data supports the meaningfulness.




Assessment and recognition in formal and non-formal learning in entrepreneurship education

Assessment-E360.png

An introduction into competency-based assessment and evaluation not only for entrepreneurship education by the project EntreComp 360, by Hazel Israel (Bantani Education) with Svanborg Rannveig Jónsdóttir and Ramón Martínez.


E360.png


Recognition

One crucial aspect of assessment in education and training is the recognition of learners’ gained competences. Or in other words – assessment and evaluation should lead to learners, and others, valuing their knowledge, skills, attitudes, values and learning progress.

Refer to Standards and Make Social Recognition Easier

Depending on where the learner needs recognition, it is worth to take care about words and aspects that can be helpful. A lot of social contexts use specific language and have specific expectations toward the learning outcome. Conscious language and choice might give a description or certificate more value.

European Skills/Competences, Qualifications and Occupations (ESCO)

ESCO offers a standardized terminology that includes skills, competences, qualifications and occupations. Its outcome for citizens and employees is to find the right terms for their formal and non-formal qualifications or for the job profiles they wish to develop. As well non-formal training provider may use the classifications for validation.[3]

  • European Skills, Competences, Qualifications and Occupations | ESCO

Key Competences for Lifelong Learning

They are a good and recognized reference point, especially in the European context. Specific competence frameworks were until 2022 elaborated for digital competence (DigComp), entrepreneurship (EntreComp), learning-to-learn (LifeComp) and sustainability (GreenComp).

A short description of these can be found here:

Citizenship, Human Rights

The EU was not yet elaborating a framework for citizenship competences. The entrepreneurship competence framework EntreComp includes some aspects - like proactivity, initiative and collaboration.

The ultimate European reference point is currently still the Council of Europe's Reference Framework of Competences for Democratic Culture (RFCDC). Originally created for democracy-related learning in schools, it is more and more applied in non-formal and lifelong learning.

European Qualification Framework (EQR)

The EQF and the subordinated EU member state's national qualification frameworks are also worth paying attention to, in particular if the description is needed also for formal recognition. The EQF seeks to make the different national education systems more compatible and describes vocational profiles and educational outcome for a broad range of educational fields. Therefore the competence model is elaborated enough to offer a basis for certification and documentation of the gain of knowledge and skills. Methodologically member states agreed on the EQF in 2008, which is concretized through every member state with a national qualification framework (in example in Germany the Deutscher Qualifizierungsrahmen). [4] The framework provides 8 levels of proficiency from level one – “basic knowledge” – to level 8 –“knowledge at the most advanced frontier of a field of work or study and at the interface between fields”. The framework provides "benchmarks for qualification levels across Europe and encourage the embedding of validation systems with formal qualifications system"[5]

  • European Qualification Framework | EQR

Other Competence-related Tools and Methods

Also other competence frameworks used in your organization or learning field can give orientation (like from OSCE, UNSECO, national curricula, or own frameworks).[6]



Selected Methods

Individual Reflection and Assessment

An overview over different methods for evaluation in between and at the end of a learning process.


Selected Methods for Group Evaluation

These methods are facilitating a collective process of reasoning. Therefore, they are less confidential, allowing the group to exchange or discuss their observations or findings.


Self-Assessment of Facilitators

Checklists for a team of educators or for individual self-assessment.

 


Inspiring Handbooks and Sources from the Community


Apps and Tools: Evaluation


Apps and Tools: Recognition, Competence Assessment

Several online tools support learners and educators in (self-)assessment. However, one should check their terms and conditions for storing and using personal data before using them in trainings.


References

  1. European Centre for the Development of Vocational Training (Cedefop): European guidelines for validating non‑formal and informal learning;; p. 59 ff.
  2. see as well: The European Centre for the Development of Vocational Training (Cedefop): European guidelines for validating non‑formal and informal learning; p. 19
  3. European Skills, Competences, Qualifications and Occupations (ESCO): https://ec.europa.eu/esco/home
  4. Europass: European QualificationFramework
  5. European Centre for the Development of Vocational Training (Cedefop) 2009: European guidelines for validating non‑formal and informal learning; p. 30
  6. Following the discussion of: European Centre for the Development of Vocational Training (Cedefop): European guidelines for validating non‑formal and informal learning; p. 44

After


Download: Banner

Teaser-competendo-big.png

Assessment-E360.png

Assessment & recognition in formal & non-formal learning in entrepreneur­ship education

An introduction not only for entrepreneurship education by the project EntreComp 360
Download