SLAAMB

Executive Summary of the Final Report
of the CAAR Evaluation Team
- end of the Fourth Quarter of Year Three

Researched and Prepared by:

Neil Nelson
Wolf Morrisseau and
Robyn Louis

With Earlier Participation by

Bartolo Pilato and
Margaret Wanlin

MARCH 22, 2010

Executive Summary of the Final Report of the CAAR Evaluation Team
- end of the Fourth Quarter of Year Three

Setting the Stage
Considerable funding was provided by HRSDC to SLAAMB, the Sioux Lookout Area Aboriginal Management Board. SLAAMB's project was named CAAR - the Centre for Aboriginal Apprenticeship Research - and was to investigate ways to increase the participation rate of First Nations persons in the SLAAMB catchment area in the construction trades with special focus on carpentry, plumbing and electrical. The core catchment area includes thirty communities, most of them remote, fly-in communities. The project began in April of 2007 and will wind up the research phase at the end of March 2010.

Overview
A summary assessment of the success of the CAAR project requires considering the 33 months of its life at three different levels.
1. Research. The CAAR project succeeded regarding this highest level goal: find out how to increase the participation rate of Aboriginal persons in three construction trades.
2. Trades Training. One the second level, the implied goal was to test specific ways to increase participation that are simple, reliable and affordable, Here the CAAR project gets mixed reviews. This will be explained momentarily.
3. Building Capacity. On a third level, building community and regional capacity among First Nations organizations, communities and partnerships, CAAR exceeded expectations.

Assumptions
Some of the problems in seeking to increase participation in the trades resulted from what seemed to be reasonable assumptions which proved to be untrue.
1. Academic readiness. CAAR learners were less skilled in Math, English in general and in trades English than was assumed.
2. Quality of trades experience. It was believed that CAAR learners admitted to the Challenge program - taking a crash course and then challenging the Ontario Certificate of Qualification exam - had stronger experience than reality demonstrated. Their experience, however skilled they were, was very narrow. Northern homes don't vary much: low INAC funding and a limited list of approved contractors mean that First Nations tradespeople (1) are typically limited to residential experience, (2) work as labourers under the direction of non-Native contractors and (3) build essentially the same house plan time after time, year after year.
3. Personal, social and cultural barriers. Some of these caused some attrition; the drop-out rates in the Pre-Apprentice and GED programs was higher than expected.

One should not be critical too quickly of the errors in core assumptions. To start something brand new, some assumptions simply must be made. Assumptions are merely "best guesses". CAAR partners were not careless - nobody else knew the real state of skills and experience either. The Evaluation team sought persons with experience in similar programs nationally and found none that were very similar to CAAR's core CofQ Challenge program. In fact there were few that were similar in any informative way. Even with SLAAMB's Northern roots and the considerable experience of Confederation College in delivery of trades training, time demonstrated that many core assumptions were incorrect. Fair enough: CAAR is a research project. No-one had done this before. The facts were not available.

CAAR partners now have better information. All have responded as quickly and as fully as possible to information via evaluators from learners, exam marks, instructors, CAAR staff and members of the communities.

Five Parts of the CAAR Project
CAAR is usually seen a having three programs or elements.
1. GED. Traditional GED curriculum offered via internet.
2. Pre-Apprenticeship, several weeks via the net, then trades training on-site in Sioux Lookout followed by a placement with journeyperson-level tradespeople.
3. The CofQ; the Certificate of Qualification challenge program. A crash course in Sioux Lookout to prepare experienced but uncertified northern tradespeople to challenge the trades exams. The provincial ministry waived the education requirements to help in recruitment.

In fact there are five elements to CAAR. The other two are:
4. CAAR's work as an employment agency and
5. CAAR as a significant builder of community and regional capacity.

The GED - Grade Equivalent Diploma
1. A recurring reminder: CAAR is a research project. The research question here was, "how can we develop and deliver a GED program via e-delivery to bring education and skills up to the standard needed for further trades training
2. The CAAR partners version of GED was, comparatively, reasonably successful.
3. CAAR (or another agency) could, if the advice here and in previous reports is heeded, provide an effective Northern delivery model for others to adopt or adapt.
4. Nevertheless, we advise that for CAAR's own purposes, using a program which focuses only on (1) math and (2) English might be of more use to CAAR, more attractive to learners and more effective re retention and passing rate.
5. So, CAAR has learned how to do a GED program pretty well. However, in the trades training context, should they be doing GED or some other more targeted upskilling activity?

Pre-Apprenticeship
1. Again: CAAR is a research project. The research question here was, "Can CAAR develop and deliver a pre-apprenticeship program (1) with a significant e-learning component (2) complemented by high quality on-site trades training (3) leading to qualification for apprenticeship?"
2. While there is still room to implement improvements we perceive that the major concerns and recommendations have been identified. The CAAR partners know what is left to do and how to prevent some of the problems they encountered. With the implementation of the suggested improvements - more detail will be included in the final report in early April of 2010 - this program is ready to be adopted or adapted by other providers of trades training.
3. The evaluators must, however, report that this younger cohort experienced a much higher drop-out rate and much of that was due to inability to thrive in the "urban" environment of Sioux Lookout. There are some ways to improve that but there are limits to what can be achieved re retention. This is a very vulnerable population.

Certificate of Qualification

1. Key Research Finding. CAAR means "Centre for Aboriginal Apprenticeship Research". With regard to the CofQ program, the research is done and has been successful. We conclude that, having made many, many changes to the initial design, the CAAR partners have produced a program that is as good as it gets, or very nearly so. There are no major new ideas to try. Incremental improvements - yes, certainly. But the core learning is that this course is now tried, tested, experienced and in the end is well designed. We say this in spite of what will seem to be pretty low success rates. It takes some explanation to understand this - please read on.

2. Summary
For the moment we will use only data from the first five CofQ groups plus one person (electrician) who wrote solo a bit later. Here are the summary figures:

Trade       Number Writing Number Passing % Passing
Carpenters         36                    4                11
Plumbers           10                    0                 0
Electricians         7                    3                 43
Total                53                   7                 13.2

• Eleven percent of carpenters passed, four of thirty-six. Carpentry has been the core offering and there will have been seven offerings by the end of the project.
• No plumbers passed; ten attended the course and tried the exam.
• Two of six electricians were successful after taking the class. One wrote separately later and passed. The final standing for electricians is 43%.

3. Interpretation

There are some factors affecting these scores which help in understanding the causes and the implications of the outcomes.

• The carpentry certificate includes qualification in (1) residential, (2) commercial and (3) industrial carpentry. This is an especially hard menu for persons from remote communities with little or no opportunity to get other than residential experience. The carpentry program grew in length from four weeks to twenty weeks over the life of the project. We see the twenty weeks as the right length.
• The electrical certificate is for "Rural and Residential" electricians: no commercial, no industrial. This course was offered twice with course lengths of four weeks and six weeks. Two from each passed, though one had to re-write a couple of weeks later to succeed. In each course there were only three learners so all had considerable attention from the instructor. In each course the "middle" performer was pretty close. A longer course might well have led to four of six passing.
• It is our understanding that plumbing is like electrical; a residential certificate. Again, this course was offered only twice in four and six week formats. That was too little time for the plumbers to succeed. They need a longer course.
4. Barriers and Solutions.

There are a few large and mostly systemic items making these courses difficult. Many have been addressed; here are some comments about those which remain.
• Breadth of the Content. With requirements in three areas, the carpentry certificate is very difficult to obtain. It is especially so for this target group who have limited (or zero) access to commercial and industrial experience. This is beyond the power of any of us, including the funder, to change.
• The Exam Itself. The carpentry exam seems to us to have some systemic flaws. We will take that up with the appropriate ministry after the CAAR program is complete.
• The Instructors and Materials. We have observed the instructors and have been given copies of the very high quality materials they have produced . The experience, skills, quality and investment of time is high. Adding a certified carpenter with university-level math and teaching experience had significantly improved math skills The instructors are a participative part of the solutions; they are not the problem.
• Course Length. Twenty weeks is barely enough but we are not comfortable suggesting adding more time. The twenty weeks is already complemented by two full weeks of safety and certificate training so carpentry is already in fact a twenty-two week course.
• Smaller and resolved Issues. Sections of this summary and of the full report will document other problems encountered and what the CAAR partners did to address them.
We do not think there are any major changes to the course that would make a major difference. We discuss this more fully later.

5. Limits to Information Presented Here.

As stated above, data presented here covers only the first five course offerings. There were great expectations for the sixth intake - a twenty-week carpentry program plus two weeks safety/certificate training. Surprisingly this did not produce any new passes out of the eight learners. While the box below documents our belief that the exam situation that group faced was very negative, we acknowledge that a reasonable expectation would have been two or three passes out of nine who wrote (eight current learners and one re-write). That would have been acceptable relative to other trades exams and other demographic groups

Problem With CofQ Scores and Data

It is necessary to intrude at this point. By this writing the grades for the sixth CofQ intake - the first twenty-week version - have come in. A problem with that data has been discovered.

Two things about the now-available results need mentioned here. First, none of the nine who challenged the exam passed. Second, circumstances around that have led us to a difficult decision. When we spoke with the CAAR Management Team, members believed that the entire group suffered from a panic attack. Post-exam interviews showed all the signs for that. That still seemed weak until we were told about the circumstances.

The learners were booked to write on a Friday. The MTCU invigilator changed that date with little warning, forcing the already anxious - but well prepared - learners to wait over a weekend. That alone put them at risk. Next, on Monday that same invigilator, driving in from Kenora, showed up forty-five minutes late having informed no-one that she would not be on time. In spite of knowing that Sioux Lookout was 235 Kilometres away, that weather delays were possible and construction delays were almost certain, the invigilator did not travel the day before but left home the morning of the exam.

There was not even a telephone to warn CAAR of the delay. After forty-five unexplained minutes of waiting it is unsurprising that a group of already anxious learners panicked.

As an evaluation team we can not ethically make public any statistical analysis of the data. The actions of the invigilator twice introduced a negative variable not experienced by earlier groups. Our ability to compare groups has been compromised. Data from this very untypical situation could skew all results and render both our inferences and our recommendations invalid.

Nevertheless the Evaluation team is required to offer some suggestions. Since the carpentry course has been offered the most (seven times by the project's end), it seems to us that the advice below, while most influenced by carpentry, is useful with regard to all three trades.

6. Option One: Change the qualifications for entry. Pre-test for all the key variables identified by the project as being essential. Test for mechanical math, mechanical aptitude and English, in that order. Require potential learners to be interviewed at length by CAAR instructors. Also include the TOWES test. Check references carefully. Prefer or even demand grade twelve or equivalent level for math. Test this prior to entry; don't trust earlier scores. Even learners wishing to repeat the course would have to meet the same criteria.

In this mode the program could be run just once or twice per year for a highly selected group. Making it exclusive and implementing other available recommendations are likely to produce a higher percentage of successful exam results. It is a Harvard model; it starts with the best first year students so it's easier to produce the best graduates. Running it once per year would cost less, though the costs or recruiting and screening would be higher than at present.

7. Option Two. Keep the current qualifications for entry. Add character references and formal interview by instructors. Screen more, especially for math ability. Implement all recommended improvements. Run the course at least twice per year. Accept the lower pass rate (we predict that 20-25 percent can be achieved) but maintain accessibility to the program. That was part of the initial rationale, to get some barriers (e.g. grade level) out of the way to allow seasoned tradespeople to challenge the CofQ.

If accessibility to the CofQ is more important than (1) pass rate and (2) cost per learner or per graduate, then the CAAR partners should run two programs per year as described in this option.

8. Option Three. Shut it down. If there is no agreement to make changes as suggested immediately above: shut it down. There is not much more to be learned. Not much more to be improved. Planned incremental changes would add to the chances of learners passing, but by small increments, not by a major amount. If (1) the cost of the present model is too high for the return and (2) an elite model isn't seen as appropriate or saleable, then we see no way to continue.

9. Other Trades. Development of a twelve-week (not twenty-week) version of the Plumbing and Electrical trades courses should be undertaken. Since in both cases the exam requirements are more realistic (v. carpentry), there is a good likelihood of a reasonable pass rate. Electricians already have a good pass rate . Plumbers have never been offered a course of a reasonable length.

10. Final Thoughts. The CofQ is hard! It is supposed to be. We think there are some language and structure issues making it harder than it needs to be, and we will, in late March and early April, document all of our concerns and suggestions. The carpentry CofQ covers residential, commercial and industrial competence - neither plumbing nor electricity do so. Numbers for Non-Native populations are very hard to uncover; licensing and certification organizations keep such information close. Still, members of various trades tell us informally that average young Non-Native learners also have a low success rate on the first try. But likely not this low.

Our target population has been significantly disadvantaged. It would feel paternalistic to say more about that here but it is a fact and we would not all be working at this if success was easy.

We believe that the major problems have been found and solutions to those that can be addressed are in place or in process. There is no "magic bullet" that has been missed which would improve process or outcomes by a significant margin. Given the qualifications of the potential learners it will be very difficult, no matter what is improved or replaced, to increase the success rate beyond 15-20 percent for carpentry. But that should be seen a acceptable success. The decisions, moving forward, must now be made solely on resources and values.

Employment
The timing of the CAAR project was influenced by the planned building of a new hospital and hostel in Sioux Lookout to serve the catchment area both in town and from twenty-five Northern (mostly fly-in) First Nations communities. It became evident early on that CAAR and its host agency SLAAMB would need to provide a variety of employment services, more than those anticipated in the planning of the project. They have become a hiring hall for employers (hospital and beyond) to find qualified Aboriginal help - journeypersons, apprentices and labourers. They recruit and screen applicants and have high standards about who gets referred.

1. The reputation of CAAR is part of the reason for the impressive increase in participation/representation of First Nations people in the regional workforce.
2. CAAR built bridges to the construction unions, thus destroying the myth that these unions were at least unwelcoming to Aboriginal persons and often explicitly racist.
3. Almost all of the trades have been glad to have new members, new apprentices and in general to have more capacity of their own to meet the needs of employers.
4. They work regularly with CAAR and both work well with employers.
5. There are few intended Employment Equity programs with the track record CAAR has, and for CAAR it was almost a side-effect.
6. CAAR spent years building good relations with the hospital. In turn the hospital CEO made it clear to all bidders on this multi-million dollar project that respect (and action) for Aboriginal employees was required.
7. Who is employed at the hospital-hostel site?
• there are nine former CAAR learners;
• two of them are "Red Seal" Journeypersons and
• seven who, though they didn't pass the CofQ, are regarded as valuable employees.
• Sixteen others from the communities are employed at the hospital, some in quite specialized and responsible positions.
• Twenty-two First Nations people from out of this area also work on the site
8. Who is employed at other locations around the region?
• .there are forty-eight more Aboriginal persons known to be working,
• forty-six of them from CAAR programs.
• Three of those are CAAR graduates; they passed their CofQ exams.
• The rest, while not journeypersons, are working, most in their home communities.
• Those report that they are doing better and more professional work than they did before their time with CAAR.
9. That adds up to ninety-five currently employed Aboriginal persons, most from First Nations and most of them in turn from the area.

This speaks well, we think, of CAAR's impact on (1) making people more employable and (2) finding good jobs for them. There is more to the goal of improving representation in the workforce than enhanced qualifications. Getting employed at all improves both participation and representation.

Building Connections and Capacity
We suggested earlier that CAAR has had considerable planned influence and impact. There has also been as much unanticipated impact. The influence of the CAAR project has, as in the discussion of Employment above, been larger than planned and larger than could reasonable been anticipated.

Connections: Connections come first. Then impact.
The core CAAR partners are: SLAAMB, Confederation College and MTCU, the Ministry of Training, Colleges and Universities. These have been working together since early in 2007 and continue to build skills, synergy and trust. To this HRSDC must be added; they have been a contributing, engaged partner and not just a distant funding source.

Other sponsors and partners include the Sioux lookout Area Chiefs, the SLAAMB Board, the project's Management Committee, the Tribal Councils, the construction and trade unions, the Meno Ya Win Health Centre, Hydro One, K-Net, the town of Sioux Lookout, the Provincial Literacy branch, Job Connect, TV Ontario and individual providers of similar services.

Capacity and Impact
Every connection above is also an increase in capacity of (1) individual people, (2) organizations and (3) the region and its inhabitants, especially but not only the members of the communities of the Sioux lookout Area Chiefs. From there much has followed.

Development of New Trades Training Programs
Other organizations have been quick to seek out CAAR. Their needs: trades training - quite different trades - for First Nations and Métis people.
• Hydro One, Wasaya Air and Meno Ya Win Health Centre have asked the CAAR partners to develop trades training programs for Aboriginal persons to work in their organizations.
• The Hydro One initiative has been in development for more than a year. First Nations people will qualify for trades certification in three Hydro-specific trades.
• Wasaya Air is a very successful major regional and Aboriginal-owned enterprise. They contracted with CAAR - in this case SLAAMB and the College - to recruit, assess and teach First Nations people to obtain commercial pilot licenses.
• Meno Ya Win Health Centre has contracted with CAAR to develop and implement a year-long program for "language interpreters". In fact graduates will be more than interpreters and even more than "Aboriginal Navigators"; a new career is being invented.
All three are under way. We will be able to say more about them in the final report next spring.

Concerning broad impact, CAAR has:
• Increased the number of certified tradespeople and apprentices,
• Increased the number of CAAR people and other Aboriginals employed,
• Improved the safety knowledge of several dozen participants and thereby improved safety standards in community construction ,
• Improved the specific skills of persons building and maintaining housing stock in the communities, and
• Increased the capacity of Northern employers by providing a larger and more skilled hiring pool of skilled tradespeople, apprentices and highly qualified labourers.
That constitutes capacity building on several levels. It is a significant outcome.

SLAAMB itself: has more learning and unique knowledge about development and delivery of a range of employment programs.
• SLAAMB/CAAR is now a successful and in-demand employment agency, a key route to making their own grads successful.
• The CAAR CofQ program is unique and is, with some final tweaking, ready for use by other jurisdictions. It is imperfect but aside from small improvements, it is nevertheless the best model known.
• The CAAR Pre-Apprentice program is at a similar stage. There are things to change but, on the whole we perceive that most of what can be learned has been learned. "Proof of concept" has, we think, been provided for this and for the CofQ.
• The CAAR version of GED compares favourably with others delivered by e-learning to remote communities.
Each of these has, in the past thirty-one months, been tried and tested and continuously improved. This goes beyond CAAR; this is now sharable capacity.

The new SLAAMB trade centre has added significant capacity for CAAR, the College and the communities of the Sioux lookout Area Chiefs as a whole.
• The building itself is a major new resource
• The shop and its good supply of tools adds to that
• Skills learned by CAAR and SLAAMB staff will be retained and are transferable
• General administrative and management capacity have also grown
• Infrastructure (computers, connectivity, tech skills and more) remain
This enhances the capacity to offer existing programs (CAAR and others) and to develop and offer other new initiatives.

Learning materials. CAAR and the College have developed a range of teaching aids beyond the curriculum materials.
• Video library. CAAR now has 4 hours of video of tools and procedures with two hours more to come. This is a generally useful teaching tool but was especially designed to address deficiencies in trades English.
• College instructors have developed their own tests for Pre-Apprentice courses and have compiled a collection of their own and Alberta Red Seal test bank questions. These have been valuable in preparing learners for the CofQ.
• A set of enhanced Power Point teaching slides. These include the slides themselves, text, videos, animation, a "Jeopardy" style test to pass before moving on and perhaps audio commentary in the future.
• The Phraselater is just about to be released and tested. This device will allow users to get translation from Ojibway/OjiCree to English and vice-versa. The content will be focused or trades names and terminology.

5. Formal Recommendations from the Evaluation Team

We will keep our recommendations at a high level and will provide only a few of them.

1. Continue Some CAAR Functions. While we have appropriately reported on frustrations and unsuccessful attempts, we have also described many successful efforts at improvement. In general the CAAR project has been shown to have had broad impact in a number of areas. We recommend that the core functions of CAAR should be continued.

2. Replace GED. The GED program has had mixed success but much has been learned about how to improve it. If there is reason to offer the GED again, CAAR has the information required to make it more successful. However, for trades training, we suggest that some other option should be adopted to provide access to grade twelve level skills in math.

3. Add Literacy to SLAAMB's Repertoire. There is no targetted literacy program among CAAR's tools and we recommend that such a program be developed for the neediest of CAAR's potential learners. We do not think that the GED or other secondary school oriented programs will meet this niche; a initiative targetting less literate person is justifiable.

4. Offer Pre-Apprenticeship. The Pre-Apprenticeship program is improving on many fronts. This program should be continued. Recommendations earlier in this report provide a road map to improve this program and those should be implemented. Run it once per year with placement beginning in mid-Spring.

5. Offer CofQ Yearly. The CofQ program cannot YET be considered a success based on the number who passed the exam. However, there is other impact of this program documented in the sections on Impact, Employment and Development of New Trades Training Programs. This program should be continued as follows:
• Offer the carpentry program once per year;
• Modify the electrical and plumbing courses; run each once per year and
• Raise the entry standards to all, though not to such an exclusive/elite level that too many potential Red Seal tradespeople are excluded.

6. Continue Employment Functions. These CAAR roles have become integral to the growth in employment for Aboriginal persons as journeypersons, apprentices and skilled labourers. This impact has been felt by those who took one or more CAAR programs, for other Anishnawbek in the region and for Aboriginal persons beyond our own catchment area. This function, while understated initially, has become quite impactful and to lose it would significantly diminish the chances of employment and career growth for our primary audience and for others. Further, this is a function whose elements and needs can be described with sufficient clarity that it can be adopted or adapted in other Canadian jurisdictions and beyond.

7. New Trades Training Opportunities Arriving Regularly. The reputation of CAAR attracted other organizations to request development or co-development of training programs in other fields. Hydro One, Wasaya Air and Meno Ya Win Health Centre have asked the CAAR partners to develop trades training programs for Aboriginal persons to work in their organizations. This function can, we expect, continue with little or no direct assistance from provincial or federal levels. However, we hope some ongoing government participation can happen. Two levels of government helped to build this new entity called CAAR which is succeeding so well in many areas - some well beyond initial expectations. We just think those levels of government should stay on board in some visible fashion so they can continue (1) to give advice, (2) to learn from this on-the-ground initiative and (3) to share earned credit for what has been developed with their support.

8. Cultural Sensitivity. We have talked about this in detail in previous reports. We will not repeat any detail here but want to remind readers - including any who are considering adopting or adapting this program - that learners from the remote and rural communities bring complex needs with them. Many have family obligations that weigh on them while they are away. Family crises occur and the learner can either feel negligent for not responding or leave the program and lose an opportunity. Some in every class so far have brought some level of learning or physical disability. For many their educational experiences have been Below standard. The whole business of coming to town is, for some, a source of culture shock - and for others a place of too many temptations. This implies providing a lot of personal support. For example the final two CAAR sessions - one CofQ and one Pre-Apprenticeship - benefited from the presence of a CAAR team member primarily dedicated to that role.

9. Have a Round Up! Over time there have been several learners in the CofQ program who have come very close to success. We urge SLAAMB, even though it would have to happen after the close of CAAR funding, to bring back as many of those as is possible. Give them enough of a "cram course" to qualify them to re-write the CofQ exam. Give them an opportunity to succeed - and give CAAR an opportunity to improve its impact numbers.