803: Program Evaluation & Design

Sunrise of Life is an organization with the aim of helping street children in Arusha, Tanzania.  Arusha is a very busy tourist city, with many people visiting on their way to visit one of the many national parks nearby: Arusha, Mount Kilimanjaro, Mount Meru, and Tarangire.  Tourism itself is good for the economy of the country, however, it also brings many problems.

Arusha has a huge number of street children: estimates range between 300 and 3000.  Why do they come to Arusha? Being a busy city means that their chances of receiving money by begging is then just that much higher from tourists.

Come along with me as we walk the streets of Arusha for a day.  You will see children as young as 5 walking the streets, carrying their only belongings.  They search for food, which is challenging early in the morning.  Aimlessly they wander the streets, looking for a safe place to spend their time.  By later afternoon they try to find odd jobs of carrying food, or working in the market, in return for small amounts of money or food.  At night these children sleep in football stadiums, behind shops, or in the public parks.  Problems of street children here include disease, rape, trouble with the law, stealing, addictions and a total lack of education.

The Tanzanian government does very little to help these children, and there is a huge shortage of organizations working specifically to help the street children here: and so Sunrise of Life is such an essential and needed one.

Sunrise of Life works in three main parts:

  • Outreach & Drop In Center

The first step of SOL’s work starts on the streets of Arusha.  The team will head onto the streets both during the day and at night, in an effort to get to know and form relationships with the children of all ages.  The children who are on the street during the day are usually newer to the streets, and so education about the dangers of this lifestyle is very important. Night outreach targets the long-term street children, who often do not want help because of their addictions.

The Outreach Center also has a number of beds for children who show a desire to live and begin a better life away from the streets.  After several months of counselling and living at the Outreach Center, if the social worker deems them ready, then they can be relocated to the residential center.  A vital part of the outreach center is reuniting children with their home communities.  Many children come to the streets against the will of their home communities, and so when they are given the opportunity many would like to be brought back home.  The primary effort of SOL is always to reunite the children with their families, and if this is not possible then other possibilities are explored.

  • Residential Center

The Residential Center is located in Maji ya Chai, a small village about an hour outside of the city of Arusha.  The Residential Center is made up of 3 dormitories, an office, a staff/volunteer house, a nurses room, a kitchen and a classroom.  The center can house up to 75 children.  The basis of this is a place for children to live who cannot be reunited with their families.  Some of the reasons for this may be addiction in a family member, loss of parents, or food not being provided.  The children then can come live in the safe environment at the residential center, where they are provided with an education (a requirement), extra educational support, and all of their basic needs.

There are many projects at the Residential Centre with help move the centre towards self-sustainability.  The milk cow project (4 cows), a garden project (providing most of the vegetables for all children to eat), a chicken project (100 chickens for laying eggs) and a goat project (6 goats).  All of these projects provide not only food, but also revenue.

IMG_9452IMG_1077

  • Transition Program

When the children complete their basic education, they are then given the support they need to continue their education.  They are supported in finding a place to live close to the school of choice, and are given more responsibility to look after themselves.  The main way the students have to gain responsibility is by providing food for themselves.  By pushing the students to further their education, allowing them to find a place to stay, and forcing the students to provide their own food SOL hopes to teach these students to be more responsible and self-sustainable.  In addition to this help, the students in the transition program also receive support to find jobs.

IMG_1126

Mission Statement of SOL:

  • Sunrise of Life provides assistance to the street children of Arusha, Tanzania. SOL dedicates itself to assisting these children with the basic needs for survival: food, shelter, healthcare, and education.  SOL believes in creating opportunity for those children who have no opportunities available to them.

Value Statements:

  • SOL believes that every child no matter where they are in the world deserves an equal chance at life.
  • SOL believes that the basic needs such as food, shelter, education and health care are necessary to help these children succeed in life.
  • SOL believes in creating opportunity such as a healthy living environment, education, and health care where there is no opportunity by introducing programs that are sustainable and will teach a child how to become independent in future life.
  • SOL believes in using each dollar donated to the best of its ability.
  • SOL believes in being accountable to its donors.
  • SOL believes in an atmosphere of openness within its organization and towards its donors.

http://www.sunriseoflife.org/

Context, Purpose & Questions

The purpose of this evaluation is two-fold:

  • Internal: as an organization we wish to find and fix implementation problems, and increase efficiency and effectiveness.
  • External: as an organization we wish to be more accountable to our stakeholders and to improve transparency.

A holistic assessment of the implementation process.

IMG_1066

Specific Evaluation Questions:

  • Do important stakeholders feel that SOL is an open organization that is accountable to them? Do they trust that their money is being spent wisely? Do they value the work that SOL is doing?
  • Is the method of doing street outreach effective?  Are we reaching the right street children?  Are we providing help to the most vulnerable and needy of the street children?
  • Are the basic needs of the children being provided at an adequate level at the residential center?  Do they get enough nutritious food to allow them to grow and develop in a healthy way?  Is the shelter provided not only comfortable, but also safe from outside predators?  Are they receiving enough health care (medications, doctor visits, etc.)?  Is sending the children to public education doing enough to prepare the children with the knowledge necessary for life after school?
  • Are the children adequately being taught the skills necessary to live a healthy, productive and self-sustaining life as an adult?

Data Collection & Analysis

For the purpose of this evaluation I will collect four types of data, with the purpose of:

  • Educating: This will inform the organization about the needs that are still there, and what is working well.
  • Developing Priorities: This will help the organization decide who needs the most help, and set a program plan or goals.
  • Acquiring Resources: The information collected will give a more clear picture of what is needed, and so can be used to obtain grants and show why funding is needed.
  • Evaluating Programs: The data collected will tell us if the programs are effective, and if we can show that our program is working then this will result in more funding.
PRIMARY DATACollected by project through direct interaction. SECONDARY DATACollected by others, available for use by me.
Quantitative– Measurable and counted.What, Who, When, How many, Where Interview given to two Transition Program students. Program artifacts:Website (sunriseoflife.org)

Planning documents on GoogleDrive from SOL: yearly budgets, manager reports, meeting minutes, street youth history reports.

 

Interview given to four Residential Center students.
Qualitative– acquired through listening to people, observing situations, recording perceptions to understand how people feel and why they feel that way. Input from community members:Interview Questions for SOL

Staff members (Peace, Jackson)- Interview completed by a board member visiting Tanzania.

 

Input from Transition Program students (on interview) completed by a board member visiting Tanzania.

 

Input from older children at Residential Center (on interview) completed by a board member visiting Tanzania.

 

Railroad Childrenhttps://www.railwaychildren.org.uk/media/11703/sts-full-report.pdf

 

Mozambique StreetYouth Research

 

Input from other stakeholders:Board members who have been to visit there (Tom, Charmaine) done by phone interview.

 

Phone interview with three funding stakeholders (Rotary Club member, business member, individual personal donator)

 

The Evaluator will:

  1. Prepare interview questions & interview to be sent to Tanzania along with a visiting board member.
  2. Share the Logic Model and Program Theory as a tool to show why this evaluation is being done, and the end use.
  3. Board member will complete interviews in Tanzania, Evaluator will complete phone interviews.
  4. Configure all responses from interviews, as well as applicable data from secondary sources, into an Excel document that is organized according to key evaluation questions.
  5. Analyze data and look for commonalities.
  6. Use both Primary & Secondary data to understand causes and look for possible alternative explanations.
  7. Generalize the findings and answer key evaluation questions.
  8. Make recommendations for further development of SOL, keeping in mind factors such as available staff, funding, and cultural differences.

 imputs outputs outcomes impact

2 charts

Approach to Enhance Use

When working with street children, in particular, evaluation use matters.  As Dr. Michael Patton stated, “there are high stakes of data-informed, timely decision making.”  The results that this evaluation uncovers need to be displayed in a clear and open way, but also in a way that gives clear recommendations for how to move forward.

The primary evaluation use for this evaluation will be Instrumental Use: the results that are collected will be applied to the practical operations within the program.  Examples of changes that may have to be made because of the evaluation:

  • Changing the budget to accommodate the changing needs (more money towards street outreach, for example, with some money being taken away from the transition program.)
  • More regular accountability reports being sent out to the funding stakeholders by way of email or bi-monthly newsletters.
  • Hiring a nurse who will take responsibility for the health of each of the children living at the residential center.
  • Combining responsibilities of two separate positions into one, and changing job responsibilities for various staff members.
  • Creating more clear behavior guidelines for the children living at the residential center in order to improve the safety of other youth.

IMG_9768Using the DAC Criteria for Evaluating Development Assistance:

Relevance The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. Does the mission statement of SOL align with the needs of the street youth it aims to help?Are the objectives that are set out by the organization the ones they really hope to accomplish?

 

Effectiveness A measure of the extent to which an aid activity attains its objectives. Does the reality of the work being done reflect what the mission statement hopes to accomplish?Do we need to adjust the mission statement based on a changing need?
Efficiency Efficiency measures the outputs — qualitative and quantitative — in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. Using the budget to compare to the actual mission and goals of SOL to decide whether the budget reflects the mission. 
Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. Are street youth truly being helped and impacted by the outreach being done?Are children living at the residential center being cared and supported in an adequate way?

Are there positive success stories of the transition program of youth who have ‘graduated’ and moved to an independent, healthy, self-sustaining lifestyle?

Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.  Is there a way to move SOL away from being solely funded by the Western world, and have it be run and funded by the people and community they wish to serve?What are some ways to improve the sustainability of the organization, while still allowing the Western funding to continue?

http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

 

After completing the evaluation I would produce two reports:

  1. A report for the board members in Canada, USA, and Tanzania outlining the results in detail, and with my recommendations.  At the next board meeting we would discuss the recommendations, and decide which of them are feasible to implement.
  2. A brief, clear list of changes and tasks for the staff members working in Tanzania.

IMG_9449