For my program evaluation course's final, we were asked to respond to a RFP for an actual impact evaluation. It was a partner paper, but here was the section that I completed.
UN
WOMEN: UNITED NATIONS ENTITY FOR GENDER EQUALITY AND THE EMPOWERMENT OF WOMEN
MAY
2012
Impact
Evaluation Proposal of the
Safe Cities Free of Violence Against Women and Girls Project
in New Delhi, India
I.
Executive Summary
Purpose: The Safe Cities Free of Violence
Against Women and Girls Project in New Delhi, India aims to be the first proven
model on how to prevent and reduce sexual harassment and violence against women
and girls in public spaces for eventual adaptation by local authorities and other
decision-makers worldwide, in partnership with grassroots women’s organizations
and community groups. This program is targeting slum areas and impoverished
neighborhoods in five different cities worldwide: Cairo (Egypt), Kigali
(Rwanda), Port Moresby (Papua New Guinea), Quito (Ecuador), and the focus of
this impact evaluation, New Delhi (India). Ideally, these efforts will empower
women and their communities to end this gender-based violence. The purpose of
this document is to lay out a strategic framework that assesses the impact of
this program in New Delhi.
Scope and Methods:
A variety of methods
will be used to evaluate the impact of this program. We propose a
quasi-experimental evaluation in conjunction with a comprehensive pre- and post-intervention
evaluation. Quantitative data will include existing and relevant government
statistics and surveys, as used in the baseline study already published.
Qualitative data will include open ended questionnaires, interviews, focus
groups, and video footage of public spaces. Data was collected from the nine
geographical districts of Delhi: Central, East, New, North, North-East,
North-West, South, South-West, and West. Within these districts, two to three
survey sites were selected which included markets, train stations, bus
terminals, parks, and school campuses.
Recommendations: To ensure the success of this
evaluation, we recommend encouraging feedback from key-stakeholders and policy
makers regarding this evaluation and progress of the program thus far, review
other Safe Cities programs and evaluations to consider implementing their best
practices, and increase outreach to religious leaders and men’s organizations
for their participation and support.
II. Introduction
This
is a proposal to create and carry out an impact evaluation for the Safe Cities
Free of Violence Against Women and Girls program in New Delhi, India. Using the
model outlined in this report, Lauren Deutsch and Allison Primack from The
George Washington University in Washington, DC, will
carry out an evaluation to determine the extent to which this program has
reduced gender-based violence, increased mobility of women and girls in public
spaces, and increased awareness about the enjoyment of women and girl’s rights
to access and use public spaces.
III. Background
In a
June 2011 survey, Thomson Reuters ranked India the fourth most dangerous place
for women in the world based on high rates of female infanticide, foeticide and
sex trafficking[1]. While India
has made impressive strides to protect their female citizens, violence against
women remains prevalent in New Delhi. In order to determine an
appropriate methodology for this study, it is crucial that scoping studies are
conducted to gain a deeper understanding of the nature of sexual violence and
harassment in New Delhi. Special care has been taken to ensure that these
activities are gender, age, and culturally appropriate.
Description
of the Program
Safe
Cities Free of Violence Against Women and Girls program aims to be the first
proven model on how to prevent and reduce sexual harassment (also referred to
as “eve-teasing”) and violence against women and girls in public spaces by
partnering with local authorities, global decision-makers, and grassroots
women’s organizations and community groups. This model can be scaled up in many
contexts—furthering women’s empowerment equality, while enhancing the quality
of urban life for all.
Key
Strategies and Components
The
Safe Cities for Women and Girls key strategies and components are outlined by
the programs description, as follows:
•
furthering
enhanced laws, policies and protocols to address these forms of violence
explicitly, end impunity for abusers, and strengthen governmental
accountability;
•
carrying
out surveys and data collection, to capture the magnitude and nature of sexual
harassment and violence in public spaces, gage the attitudes of men, women and
young people; as well as to inform public policies and assess progress;
•
undertaking
“safety audits”, considered a best practice internationally, which involve
women and other community members in identifying unsafe areas and needed
interventions, mapping “hotspots” of risks of assault and harassment, and
formulating solutions in dialogue and partnership with local authorities;
•
improving
municipal planning in various sectors, including urban design through the
introduction of practical safety measures by local authorities, in
collaboration with women and their communities — such as changes in street
lighting, signage, location of bus stops, and access to emergency hotlines in
bus and train stations;
•
advancing
prevention efforts, including through mass media campaigns and community
mobilization on `zero tolerance’ for sexual harassment and lewd behavior
towards women — with a special focus on engaging young people and men of all
ages;
•
training
and improving the capacities of local authorities and other key actors to
respond to violence against women and girls in public spaces, including the
police, judges, social services and the media;
•
applying
gender-responsive budgeting, a methodology utilized to analyze resource flows
and their responsiveness to women’s needs and rights, in order to identify the
level of existing allocations to address violence against women issues, inform
budgetary appropriations, and track relevant public sector investments;
•
crafting
and pursuing a first-of-a-kind rigorous impact evaluation, in order to
demonstrate the model’s value and relevance for policy-makers and others in
cities and countries around the world working towards making cities safer for
women and girls.
This
list will be used later in the Theory of Change logic model as the activities
used to create the outputs and outcomes. The strategy set forth in this report
will fulfill the last component on the list — a rigorous impact evaluation.
Relevant
Past Research and Evaluation Findings
In
order to effectively determine an impact evaluation design for the Safe Cities
Program in New Delhi, it is useful to look to the baseline data - a
joint-action research survey initiative carried out by the Indian Department of
Women and Child Development, Government of New Delhi, JAGORI (which means
“awaken women!” in Hindi), the UNIFEM (now UN Women) South Asia Regional
Office, and UN Habitat. The survey was conducted using a sample of 5,010 women
and men in 23 areas and 50 interview sites between January and March 2010. Data
was also collected from “common witnesses,” people who live close to large
public spaces that frequently observe sexual harassment crimes.
The
study found that a high percentage of the respondents believe that sexual
harassment in public places is the single most important factor that renders
New Delhi an unsafe city. 85.4% women, 87% men, and 93% common witnesses
responded that these problems are “rampant” in New Delhi. Verbal harassment is
the most common form, followed by visual harassment and stalking. This
harassment is most commonly experienced in market places, metro stations, areas
around schools and colleges, and industrial areas.
Many
factors lead to women feeling unsafe in public spaces. These include lack of
gender-friendly functional infrastructure such as public transportation
systems, public use of drugs and alcohol, lack of clean and safe public
restrooms, and lack of effective and visible police presence. If they were ever
subject to sexual harassment, 58% of women stated that they would not even
consider going to the police, and only 0.8% of the women surveyed have ever
reported incidents. Police are avoided because it is perceived that they will
not do anything to solve the problem, they will trivialize the incident, or
shift blame to the victim. Additionally, over half of respondents reported that
if they saw an incident occurring that they would not get involved or contact
the authorities to help.
Relevant
Literature on the Program
IV. Evaluation Questions
This
impact evaluation establishes three goals to determine the extent to which the
program has:
1.
Reduced
gender-based violence;
2.
Increased
mobility of women and girls in public spaces, and;
3.
Increased
awareness about the enjoyment of women and girl’s rights to access and use
public space in New Delhi.
In
order to determine if these goals are achieved, we should specifically answer
the following questions. These questions were derived from the baseline data
and its key components. They are aimed to measure program impact, program
outcomes, program outputs, and the effectiveness of the program’s strategies
and processes:
1.
As
a result of Safe Cities initiatives, do women perceive to be safer in public
spaces?
2.
As
a result of Safe Cities initiatives, are women more inclined to report
incidents to the police?
3.
Has
the number of sexual harassment cases decreased in public spaces since the Safe
Cities program began, or changed the type and occurrence of these incidents?
4.
Has
the Safe Cities program led to laws or government led initiatives to promote
gender equality and women safety in public spaces?
5.
Has
the Safe Cities program produced policies or budgets to improve infrastructure
(i.e. public transportation and public restrooms) to make it safer for women?
6.
Has
the Safe Cities program changed men/boys perspective of appropriate behavior
towards women in public spaces?
7.
As
a result of Safe Cities initiatives, are citizens more likely to intervene if
they witness sexual harassment occurring in a public space?
8.
Have
the multimedia methods effectively shaped public perception of these issues?
9.
Were
there any unanticipated outcomes from the Safe Cities program?
10. How well did the Safe Cities program
implement its core strategies?
11. What were the main challenges of
effective implementation, and how were they overcome?
V. Evaluation Design
This
impact evaluation design will incorporate gender equality and human rights
approaches, participatory techniques, a mixed-methods approach incorporating
qualitative and quantitative methodologies, construction of counterfactuals to
help assess impact attribution, and a longitudinal study involving baseline,
mid-term, end line, and ex-post assessments.
In
order to better understand what impact will be evaluated, it is critical to
outline the Safe Cities Theory of Change (TOC). By analyzing the long term
goals of the program and their supporting assumptions, we will be able to
easily connect the preconditions and requirements necessary to achieve the goal
of reducing sexual harassment in public spaces, identify which interventions
will be most effective, and develop indicators to measure the success of these
interventions.
Long
Term Goals and the Assumptions Behind Them
The
ultimate goal of this initiative is to prevent and reduce sexual harassment of
women and girls in public spaces in New Delhi, India. As previously mentioned,
this is to be achieved by three broad goals. First is to reduce gender-based
violence, which assumes that this type of violence is prevalent, and that women
and girls currently do not feel comfortable using available resources. Second
is to increase mobility of women and girls in public spaces, which assumes that
this fear of harassment is preventing women and girls from using these public
spaces safely, and that there is a lack of infrastructure to help prevent these
crimes. The final goal is to increase awareness about the enjoyment of women
and girl’s rights to access and use public space in New Delhi, which assumes
that there is a lack of community effort to raise awareness on this issue. This
is the basis of the logic used in our TOC model.
Connecting
Preconditions and Requirements Necessary to Achieve the Long-Term Goal
The
following logic model is a diagram mapping the inputs, activities, outputs, and
outcomes of this program. The activities/key components are adopted from the
list of “key strategies and components,” as listed earlier in the report. This
model can be seen on the following page.
There
are some external factors that could affect this logic model, the most
prevalent being the cultural differences between India (New Delhi in
particular) and the United States. The response to these programs may be
different based on their cultural norms, their patriarchal society and
historically misogynistic beliefs. Sexual acts of violence such as rape are
still very much taboo in Indian culture and the timeframe associated with this
logic model may be too progressive.
Indicators of Success
In
order to determine if the program is successful, several indicators must be in
place to analyze various outputs of the program. The indicators are outlined in
the table below:
|
Output
|
Indicator
|
Population Measured
|
Performance Threshold
|
|
Women
better protected by the law
|
More
provisions about sexual harassment added to the law
|
Lawmakers,
Law Enforcement Officers
|
At
least two provisions added per year, or in the process of being added
|
|
Dangerous
“hotspots” identified and targeted for infrastructure improvement
|
Safety
audit results, plans for infrastructure improvements set into place and put
into action
|
Women/Girls,
Lawmakers, City/Urban Planners, Law Enforcement Officers
|
Make
infrastructure improvements in at least two hotspots per district per year
|
|
Changed
public perception of sexual harassment issues
|
Public
opinion surveys
|
Local
Media Outlets, Women/Girls, Men/Boys
|
At
least 25% more aware of sexual harassment issues per year
|
|
Women
feel safer in public spaces
|
Interviews
with women, public opinion surveys
|
Women/Girls,
Law Enforcement Officers
|
Safety
ratings increase by 25%
|
|
More
public money allocated to community resources for women
|
Budgets,
provisions added to the law
|
Lawmakers,
Community Directors
|
At
least 10% more funds allocated to community resources
|
|
Police
better equipped to properly deal with these cases
|
Surveys
rating the additional training programs, surveys from women’s perceptions
|
Law
Enforcement Officers, Lawmakers
|
85% of
officers complete training in first year, 100% by second year
|
Evaluating
Impact
The
RFP presents three main options that are to be considered for the impact
evaluation design to determine if the aforementioned goals were achieved:
1.
Experimental
evaluation, using randomized trials with control and treatment groups;
2.
Quasi-experimental
evaluation, involving cluster trials (to compare the interventions’ effects on
beneficiaries with comparable communities in which the intervention was not
implemented; and/or
3.
Comprehensive
pre- and post-intervention evaluation, without a comparison or treatment or
control group.
For
this program, an experimental evaluation is not feasible. Due to the sensitive
nature of topic, it is not ethical to allow some women to participate but then
withhold victim services if they were in fact required. We propose to use a
quasi-experimental evaluation, not only because it is indeed viable, but it is
also only second best to a random control trial and allows us to compare
communities in New Delhi with one another. A comprehensive pre- and
post-intervention evaluation will be used in conjunction with this to support the
quasi-experimental results by using quantitative data from the government.
VI. Data Collection
To
conduct our impact evaluation, we will be collecting a wide range of data from
various sources. These methods are outlined below.
Sources
of Data Available
Unfortunately,
due to its taboo nature in New Delhi, there is not much existing data on this
issue yet. The most comprehensive report we currently have is the baseline data
collected in 2010. However, it would be useful to obtain government data on
reported incidents, and compare this information to the baseline data. It may
be that the number of reported incidents may go up even though that the total
number of sexual harassment cases decrease — this would be helpful to see in
the study, because it would show that the program is effective in empowering
women to report these cases to the police.
Measures
Used to Address the Research Questions
Various
kinds of data will be collected before, during, and after the program in order
to portray an accurate depiction of the city at that time. Quantitative data
will include existing, relevant government statistics and surveys, as used in
the baseline study. Qualitative data will include open-ended questionnaires and
interviews of women who are involved in the program’s activities, and of random
people approached on the street. Focus groups will be hosted with program
directors and other leaders to brainstorm ways to confront these issues from a
legal standpoint. Additionally, video footage will be collected in public
spaces as evidence for the researchers to observe and record changes over time.
Data
collection methods
In
order to conduct a rigorous impact evaluation, data will be collected at four
points in time: before the program begins (baseline), halfway through the
program (mid-term), at the end of the program (end line), and at the end of all
activities that supported the program (ex-post). Since the baseline evaluation
has already been conducted, we can only rely on the resulting survey data. For
the midterm evaluation moving forward, all of the data mentioned above will be
collected. We will rely on participants to describe incidents based on memory
and provide reactions to baseline data during the midpoint evaluation.
Sampling
procedures
For
this impact evaluation, both non-probability and probability sampling methods
will be used to create a complete picture of the situation in New Delhi. For
the interviews and focus group discussions, it is essential to use purposive
samples in order to include community leaders, and women having endured a
variety of incidents. The surveys and questionnaires will be distributed at
random to the population. This will be achieved through stratified random
sampling, dividing the population based on their age, location, and other
demographic factors in an attempt to compile a diverse data set. The police
data collected will be sampling the entire population.
Limitations
to Validity and Reliability
The
nature of the data in this study poses some threats to validity and
reliability. Fortunately, there are precautions that can be taken to minimize
the impacts of these limitations on our impact evaluation:
•
Selection
is a major threat to this study. Because sexual harassment is not openly
discussed in Indian culture, it is expected that not every woman will be openly
willing to participate in the study. Whatever group volunteers to opt in to the
study may not truly represent the feelings of the majority. To attempt to
correct for this it should be made clear upfront that their participation or
responses will not jeopardize their safety or well-being so as to successfully
reach along the entire spectrum of respondents.
•
History
can be a threat to internal validity, because a particular event in the region
pertaining to sexual harassment may cause a particular reaction or change that
has nothing to do with the Safe Cities program, and thus throw off the results.
For this reason, it is essential to be aware of all events pertaining to sexual
harassment in the New Delhi region, recording these events in the study,
especially those leading directly to changes in government intervention, the
response over the duration of the study, and violent incidents shared in the
media.
•
Attrition,
also referred to as mortality, can also affect the results of the study,
especially pertaining to individuals we hope to track over time through
interviews and surveys. If for any reasons the participant feels like they are
scared or threatened by being involved, we could lose their input or support.
To make sure this does not occur, it is essential to maintain anonymity, and go
to extra measures to ensure that they feel safe sharing private and somewhat
embarrassing information.
•
Spillover
effects are likely to occur with the onset of this program and affect sexual
harassment and violence outcomes in areas outside of New Delhi that are used
for comparison. This is most likely to happen if the Safe Cities program
incites new government regulations regarding sexual harassment in public
places. This is not necessarily a bad thing because the program is helping more
women than expected, but it could also contaminate the comparison group, which
makes it difficult to determine the actual impact of treatment. Indicating if
and when local or national changes are made can combat this.
•
Measurement
is a large threat to both validity and reliability in this study. There are
concerns about whether the data collected is actually representative of the
population, and if it is consistent. To help combat this, it is extremely
important that the same measures are used for baseline, midpoint, and endpoint
evaluations in order to avoid the easy pitfall of instrumentation issues.
Design
Matrix
The
following design matrix summarizes the data collection techniques in a table
format:
|
Evaluation
Question
|
Information
Source
|
Sampling
|
Data
Collection Mode/
Respondents
|
Specific
Questions
|
|
As a
result of Safe Cities initiatives, do women perceive to be more safe in
public spaces?
|
Women
and girls
|
Stratified
random sampling of women and girls
|
Survey,
Open-ended questionnaire, Interviews
|
- On a
scale of 1-5, how safe do you feel in Space X?
- What
makes you feel safe in public spaces?
|
|
As a
result of Safe Cities initiatives, are women more inclined to report
incidents to the police?
|
Women
and girls
|
Stratified
random sampling of women and girls
|
Survey,
Open-ended questionnaire, Interviews, Government Data
|
- Have
you ever reported a sexual harassment case to the police?
- Would
you feel comfortable reporting a sexual harassment case to the police?
|
|
Has the
number of sexual harassment cases decreased in public spaces since the Safe
Cities program began, or changed the type and occurrence of these incidents?
|
Lawmakers,
Law Enforcement
|
Collection
of all police/
government
data
|
Government
Data, Video footage
|
N/A,
only data analysis
|
|
Has the
Safe Cities program led to laws or government led initiatives to promote
gender equality and women safety in public spaces?
|
Lawmakers
|
Collection
of government data
|
Government
Data
|
N/A,
only data analysis
|
|
Has the
Safe Cities program produced policies or budgets to improve infrastructure
(i.e. public transportation and public restrooms) to make it safer for women?
|
Lawmakers,
City/Urban planners
|
Collection
of government data
|
Government
Data
|
N/A,
only data analysis
|
|
Has the
Safe Cities program changed men/boys perspective of appropriate behavior
towards women in public spaces?
|
Men and
boys
|
Stratified
random sampling of public
|
Survey,
Open-ended questionnaire, Interviews
|
- What
type of behavior is appropriate towards women?
- Do
you think women and men are treated the same in public spaces?
|
|
As a
result of Safe Cities initiatives, are citizens more likely to intervene if
they witness sexual harassment occurring in a public space?
|
Women
and Girls, Men and Boys
|
Stratified
random sampling of public
|
Survey,
Open-ended questionnaire, Interviews, Video Footage
|
- Would
you intervene if you saw someone being sexually harassed?
|
|
Have
the multimedia methods effectively shaped public perception of these issues?
|
Local
Media Outlets, Women and Girls, Men and Boys
|
Stratified
random sampling of public, Purposive sampling of local media
|
Survey,
Open-ended questionnaire, Interviews
|
- What
is sexual harassment?
- Have
you seen the local media campaign? Do these issues apply to you?
|
|
Were
there any unanticipated outcomes from the Safe Cities program?
|
Community
directors, Women and girls
|
Purposive
sampling of community directors, public
|
Open-ended
questionnaire, Interviews
|
- What
was the most successful aspect of this program?
- What
did you gain from this program?
|
|
How
well did the Safe Cities program implement its core strategies?
|
Community
directors
|
Purposive
Sampling of program and community directors
|
Survey,
Open-ended questionnaire, Interviews
|
Ask
questions about success of the core outcomes
|
|
What
were the main challenges to effective implementation of this program, and how
were they overcome?
|
Community
directors
|
Purposive
Sampling of program administrators, community directors
|
Open-ended
questionnaire, Interviews
|
- What
was the biggest challenge in implementing this program?
|
No comments:
Post a Comment