What were our protocols?
Surveyors will encounter many different situations making it necessary to develop protocols that include the information and procedures they need for each unique situation. While we describe the protocols that we developed in this section, we recognise that organisations interested in this topic may need to develop their own protocols tailored to their goals and the communities they will be surveying.
We recommend the following considerations when developing the surveying protocols:
Surveyor work in pairs
Not only for safety purposes but also logistically, we have found that having surveyors work in pairs makes surveying more enjoyable and efficient. The one exception that can be made to this recommendation is having community researchers survey visit the neighbourhoods they live in alone. Community advisors also play a crucial role in giving context around neighbourhoods that allow us to keep surveyors safe and use our time in the field efficiently.
Surveyors work after 5:00 pm until sunset on weekdays (in spring and summer)
We have found that surveying before 5:00 pm yields a low response rate because of working hours. We also recommend stopping surveying before or shortly after sunset for safety purposes. Keep in mind the sunset time changes depending on the season, which will influence how much time surveyors spend in the field and survey responses. Our weekday shifts were approximately three hours long.
Surveyors work from 2:00 pm until sunset on weekends
We have found the highest response rate on weekends when we visited households after lunchtime. Shifts were approximately six hours long on weekends with a half-hour break in between.
Incentives increase response rate
We provided a $20 incentive to participants as cash or an online gift card. We did not mail any gift cards or cash because we wanted to track who received our incentives, which required an in-person or online transaction.
Provide letters and flyers
For residents who were not available at the door, we left letters and flyers that provided all the information they needed to complete the survey online. We expected an average of two to five responses per three-hour shift, and a roughly equal amount completed online by using the letters or flyers. Click here to jump to the resources section.
Develop a field handbook
We recommend creating a one-stop-shop handbook for surveyors with all the information and procedures they need to respond to the different situations they may encounter when in the field.
Develop protocols with community members
We developed our protocols through discussion with community members of the areas we intended to survey, as well as organisations that regularly survey or canvass in Toronto’s suburban communities. We developed two protocols that we describe below: field surveying and high-traffic protocol.
A. Field Surveying
Field surveying was our most randomised approach. Surveyors used physical and digital materials (e.g., route handouts, letters, flyers, and Google Sheets) to visit each household in their route. Click here to learn more about the resources we used. Surveyors were prepared for four responses:
When residents agreed to complete the survey, surveyors found the resident’s unique code available on the Google Sheets to enter on the survey screen, and then gave it to the resident. Once the survey was complete, surveyors filled out a receipt, gave the resident a copy of the receipt with the cash incentive, and thanked them for their participation. Surveyors also recorded the response on Google Sheets.
At least a quarter of residents refused to do the survey each shift. In about half of these cases, we found that a “no” response meant “not now.” Therefore, surveyors responded to “no” by asking residents if there was another time that might work better for them, or if they preferred to do the survey online. In most cases, residents agreed to meeting at a different time. In some cases, however, a “no” meant a “no”; surveyors thanked these residents, gave the letter or flyer just in case they changed their mind, and left.
Later or online
Some residents mentioned immediately after opening the door that they are unavailable at the moment and surveyors should come back later. In these cases, we followed the protocol for the “no” response to schedule a time convenient for the resident. In other cases, residents preferred to complete the survey online, and surveyors gave these residents their letter or flyer. We monitored residents’ responses in the survey software; if they still did not complete the survey, surveyors visited residents again to remind them to complete the survey.
Did not open their door
For approximately half households, residents did not open the door. Surveyors marked these houses in their route handout and visited them again at the end of the same shift. We found that visiting each house at least twice within the span of three hours doubled the number of opened doors. However, about a quarter of houses still did not open their doors at the second visit. In this case, surveyors left a letter for residents to complete the survey online. Surveyors visited the same route on a second day to repeat the same process (i.e., visit 1 at the beginning and visit 2 at the end of shift). In total, surveyors knocked on each house at least four times spanning two days. If residents still did not open the door at the fourth visit, surveyors left a flyer for them to complete the survey online. Surveyors also noted if the letter they left in their mailbox on the first day was still in their mailbox or had been taken by the resident.
Case Study: Surveying during election season
B. High Traffic Protocol
We used a second surveying approach that we called the high-traffic protocol (HTP). The aim of this approach was to have surveyors stand at strategic locations –such as grocery stores, community hubs, libraries, and apartment buildings –and approach residents as they entered or exited the location. We used this approach to achieve the following two goals:
- Increase the overall response rate
- Increase the representation of certain groups
Pilot Testing HTP
We selected the HTP locations through discussion with community members and viewing Google Maps. There were some challenges that we encountered when implementing HTP. The major challenge was ensuring that interact with residents from inside our sampled neighbourhoods. We minimised the number of residents from outside sampled neighbourhoods by conducting HTP at locations within the neighbourhoods and focusing on local locations such as grocery stores and libraries rather than large malls. However, it is difficult for surveyors to identify if a resident resides in the area unless they ask the resident. It might be essential for surveyors to ask the resident if they reside in the area. We do not recommend asking specific questions that might arouse suspicion; instead, we recommend conducting HTP at local locations, and asking general questions such as “we are interested in getting responses from those living in this area. Do you live around here?”
HTP is also not a fully randomised protocol compared to field surveying, but there are certain strategies that can make it quasi-randomised by approaching every fifth resident and spending an equal amount of time at different types of locations (i.e., restaurants, community centres, libraries, grocery stores).
Once per week, we invited surveyors to a meeting we called replenish. Prior to each replenish, the survey coordinator created a replenish protocol that outlined who needs to attend the replenish, what materials they need, and the protocol for transferring materials between surveyors. At the replenish, each surveyor (1) received route materials (handouts, letters and flyers), (2) received cash incentives, (3) received access to the Google Sheets they will need to use in the next week, (4) provided the survey coordinator with completed receipts, and (5) asked any questions about the schedule or challenges in the field.
There were four types of materials we exchanged with surveyors at replenish:
Handouts: Mails, letters, and route handouts
- The survey coordinator organised mails, letters, and route handouts for each surveyor based on the replenish protocol.
- If any surveyor was supposed to transfer route materials to another surveyor, then the coordinator ensured that both surveyors were on the same page.
Money: Each route is allocated a set amount of money based on average response rate.
- At each replenish, we first discussed how much money each surveyor had.
- Per surveying shift, we expected 0 to 4 responses, which meant surveyors needed up to $80 per shift. This number was multiplied by the number of days the fanny pack will be in the field until the next replenish.
- When the survey coordinator gave cash to surveyors, both signed a transfer form and recorded in a paper log located inside the money box that stored all the cash. Each moment cash, receipts, and/or fanny pack and materials was transferred between surveyors or survey coordinator, we completed the form to document the exchange. Surveyors signed the form and sent a photo of it to the survey coordinator. This strategy promoted accountability of all surveying materials when in the field.
- When surveyors ran out of cash, they gave gift cards until the next replenish.
Receipts: Each resident who responded in-person was issued a receipt.
- At each replenish, the survey coordinator received all receipts from receipt books.
- The survey coordinator also ensured that each receipt was an actual receipt and not a duplicate. The coordinator removed duplicates by recording the household ID associated with each response on completed receipts.
Access: The survey coordinator granted each tablet access to the Google Sheets that surveyors will need until the next replenish meeting.
- Each tablet was associated with a number from 1 to 4 that corresponded to one of four Google Drive Gmail addresses.
- At each replenish, surveyors ensured that their tablet has access to the Google Sheets.
D. Field Reflections
Our surveyors visited neighbourhoods with a considerable diversity of ethnicities, races, and backgrounds. It was essential for us to document and/or discuss their experiences interacting with residents. We developed a Community Voices Reflexive Log with the following prompts to be answered after each shift:
- Who answered the door?
- Who accepted/declined the survey?
- Describe your interactions with resident(s)
- Describe the characteristics of resident(s)
- Describe your impressions of the field handouts and surveying protocols
- Describe the neighbourhood you visited today
- Discuss how your social location influenced your interaction with residents