The Bellingham Residential Survey Makes My Head Hurt

This “Residential Survey” of Bellingham residents is a quasi-yearly affair with this stated purpose: “... the Survey is to seek feedback from residents [bolding mine] about their experience as citizens of the city of Bellingham. A combination of 50 multiple choice questions, 3 rank order questions, and 6 open-ended questions aim to gather information from respondents about their demographic details, quality of life, satisfaction with City services, opinions on budget priorities, views on challenges facing the community, and ideas for the future.” Tim Johnson at Cascadia Weekly did a good job reporting on the actual survey results, such as they are, which you can read in his Gristle column of 29 March. However, the underlying results are based on a flawed mechanism that mainly surveys homeowners.

The annual survey of “resident” attitudes is essentially an exercise based on an unverifiable “random” sample from a list of 7,000 Bellingham addresses prepared by the city and then provided to the survey creation team at WWU. Residents were then notified by postcard to take the on-line survey. Eighty-four percent of the respondents on this year’s survey were identified as homeowners. Only 14% were renters although they comprise over 50% of the city’s population. Over half the respondents were over 55 years old and in the age group 18-30 there was only a 6% representation. Keep in mind, renters and youth also tend to be those with lower incomes. Compounding the issue of the problematic randomness of the sample is the self-selection among respondents, a known survey problem that skews results, likely toward those who feel strongly about something or have time to take the survey. I want to know what the “quiet ones” are thinking. Don’t you?

Surprisingly, the report is self-condemning in this regard, raising exactly the same issues. Then it plods on with seemingly confident language about the results we are supposed to base actions and policies on in the near term. “The results of this survey analysis are provided with a 95% confidence level (meaning that a reader should be 95% confident the answers are reflective of the broader population) with a confidence interval of 1.2% (meaning that the answers provided by the respondents reflect the broader population’s thoughts +/- 1.2%). This is an extremely high accuracy rating for a survey where many have a range of +/- 3-5%.” I don’t think so.

The flawed mechanism I referred to above was, surprisingly, attested to (somewhat) by City Council after the results were presented by the survey authors on 27 March during a session of the committee of the whole that you can view here at 1:12:43 on the video counter. Earlier the same day, I had a cordial email exchange with two of the authors of the survey from Western Washington University, Dr. Hart Hodges and Mr. James McCafferty, who both acknowledged my issues with survey. However, Dr. Hart offered a defense by saying to me:

“The comments about the confidence interval and power of the sample come from the size of the overall population, the size of the sample, and the response rate. And in this case you can say the statistics have a fairly high degree of precision or power.

Your question highlights the fact that precision does not equal accuracy.

A relatively high percentage of respondents are homeowners and have lived in Bellingham for a number of years. I suspect James [McCafferty] highlighted those findings to make sure the reader knows the results of the survey are not necessarily accurate (meaning they do not necessarily reflect the views of the population as a whole).

Accuracy increases with sample size. So we would have expected fairly accurate estimates with the given sample size. Unfortunately, the respondents, as a group, differ from the overall population.”

So we are proffered a survey that may accurately measure limited and even internally biased samples, e.g. homeowners or the elderly, but leaves out substantial portions of the city’s population. Also, if you are among the hundreds of homeless, you don’t get to play from the outset since you have no address. Nonetheless, the city’s website touts the survey as being representative of Bellingham residents.

I would like to see a city survey that is not dependent on self-selection and comes from a truly random sample; even the requirement for having a computer in order to respond introduces bias. A lot goes on with renters and the poor in this city, much of which was more than likely not captured by this survey and its antecedents. So presentation of trends over time is also suspect. Similarly, many proposals for responding to the perceived lack of housing availability, such as accessory dwelling units or the Infill Tool Kit, tend to affect the older neighborhoods to a greater degree since solutions tend not to be applicable in new neighborhoods where restrictive covenants on private lots essentially obviate adverse effects since new rules cannot be applied.

Since so much depends on the outcomes of this annual survey, it may be worthwhile to do it correctly.