The Bellingham Residential Survey Makes My Head Hurt

84% of the respondents to the 2017 Bellingham Residential Survey are homeowners. What?

84% of the respondents to the 2017 Bellingham Residential Survey are homeowners. What?

This “Residential Survey” of Bellingham residents is a quasi-yearly affair with this stated purpose: “... the Survey is to seek feedback from residents [bolding mine] about their experience as citizens of the city of Bellingham. A combination of 50 multiple choice questions, 3 rank order questions, and 6 open-ended questions aim to gather information from respondents about their demographic details, quality of life, satisfaction with City services, opinions on budget priorities, views on challenges facing the community, and ideas for the future.” Tim Johnson at Cascadia Weekly did a good job reporting on the actual survey results, such as they are, which you can read in his Gristle column of 29 March. However, the underlying results are based on a flawed mechanism that mainly surveys homeowners.

The annual survey of “resident” attitudes is essentially an exercise based on an unverifiable “random” sample from a list of 7,000 Bellingham addresses prepared by the city and then provided to the survey creation team at WWU. Residents were then notified by postcard to take the on-line survey. Eighty-four percent of the respondents on this year’s survey were identified as homeowners. Only 14% were renters although they comprise over 50% of the city’s population. Over half the respondents were over 55 years old and in the age group 18-30 there was only a 6% representation. Keep in mind, renters and youth also tend to be those with lower incomes. Compounding the issue of the problematic randomness of the sample is the self-selection among respondents, a known survey problem that skews results, likely toward those who feel strongly about something or have time to take the survey. I want to know what the “quiet ones” are thinking. Don’t you?

Surprisingly, the report is self-condemning in this regard, raising exactly the same issues. Then it plods on with seemingly confident language about the results we are supposed to base actions and policies on in the near term. “The results of this survey analysis are provided with a 95% confidence level (meaning that a reader should be 95% confident the answers are reflective of the broader population) with a confidence interval of 1.2% (meaning that the answers provided by the respondents reflect the broader population’s thoughts +/- 1.2%). This is an extremely high accuracy rating for a survey where many have a range of +/- 3-5%.” I don’t think so.

The flawed mechanism I referred to above was, surprisingly, attested to (somewhat) by City Council after the results were presented by the survey authors on 27 March during a session of the committee of the whole that you can view here at 1:12:43 on the video counter. Earlier the same day, I had a cordial email exchange with two of the authors of the survey from Western Washington University, Dr. Hart Hodges and Mr. James McCafferty, who both acknowledged my issues with survey. However, Dr. Hart offered a defense by saying to me:

“The comments about the confidence interval and power of the sample come from the size of the overall population, the size of the sample, and the response rate. And in this case you can say the statistics have a fairly high degree of precision or power.

Your question highlights the fact that precision does not equal accuracy.

A relatively high percentage of respondents are homeowners and have lived in Bellingham for a number of years. I suspect James [McCafferty] highlighted those findings to make sure the reader knows the results of the survey are not necessarily accurate (meaning they do not necessarily reflect the views of the population as a whole).

Accuracy increases with sample size. So we would have expected fairly accurate estimates with the given sample size. Unfortunately, the respondents, as a group, differ from the overall population.

So we are proffered a survey that may accurately measure limited and even internally biased samples, e.g. homeowners or the elderly, but leaves out substantial portions of the city’s population. Also, if you are among the hundreds of homeless, you don’t get to play from the outset since you have no address. Nonetheless, the city’s website touts the survey as being representative of Bellingham residents.

I would like to see a city survey that is not dependent on self-selection and comes from a truly random sample; even the requirement for having a computer in order to respond introduces bias. A lot goes on with renters and the poor in this city, much of which was more than likely not captured by this survey and its antecedents. So presentation of trends over time is also suspect. Similarly, many proposals for responding to the perceived lack of housing availability, such as accessory dwelling units or the Infill Tool Kit, tend to affect the older neighborhoods to a greater degree since solutions tend not to be applicable in new neighborhoods where restrictive covenants on private lots essentially obviate adverse effects since new rules cannot be applied.

Since so much depends on the outcomes of this annual survey, it may be worthwhile to do it correctly.

About Dick Conoboy

Citizen Journalist and Editor • Member since Jan 26, 2008

Dick Conoboy is a recovering civilian federal worker and military officer who was offered and accepted an all-expense paid, one year trip to Vietnam in 1968. He is a former Army [...]

Comments by Readers

Dick Conoboy

Apr 14, 2017

I received the following message from a member of a local firm specializing in surveys,  “I saw your posting in NW Citizen in regards to the City’s Residential Survey. This is an area of specialization for my firm and I agree with many of your comments. Regarding the margin of error they cite. I can’t even figure out where that number would come from: standard number for that sample size would be + or - 2.8%. Their response rate is actually quite low and as you point out not representative of the resident population.”

While it is nice to have validation of one’s concerns, I am a bit miffed that the citizens of Bellingham have been given short shrift on these surveys seemingly because not enough money is  allocated to do it correctly. 


Michael Lilliquist

Mar 20, 2019

This is in reply to the 2018 survey. I posted it here on accident.  I wil re-post to the more recent blog entry.

Dick shared his blog post directly with the members of the city council, by email.  I replied to him by email, and I share it with you all her as well:

“Thanks for your constructive feedback.  As you know, I agree with much of what you say about the methodological shortcomings of the survey.  However, the survey was conducted with “scientific rigor,” and that is why it is possible to adjust for the un-representativeness of the sample. As was indicated during the presentation to the city council, this can be done in a number of ways.  James McCafferty offered to perform some “cross tabulations” to look into whether or not the skewed sample affected the main results.  I have taken him up on that offer, and I am awaiting the results.

“There are other problems with the survey, as well. Most glaringly, we do not have any yardstick or basis for comparison for most of the results. Again, James McCafferty has offered to re-dress those shortcomings, in a proposal to the administration. That would require extra work and funding, however, and I have not yet had a chance to discuss the matter with the mayor.”
To comment, Log In or Register