“Breach of Research Ethics” in East Lansing Survey on Policing Perspectives
A recent study of East Lansing residents’ attitudes towards local policing claimed to include 51 African Americans and/or Black people among the 500 people surveyed by phone. But now, ELi has confirmed that only 34 self-identified in the sample as Black/African American, a fact that is unclear in the report provided by the company hired to do the survey, EPIC-MRA.
ELi was made aware of the discrepancy through the analysis of survey materials by Dr. Cedrick Heraux, a criminal justice researcher in sociology and appointee to a special East Lansing task force studying police oversight.
Back in April, the Study Committee on an Independent Police Oversight Commission, the task force on which Heraux is serving, expressed frustrations with the City’s lack of transparency concerning the raw data obtained by the survey conducted by external consultant EPIC-MRA.
City staff initially told ELi that the City had not received the raw data from EPIC-MRA, but a Freedom of Information Act (FOIA) request filed by ELi later revealed the City had received the data before ELi inquired.
The survey, conducted earlier this year at the cost of $19,750, was designed to poll 500 East Lansing residents and to “overrepresent” Black East Lansing citizens in order to glean an understanding about attitudes toward local policing with a specific eye toward different experiences based on the race of the respondents.
In its official presentation on the survey results, EPIC-MRA told East Lansing’s City Council, “the 500 base sample included an oversample of 51 interviews with African American residents.”
But Heraux, who is an Associate Professor of Sociology and Criminal Justice at Adrian College and who specifically researches policing, reviewed the data later made public. In doing so, he found that EPIC-MRA only conducted interviews with 34 respondents that self-identified as Black or African American.
EPIC-MRA counted as Black some respondents who did not provide their race, by using other information associated with the respondents’ phone numbers. The survey company also weighted the responses of Black respondents to “statistically” reach what they believe is the equivalent of 51 African American and/or Black respondents in the sample of 500.
Heraux told ELi by email, “The way that EPIC-MRA described their survey obfuscates this fact and leads the reader to believe that they spoke to 51 African-American/Black respondents, when this is not the case. While this [weighting] technique is acceptable (assuming that they weighted appropriately, which I was not able to verify), it is a breach of research ethics to not inform the reader that the technique was used.”
Changing the weight of responses in this way can increase the margin of error, especially in a small sample size. Heraux believes this is something EPIC-MRA should have made clear in their report to the City.
ELi reached out to EPIC-MRA to seek more clarification on the concerns raised by Heraux.
Regarding the margin of error and weighting, Kelly Sullivan, Director of Operations for EPIC-MRA, told ELi, that, “As occasional reweighting is a fairly common practice in survey research, and in this case resulted in no substantial differences in the data ultimately reported, we did not feel it necessary to draw attention to it.”
Sullivan further responded to ELi, saying that, “our landline and cell samples allow us – in some instances – to append the correct age and race to residents that may have refused to answer those questions.”
Essentially, the company’s researchers use other data sources, which they have not made public, to determine a person’s likely racial identity.
While Hearuax says this is not necessarily abnormal procedure for the practice of collecting data, he finds it “odd” that some of these methodological details weren’t included in the report to the City.
When asked by ELi why EPIC-MRA didn’t keep working to reach at least 51 Black/African American respondents, Sullivan wrote that “getting those interviews was quite a struggle.”
ELi also asked EPIC-MRA, “Why wasn’t information on how the respondents were contacted, the response rate, the refusal rate, or how the oversample was obtained included in the public analysis?”
Sullivan answered, “this was not requested by the City, nor is it something we typically report on unless asked in advance.”
In response to questions about the lack of comparisons between Black/African American respondents and other ethnicities/races Sullivan wrote, “Demographic segment sample sizes of ‘other races’ would be too small to offer meaningful analysis.”
EPIC-MRA compared “all” respondents to “African-American/Black” respondents, which Heraux terms “an astounding fundamental error.” That’s because the “all” would include the subset of African-American/Black, reducing the real margins between, for example, white respondents’ perceptions versus African-American/Black respondents’ perceptions.
Heraux wrote, “While they [EPIC-MRA] note that there is a ‘small N’ [number of respondents] problem for other racial groups, this does not explain why there were no simple ‘White’ vs. ‘African-American/Black’ comparisons.”
According to Heraux, he received “almost verbatim” responses from EPIC-MRA when he asked questions similar to ELi’s.
He also raised other potential statistical problems with EPIC-MRA and was not satisfied with the responses, or lack thereof.
“Notably, this lack of detail makes the raw data posted on the City website virtually useless to anyone who wanted to attempt to replicate these analyses,” Heraux wrote to ELi.
City Manager George Lahanas had previously stated that the plan was to use this survey from EPIC-MRA as a baseline for annual surveys, to see if various changes in the East Lansing Police Department’s approaches were changing perceptions of policing.
According to EPIC-MRA, the survey can be adapted in the future and still used to track changes in local perceptions of policing over time, but Sullivan did point out that “‘best’ way to track time-series data would be to keep questions identical.”