Report

Scientist Opinion Panel Survey Report on “Survey of Surveys”

Authors | Shaika Islam, Tipeng Chen, Timothy Johnson, Jinghuan Ma, Lesley Michalegko, Eric Welch

1. Introduction

This report represents findings from a national survey of US scientists which is designed to better understand the perspectives of the academic community about participation in public opinion surveys. In this survey, we first asked a random sample of university-based scientists about their personal experiences participating in surveys in the past. This was followed by various questions on whether they enjoy doing surveys or they think of them as burdensome. We also asked if they thought surveys were an invasion of privacy and whether surveys are a reliable source of information for the general public. In addition to investigating scientists’ attitudes about surveys as a research method, we have also asked about their future expectations from SciOPS and their recommendations regarding future survey topics for SciOPS. Proposed topics included those representing broader public interest in science, the science process and science and technology policy topics. 

Based on our review, this is the first national survey of academic scientists in the United States that attempts to understand scientists’ experiences and opinions about surveys. Although Kılınç and Firat (2017) have also explored academicians’ opinions on collecting data through online surveys, unlike our study they took more of a qualitative approach using semi-structured interview technique to understand the opinions of a small purposeful sample of scientists. 

This national survey on academic scientists in the US was conducted by the Center for Science, Technology and Environmental Policy Studies (CSTEPS) at Arizona State University. Our survey was approved by Institutional Review Boards at Arizona State University. The survey was field between 17th October, 2022 and 19th December, 2022. We randomly selected 400 academic scientists from our SciOPS panel, which is recruited from a random sample of PhD-level faculty in R1 research-intensive institutions covering departments from biology, civil and environmental engineering, geography and public health. A total of 135 usable responses were obtained, representing an AAPOR individual survey response rate (RR4) of 33.5% and an AAPOR Cumulative Response Rate (CUMRR) of 8%. The appendix documents the full survey methodology and administration details. We have also provided detailed respondent demographics, analysis of sample composition, the complete survey instrument as appendices. 

Survey findings are presented in four sections: (1) Scientists’ personal experience with surveys; (2) Scientists’ opinions about surveys; (3) Scientists’ attitude toward participation in future and (4) Scientists’ input on survey topics. We conclude with an overall summary of this report.

---

2. Results

This section presents findings from the survey. 

2.1. Scientists’ personal experience with surveys In this section of the survey, we asked academic scientists questions about their personal experiences with surveys during the past year. 

2.1.1. Number of surveys completed in the past year 

We asked the respondents how many other surveys they have completed during the past year- in person, by telephone, by mail, or online (N=129). 

Figure 1

Number of surveys (in person, telephone, mail or online) completed by scientists in the past year (Exact question wording: “Aside from this survey, about how many other surveys have you completed during the past year—in person, by telephone, by mail, or online?”)

• 18% of respondents did not complete any survey in the past year. 
• As Figure 1 shows, 37% of the scientists completed 1 to 4 surveys in the past year either by in person, by telephone, by mail, or online. 32% of scientists completed 5 to 14 surveys. Fewer scientists (13%) completed surveys more than 15 surveys in the past year.

---

Figure 2

Number of surveys completed by scientists in the past year (differences in gender) (Exact question wording: “Aside from this survey, about how many other surveys have you completed during the past year—in person, by telephone, by mail, or online?”)

• We found a significant gender difference in responses at the 95% confidence level (p-value = 0.01). As seen in Figure 2, male scientists reported that they have completed surveys more than their female counterparts at each level except for the 15-30 category. It is noticeable that 11% of the male scientists reported that they have completed at least 30 surveys during the past year, while none of the female scientists reported doing so. 

---

2.1.2. Number of times survey declined in the past year 

We asked academic scientists the number of times they have declined to participate in a survey in the past year (N=129). 

Figure 3

Number of times surveys declined (Exact question wording: “Also during the past year, about how many times have you declined to participate in a survey?”)

• 20% of scientists reported that they have not declined any surveys in the past year. As Figure 3 shows, half of the scientists (50%) declined to participate in surveys at most 9 times. 25% of the scientists declined 10 to 30 surveys, while only 5% of scientists declined 31 to 50 surveys in the past year. One respondent answered 1000 for this question, an outlier which is excluded from the graph above. 

---

2.2. Scientists’ opinion about surveys 

We then asked scientists questions about their opinions about five specific survey methods-related characteristics following De Leeuw et al. (2019): (i) survey enjoyment, (ii) survey value, (iii) survey burden, (iv) survey reliability and (v) survey privacy. 

2.2.1. Survey enjoyment We first started by asking about enjoyment doing surveys (N=133). 

Figure 4

Scientists’ opinion on whether they enjoy doing surveys (Exact question wording: “Please indicate your agreement with the following statements about surveys”)

• Over half of the respondents (57%) agree that surveys are interesting compared to 18% of scientists who disagree that surveys are interesting. 
• 36% of the respondents agree that they enjoy responding to surveys through the mail or the internet as compared to 32% of respondents who do not enjoy them. More respondents (38%) disagree that they enjoy being interviewed for a survey as compared to those who do enjoy them (25%).

---

2.2.2. Survey value 

We then asked scientists questions about the value of surveys as a research method (N=133). 

Figure 5

Scientists’ opinion on the value of survey (Exact question wording: “Please indicate your agreement with the following statements about surveys”)

• Most respondents agree that surveys are important for society (86%), for science (83%) and that a lot can be learned from information collected through surveys (86%). A majority of scientists disagree that completing surveys is a waste of time (74%).

---

2.2.3. Survey burden 

Next, we asked scientists questions about the burden of participating in surveys (N=132). 

Figure 6

Scientists’ opinion on the burden of participating in surveys (Exact question wording: “Please indicate your agreement with the following statements about surveys”)

• Over half of the respondents (54%) agree that they receive far too many requests to participate in surveys. Meanwhile, the majority of respondents (73%) disagree that opinion polls are an invasion of privacy. Over half of the respondents (54%) reported that they agree that answering many questions in surveys is exhausting.

---

Figure 7

Scientists’ opinion on the burden of participating in surveys (differences by rank) (Exact question wording: “Please indicate your agreement with the following statements about surveys: I receive far too many requests to participate in surveys”)

• We found statistically significant differences by rank (p-value = 0.009). As seen in Figure 7, 65% of the full professors and 57% of the assistant professors agree that they receive far too many requests to participate in surveys while only 40% of associate professors and 32% of non-tenure-track professors think so.

---

2.2.4. Survey reliability 

Fourth, we asked scientists questions about whether the survey method is reliable (N=132). 

Figure 8

Scientists’ opinion on the reliability of surveys (Exact question wording: “Please indicate your agreement with the following statements about surveys”)

• As Figure 8 shows, the majority of scientists agree with the statement “if surveys are well done, it will give very accurate information about the views of the people surveyed” (85%). 
• 62% of respondents agree with the statement “most institutions that conduct public opinion surveys work hard to make their surveys as accurate as possible”. 68% of respondents agree with the statement “participants in surveys do their best to provide truthful answers to the questions they are asked”.

---

2.2.5. Survey privacy 

Finally, we asked scientists questions about privacy concerns regarding surveys (N=132). 

Figure 9

Scientists’ opinion on privacy concerns regarding surveys (Exact question wording: “Please indicate your agreement with the following statements about surveys”)

• As seen in Figure 9, most respondents (71%) disagree that surveys tend to ask questions that are too personal”, nor do they (69%) agree that surveys ask questions that are too controversial or sensitive. 
• Less than 5% of the scientists agree that surveys tend to ask about subjects that are too controversial or sensitive (3%) or questions that are too personal (2%).

---

2.3. Scientists’ attitude toward participation 

in future We asked scientists about their future expectations of SciOPS as a panel member and a contributor to surveys (N=132). 

Figure 10

Scientists’ future expectations for SciOPS (Exact question wording: “As a member and contributor to SciOPS, how important are the following to you”)

• As Figure 10 shows, more than half of the respondents (70%) reported that it is not important to receive public acknowledgment as a survey panel member on the SciOPS website. Similarly, 66% of scientists reported that it is not important to receive annual recognition, in the form of a formal letter or certificate, verifying their participation in SciOPS. 
• A majority of the respondents (91%) reported that it is important to have the opportunity to contribute their opinions regarding science-related topics on SciOPS surveys and that it is important to receive notification of findings from the surveys they participate in (92%). A large group of respondents (68%) also indicated that it is important to receive notification of findings from all surveys, whether they are invited to participate or not. 

---

Figure 11

Importance of annual recognition (differences by rank) (Exact question wording: “As a member and contributor to SciOPS, how important are the following to you: Receive annual recognition, in the form of a formal letter or certificate, verifying your participation in SciOPS.”)

• We further looked at the breakdown by scientists’ ranking and found a statistically significant difference in opinions by rank regarding the importance of receiving annual recognition of their participation in SciOPS (p-value = 0.003). In general, the higher the tenure track rank, the lower the importance of recognition.
• A minority of full professors (20%) and associate professors (35%) responded it is to some extent important to receive annual recognition, while over half of the assistant professors (57%) and non-tenure track professors (53%) did so.

---

2.4. Scientists’ input on survey topics 

We asked scientists about their preferences regarding the subjects of science-related topics for future SciOPS surveys. First, we asked scientists which one of the following science-related topics is more important for SciOPS to focus on in future surveys (N=131). 

Figure 12

Importance of general science-related survey topics (Exact question wording: “In your opinion, which one of the following science-related topics is more important for SciOPS to focus on in future surveys?”)

• As seen in Figure 12, according to more than half of the respondents (53%) reported that broader interest topics (e.g., public health, science communication, trust in science, mis- /dis-information, climate change, food security, etc.) are more important for future SciOPS surveys. 
• A lower percentage of scientists reported that science process (e.g., data sharing, collaboration, innovation and technology transfer, science funding, workplace, work-life balance, research ethics, etc.) (33%) or science and technology policy (e.g., science competitiveness, genome editing, open science, etc.) (13%) topics are more important.

---

Figure 13

• We found a significant difference in responses amongst these four different fields of scientists (p-value = 0.002). 
• More than half of the biologists (55%), geographers (57%) and public health scientists (69%) reported that broader public interest is a more important topic for SciOPS to focus on for future surveys. 
• By contrast, only 15% of the engineers reported that the broader public interest is a more important topic, while more than half of the engineers (55%) believe that the science process is more important for future SciOPS surveys. 

---

We then asked scientists opinions on which science-related topics they think the public is most interested in (N=132). 

Figure 14

Scientists’ opinion regarding the public interest (Exact question wording: “Which one of the following science-related topics do you think the public is most interested in?”)

• A large majority (90%) of scientists reported that the public is most interested in broader science-related topics (e.g., public health, science communication, trust in science, mis- /dis-information, climate change, food security, etc.). 
• Only a small percentage (5%) of scientists reported that the public is interested in science and technology policy or science process (4%).

Last, we asked scientists whether the previous science-related topics, such as science process, science and technology policy and broader public interest, are relevant to professional advancement in their fields (N=132). 

---

Figure 15

The relevance of science-related topics (such as science process, science and technology policy and broader public interest) with scientists’ professional advancement (Exact question wording: “Are the topics listed in the previous question relevant to professional advancement in your field?”)

• As Figure 15 shows, 84% of the scientists reported that science-related topics, such as science process, science and technology policy and broader public interest, are relevant to professional advancement in their field.

---

3. Conclusion

This survey investigates the importance that scientists give to surveys as a research method. Further, it gives us a better understanding on academic scientists’ attitudes on surveys. First, with regard to scientists’ experience with past surveys, most scientists reported that they participated in surveys at most 9 times in the previous year. More than half of the male scientists participated in surveys 1 to 9 times in the past year, compared to female scientists (47%). Further, 11% of the male scientists reported that they have completed at least 30 surveys during the past year, while none of the female scientists reported doing so. This means that male scientists, in general, are more likely to participate in a higher number of surveys compared to female scientists. Second, scientists think highly of surveys as a research method. The majority of scientists think that surveys can be valuable for society and science as most of them believe that surveys can be a medium by which considerable information can be gained. Most scientists believe that the survey method is a reliable source and does not invade the personal privacy of respondents. On the other hand, over half of the respondents think that they face a high burden from surveys, for example, receiving too many survey requests and answering too many questions.

Third, over half of the scientists would like to participate in surveys but do not care about receiving any sort of additional recognition for their participation. They reported that it is important to contribute their opinions for SciOPS’ surveys and to receive notification of findings from surveys but that it is not important to receive recognition for survey participation. We also find that the scientists with a higher academic ranking are less likely to care about recognition of participation compared to junior scientists.

Lastly, most scientists suggest SciOPS should focus on topics related to the broader public interest (e.g., public health, science communication, trust in science, mis-/dis-information, climate change, food security, etc.) in future surveys. However, unlike biologists, geographers and public health scientists, who think that SciOPS should focus more on science related topics on broader public interest, engineers believe that the science process is a more important topic for SciOPS to focus on. This may be because science processes as a topic are more relevant to engineers in their research field which they can connect to for advancement in their own profession. 

We acknowledge several limitations of our survey. First, the result of this survey has limitations on its generalizability as our respondents are restricted to biologists, engineers, geographers, and public health researchers in R1 universities. Their views may not represent the perspectives of scientists in other academic fields. Second, although, this survey obtained a relatively high AAPOR response rate, it is limited as it has a relatively small number of completed responses. Third, the respondents’ attitudes about surveys may not be representative of the population of academic scientists in the US due to panel conditioning on the survey results and selection bias in SciOPS panel recruitment. For one, their experience participating in SciOPS panel surveys may influence their attitudes toward our subsequent surveys. For another, the respondents for this national survey opt-in to the SciOPS panel voluntarily. Their consent to become survey panel members indicates that they may have a more positive attitude about the survey and about SciOPS than the general population of scientists.

---

APPENDIX A. 

A1. Survey methodology and survey administration 

The sample for this survey aims to represent the population of PhD-level faculty in four fields of science, i.e., biology and genetics, civil and environmental engineering, geography, and public health. 

At first, a full sample frame of 11967 scientists was invited to become SciOPS panel members from those four fields. The SciOPS research team used the probability sampling method to randomly select Carnegie-designated Research Extensive and Intensive (R1) universities covering departments of the four fields in the United States (US). Table A.1 shows the number of institutions that we randomly selected. Contact information of the faculties in these institutions was collected. 986 eligible scientists out of 11967 scientists consented to become SciOPS panel members. The AAPOR recruitment rate (RECR) was 8.1%. 

Second, 400 scientists out of the 986 eligible scientists were randomly selected as the sample for this survey. Individuals were recruited to participate via personal email invitations. Electronic email alert messages were sent on October 14th, 2022, 3 days prior to the survey launch to notify sampled individuals that they would be receiving the questionnaire shortly. Email invitations (including unique ID, passwords, and hyperlink to the survey) were sent on October 17th, 2022, followed by two reminder messages on October 26th and November 16th. The survey was closed on December 19th, 2022. The survey instrument was electronically programmed (in English) using the NubiS® system. NubiS® is an accessible and versatile software system specifically designed to administer questionnaires with protection for the confidentiality of the survey respondents.

A total of 135 usable responses were obtained, representing an AAPOR individual survey response rate (RR4) of 33.5% and an AAPOR Cumulative Response Rate (CUMRR) of 8%. Table A.2 represents the overall summary of survey responses.

A2. Sample weighting and precision 

The sample of respondents for this survey was weighted by the inverse of selection probabilities and post-stratified probabilities by gender and academic field to represent the full sample frame for recruiting SciOPS panel members as closely as possible. A conservative measure of sampling error for questions answered by the sample of respondents is plus or minus 5 percentage points. 

---

APPENDIX B.

 B1. Respondents’ demographics 

Table B.1 shows a breakdown of survey respondents by ranking of academic scientists. 45.9% of the respondents were assistant professors followed by associate professors (20.7%) and full professors (17.8%).

Survey respondents consist of academic scientists from five departments. Shown in Table B.2, the largest number, nearly 50% worked in the biology department and about 30% worked in a public health department.

The survey respondents were predominantly white (83%) and male (52%). All respondents were recruited from academic departments. Scientists employed in government or industry are not represented in SciOPS.

B2. Analysis of sample composition We made comparisons between the sample composition of the final sample for this survey with the (1) full sample frame for recruiting SciOPS panel members and (2) the final frame of SciOPS panel members. Sample t-tests were calculated to see whether there were statistically significant differences across each demographic measure across sample groups. The results are as follows:

• Females are overrepresented in the final sample of respondents for this survey and in the group of SciOPS panel members, relative to the composition of the full sample frame for recruiting SciOPS panel members. 
• Whites are slightly overrepresented in the SciOPS panel (p-value =0.06) and final respondent samples (p-value =0.08), relative to the full sample frame, although differences were not significant at the 0.05 level. 
• Differences by academic rank (i.e., assistant professors, associate professors and full Professors) were not observed across samples. 
• We observed some differences by field of specialty. Engineers are under-represented in the final sample of respondents. The group of SciOPS panelists and the group of survey respondents each overrepresented geographers relative to their representation in the original sample frame. There is a borderline significant difference (p-value = 0.07) with public health faculty being slightly under-represented in the panelist sample. The representation of biologists, however, does not vary across the full sample frame, the sample of panel members, and the final survey sample.

Sample weighting, described earlier at Appendix A, was employed to address these discrepancies between the final survey sample and the full sample frame from which respondents were recruited.

---

APPENDIX C. 

Survey Instrument 

Part 1: Topics for future surveys

1. In your opinion, which one of the following science-related topics is more important for SciOPS to focus on in future surveys? 

a. Science process (e.g., data sharing, collaboration, innovation and technology transfer, science funding, workplace, work-life balance, research ethics, etc.) 

b. Science and technology policy topics (e.g., science competitiveness, genome editing, open science, etc.) 

c. Broader public interest (e.g., public health, science communication, trust in science, mis-/dis-information, climate change, food security, etc.) 

d. Other:_____ [Open option]

2. Which one of the following science-related topics do you think the public is most interested in? 

a. Science process (e.g., data sharing, collaboration, innovation and technology transfer, science funding, workplace, work-life balance, research ethics, etc.) 

b. Science and technology policy topics (e.g., science competitiveness, genome editing, open science, etc.) 

c. Broader public interest (e.g., public health, science communication, trust in science, mis-/dis-information, climate change, food security, etc.) 

d. Other:_____ [Open option] 

3. Are the topics listed in the previous question relevant to professional advancement in your field? 

a. Yes 

b. No

Part 2: Personal experience with surveys in the last year 

1. Aside from this survey, about how many other surveys have you completed during the past year—in person, by telephone, by mail, or online? 

2. Also during the past year, about how many times have you declined to participate in a survey?

Part 3: Scientists’ attitudes about surveys compared to the general population 

1. Please indicate your agreement with the following statements about surveys:

[Strongly disagree, Disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Agree, Strongly agree]

o Survey enjoyment 

▪ E1: I really enjoy responding to questionnaires through the mail or Internet 

▪ E2: I really enjoy being interviewed for a survey 

▪ E3: Surveys are interesting 

o Survey value 

▪ V1: Surveys are important for society 

▪ V2: A lot can be learned from information collected through surveys 

▪ V3: Completing surveys is a waste of time 

▪ V4: Surveys are important for science 

o Survey burden 

▪ B1: I receive far too many requests to participate in surveys 

▪ B2: Opinion polls are an invasion of privacy 

▪ B3: It is exhausting to answer so many questions in a survey 

o Survey reliability 

▪ R1 Most institutions that conduct public opinion surveys work hard to make their surveys as accurate as possible. 

▪ R2 Participants in surveys do their best to give truthful answers to the questions they are asked. 

▪ R3 If a survey is well done, it will give very accurate information about the views of the people surveyed. 

o Survey Privacy 

▪ P1 Surveys tend to ask questions that are too personal 

▪ P2 Surveys tend to ask about subjects that are too controversial or sensitive 

2. As a member and contributor to SciOPS, how important are the following to you: [Not at all important, Not very important, Somewhat important, Very important, Extremely important] 

A. Receive notification of findings from the surveys you participate in. 

B. Receive notification of findings from all surveys (whether you were invited to participate or not). 

C. Receive annual recognition, in the form of a formal letter or certificate, verifying your participation in SciOPS. 

D. Receive public acknowledgement that you are a survey panel member on the SciOPS website. 

E. Having the opportunity to contribute your opinions regarding science-related topics on SciOPS surveys. 

F. Having the opportunity to provide testimonials or quotations about the value of science communication. 

G. Being involved in the development of survey topics and ideas.