Issues of non-response: greater «coverage» of the sample or consolidation of the comparability criteria in data collection?

PDF (ελληνικά)
Published: Jan 27, 2023
Updated: 2023-01-27
DOI: https://doi.org/10.12681/grsr.33164
Keywords:
non-response, comparability, refusals, call-backs, European Social Survey
Eva Tsouparopoulou
Panteion University
Abstract

The issue of non-response is of great concern to social scientists as response rates decline rapidly. The increase of the number of call-backs is commonly used as a practice to address this issue. This study investigates the effectiveness of call-backs for all 36 participating countries in the European Social Survey (ESS) and for the eight rounds (2002-2016). Also, the application (or not) of harmonized methodological rules for minimum response rates and the number of call-backs so as to ensure the overtime and cross-national comparability of data is examined. The results confirm the magnitude of the problem of non-response in social sample surveys and the inability of the participating countries to apply the strict methodological standards.

Article Details
  • Section
  • Articles
Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Authors who publish with this journal agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution Non-Commercial License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g. post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (preferably in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Downloads
Download data is not yet available.
References
Κοκολάκης, Γ. και Χρυσαφίνου, Ο. (1988). Λεξικό αγγλο-ελληνικό και ελληνο-αγγλικό στατιστικών όρων με βάση το «Dictionary of Statistical Terms» των M.G. Kendall and W.R. Buckland.
Μιχαλοπούλου, Κ. (2004). Στην αυτοκρατορία των ενδείξεων: Η ιστορία της δειγματοληπτικής πρακτικής στην Ελλάδα. Παπαζήσης.
Μιχαλοπούλου, Κ. (2012). Οι περιπέτειες της απόδοσης των όρων της δειγματοληπτικής θεωρίας και πρακτικής στις κοινωνικές διερευνήσεις και το ανύπαρκτο «στατιστικό σφάλμα»: Ένα σχολιασμένο αγγλοελληνικό γλωσσάριο, Επιθεώρηση Κοινωνικών Ερευνών, 137-138 (Α-B), σελ. 3-49. https://doi.org/10.12681/grsr.5
Τσουπαροπούλου, Ε. (2021). Η μη-απόκριση στις δειγματοληπτικές έρευνες μεγάλης-κλίμακας (Διδακτορική Διατριβή). Αθήνα: Πάντειον Πανεπιστήμιο Κοινωνικών και Πολιτικών Επιστημών. Ανάκτηση από http://pandemos.
panteion.gr/index.phpop=record&type=0&q=%CE%A4%CF%83%CE%B%CF%85%CF%80%CE%B1%CF%81%CE%BF%CF%80%CE%BF%CF%8D%CE%BB%CE%BF%CF%85&page=1&scope=0&lang=el&pid=iid:20808
Chtouris, S. (2019). H δυναμική αμφίδρομη σχέση εμπειρικών δεδομένων και θεωρητικών εννοιών. Ένα πλαίσιο για την καταγραφή και κατανόηση των κοινωνικών φαινομένων. Επιθεώρηση Κοινωνικών Ερευνών, 151, σελ. 137-170. https://doi.org/10.12681/grsr.19859
Bernhardt, R., Munro, D., and Wolcott, E. (2021). How does the dramatic rise of CPS non-response impact labor market indicators? GLO Discussion Paper, No. 781, pp. 1-42. Essen: Global Labor Organization (GLO). https://www.econstor.eu/bitstream/10419/229653/1/GLO-DP-0781.pdf
Bethlehem, J. G., Cobben, F., and Schouten, B. (2011). Handbook of nonresponse in household surveys. John Wiley.
Beullens, K., Billiet, J., and Loosveldt, G. (2010). The effect of the elapsed time between the initial refusal and conversion contact on conversion success: Evidence from the 2nd round of the European Social Survey. Quality & Quantity, 44 (6), pp. 1053-1065. https://doi.org/10.1007/s11135-009-9257-4
Beullens, K., Loosveldt, G., Vandenplas, C., and Stoop, I. (2018). Response rates in the European Social Survey: Increasing, decreasing, or a matter of fieldwork efforts? Survey methods: Insights from the field. https://surveyinsights.org/?p=9673
Beullens, K., Vandecasteele, L., and Billiet, J. (2007). Refusal conversion in the second round of the European Social Survey (Working paper). K.U.Leuven, Centrum voor Sociologisch Onderzoek (CeSO).
https://www.researchgate.net/publication/293781720_Refusal_Conversion_in_the_Second_Round_of_the_European_Social_Survey
Billiet, J., Matsuo, H., Beullens, K., and Vehovar, V. (2009). Non-response bias in cross-national surveys: Designs for detection and adjustment in the ESS. Ask: Research and Methods, 18(1), pp. 3-43. https://kb.osu.edu/handle/1811/69563
Billiet, J., Philippens, M., Fitzgerald, R., and Stoop, I. (2007). Estimation of nonresponse bias in the European Social Survey: Using information from reluctant respondents. Journal of Official Statistics, 23 (2), pp. 135-162.
Billiet, J., and Pleysier, S. (2007). Response based quality assessment in the ESS – Round 2: An Update for 26 Countries. K.U. Leuven: Center of Sociological Research (CeSO). https://www.europeansocialsurvey.org/docs/round2/methods/ESS2_response_based_quality_assessment.pdf
Blasius, J. (2006). Assessing data quality and construct comparability in cross-national surveys. European Sociological Review, 22 (3), pp. 229–242. https://doi.org/10.1093/esr/jci054
Brick, J. M., and Williams, D. (2013). Explaining rising nonresponse rates in cross-sectional surveys. The ANNALS of the American Academy of Political and Social Science,645 (1), pp. 36–59. https://doi.org/10.1177/0002716212456834
Cai, Y., and Baker, D. (2021). Masking real unemployment: The overall and racial impact of survey non-response on measured labor market outcomes. Institute for New Economic Thinking Working Paper Series, No. 150, pp. 1-22. https://doi.org/10.36687/inetwp150
Cochran, W. C. (1977). Sampling techniques (3rd edition). John Wiley.
Curtin, R., Presser, S., and Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64, pp. 413-428.
Deming, W. Ε. (1944). Οn errors in surveys. American Sociological Review, 9, pp. 359-369. https://doi.org/10.2307/2085979
Deming, W. Ε. (1950). Some theory of sampling. Dover Publication Inc.
Deming, E. (1953). On a probability mechanism to attain an economic balance between the error of response and the bias of nonresponse. Journal of the American Statistical Association, 48, pp. 743-772. https://doi.org/10.2307/2281069
Durbin, J., and Stuart, A. (1951). Differences in response rates of experienced and inexperienced interviewers. Journal of the Royal Statistical Society, Series A (General), 114 (2), pp. 163-206. https://doi.org/10.2307/2980975
Durbin, J., and Stuart, A. (1954). Callbacks and clustering in sample surveys: An experimental study. Journal of the Royal Statistical Society, Series A (General), 117 (4), pp. 387-428. https://doi.org/10.2307/2342679
Dutwin, D., Loft, J. D., Darling, J., Holbrook, A., Johnson, T., Langley, R. E.,Lavrakas, P. J., Olson, K., Peytcheva, E., Stec, J., Triplett, T., and Zukerberg, A. (2014). Current knowledge and consideration regarding survey refusals (Report). American Association for Public Opinion Research Task Force on Survey Refusals. https://www.aapor.org/AAPOR_Main/media/MainSiteFiles/RefusalTF_FINAL090814.pdf
Ellis, R. A., Endo, C. H., and Armer, J. M. (1970). The use of potential nonrespondents for studying nonresponse bias. Pacific Sociological Review, 13 (2), pp. 103-109. https://doi.org/10.2307/1388313
European Social Survey (2001). Round 1 Specification for Participating Countries. Centre for Comparative Social Surveys, City University London.
European Social Survey (2002a). ESS Round 1 Project Instructions (PAPI). Centre for Comparative Social Surveys, City University London.
European Social Survey (2002b). Field Procedures in the European Social Survey: Enhancing Response Rates. Centre for Comparative Social Surveys, City University London.
European Social Survey (2014). Round 7 Specification for Participating Countries. Centre for Comparative Social Surveys, City University London.
European Social Survey (2016a). ESS Round 8 Project Instructions (PAPI). ESS ERIC Headquarters, Centre for Comparative Social Surveys, City University London.
European Social Survey (2016b):ESS-1 2002 Documentation Report. Edition 6.5. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016c): ESS-2 2004 Documentation Report. Edition 3.6. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016d): ESS-3 2006 Documentation Report. Edition 3.6. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016e): ESS-4 2008 Documentation Report. Edition 5.4. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016f): ESS-5 2010 Documentation Report. Edition 4.1. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016g): ESS-6 2012 Documentation Report. Edition 2.3. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2016h): ESS-7 2014 Documentation Report. Edition 3.1. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey (2018): ESS-8 2016 Documentation Report. Edition 2.0. European Social Survey Data Archive, NSD - Norwegian Centre for Research Data for ESS ERIC.
European Social Survey Round 1 Data (2002a). Contact form data file edition 1.0. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 2 Data (2004a). Contact form data file edition 3.2. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 3 Data (2006a). Contact form data file edition 1.1. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 4 Data (2008a). Contact form data file edition 2.0. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 5 Data (2010a). Contact form data file edition 2.1. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 6 Data (2012a). Contact form data file edition 2.0. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 7 Data (2014a). Contact form data file edition 2.1. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Round 8 Data (2016a). Contact form data file edition 2.0. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
European Social Survey Cumulative File, ESS 1-8 (2018). Data file edition 1.0. NSD - Norwegian Centre for Research Data, Data Archive and distributor of ESS data for ESS ERIC.
Goyder, J., Warriner, K., and Miller, S. (2002). Evaluating socio-economic status (SES) bias in survey nonresponse. Journal of Official Statistics, 18 (1), pp. 1-11. Retrieved from https://www.scb.se/contentassets/ca21efb41fee47d2
bbee5bf7be7fb3/evaluating-socio-economic-status-ses-bias-in-survey-nonresponse.pdf
Groves, R. M. (2006). Non response rates and non response bias in household surveys. Public Opinion Quarterly, 70 (5), pp. 646-675. https://doi.org/10.1093/poq/nfl033
Groves, R., and Couper M. (1998). Nonresponse in household interview surveys. John Wiley.
Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Piani Acosta, G., and Nelson, L. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70 (5), pp. 720-736. https://doi.org/10.1093/poq/nfl036
Groves, R.M., and Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72 (2), pp. 167-189. https://doi.org/10.1093/poq/nfn011
Groves, R.M., Presser, S., and Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68 (1), pp. 2–31. https://doi.org/10.1093/poq/nfh002
Groves, R. M., Singer, E., and Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64 (3), pp. 299-308. https://doi.org/10.1086/317990
Hansen, M. H., and Hurwitz, W. N. (1946). The problem of non-response in sample surveys. Journal of the American Statistical Association, 41 (236), pp. 517–529. https://doi.org/10.1080/01621459.1946.10501894
Hedlin, D. (2020). Is there a 'safe area' where the nonresponse rate has only a modest effect on bias despite non‐ignorable nonresponse?. International Statistical Review, 88 (3), pp. 642-657.https://doi.org/10.1111/insr.12359
Heerwegh, D., Abts, K., and Loosveldt, G. (2007). Minimizing survey refusal and noncontact rates: Do our efforts pay off? Survey Research Methods, 1 (1), pp. 3-10. https://doi.org/10.18148/srm/2007.v1i1.46
Jones, J. (1996). The effects of non-response on statistical inference. Journal of Health and Social Policy, 8 (1), pp. 49-62. https://doi.org/10.1300/J045v08n01_05
Kalton, G. (1983). Introduction to survey sampling. Sage.
Keeter, S., Miller, C., Kohut, A., Groves, R.M., and Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, pp. 125–148. https://doi.org/10.1086/317759
Kish, L. (1965). Survey Sampling. John Wiley.
Kish, L. (1994). Multipopulation survey designs: Five types with seven shared aspects. International Statistical Review, 62 (2), pp. 167-186. https://doi.org/10.2307/1403507
Kreuter, F. (Ed.) (2013). Improving surveys with paradata: Analytic uses of process information. John Wiley.
Kviz, F. J. (1977). Toward a Standard Definition of Response Rate.The Public Opinion Quarterly, 41(2), pp. 265-267. https://doi.org/10.1086/268382
Lazarsfeld, P. (1941). Repeated interviews as a tool for studying changes in opinion and their causes. American Statistical Association Bulletin, 2 (1), pp. 3-7. http://www.jstor.org/stable/42731708
Likert, R. (1947). The sample interview survey: A fundamental research tool of the social sciences. In W. Dennis (Ed.), Current trends in psychology (pp. 196–225). University of Pittsburgh Press.
Loosveldt, G., Philippens, M., Stoop, I., and Billiet, J. (2004). Refusal conversion procedures in the European Social Survey. Paper for the International Workshop on Household Nonresponse in Leuven, Belgium, September 22–24.
Lowe, F. E., and McCormick, T. C. (1955). Some survey sampling biases. Public Opinion Quarterly, 19 (3), pp. 303-315. https://doi.org/10.1086/266574
Mauchly, J. W. (1940). Significance test for sphericity of a normal n-variate distribution. The Annals of Mathematical Statistics, 11 (2), pp. 204-209. https://doi.org/10.1214/aoms/1177731915
Michalopoulou, C. (2016). Statistical internationalism: From Quetelet's census uniformity to Kish's cross-national sample survey comparability. Statistical Journal of the IAOS, 32 (4), pp. 545-554. https://doi.org/10.3233/SJI-160960
Moser, C. A. (1949). The use of sampling in Great Britain. Journal of the American Statistical Association, 44 (246), pp. 231-259. https://doi.org/10.2307/2280597
Moser, C. A. (1958). Survey methods in social investigation (1st edition). William Heinemann (Heinemann Books on Sociology).
Moser, C. A. and Kalton, G. (1971). Survey methods in social investigation. HeinemannEducational Books.
Osier, G. (2016). Unit non-response in household wealth surveys: Experience from the Eurosystem's Household Finance and Consumption Survey.European Central Bank (ECB), Statistics Paper Series, No 15. https://doi.org/10.2866/84445
Sax, L. J., Gilmartin, S. K. and Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44(4), pp. 409-433. https://doi.org/10.1023/A:1024232915870
Schutz, A. (1954). Concept and theory formation in the social sciences. The Journal of Philosophy, 51 (9), pp. 257-273.
Steeh, C. J. (1981). Trends in nonresponse rates 1952-1979. Public Opinion Quarterly, 5(1), pp. 40-57. https://doi.org/10.1086/268633
Stewart, F. (1948). Some sampling problems in sociometric surveys. Sociometry, 11 (4), pp. 301-307. https://doi.org/10.2307/2785190
Stocké, V., and Langfeldt, B. (2004). Effects of survey experience on respondents’ attitudes towards surveys. Bulletin of Sociological Methodology/Bulletin de MéthodologieSociologique, 81 (1), pp. 5-32. https://doi.org/10.1177/075910630408100103
Stoop, I., Billiet, J., Koch, A., and Fitzgerald, R. (2010). Improving survey response: lessons learned from the European Social Survey. John Wiley. https://doi.org/10.1002/9780470688335
Stoop, I., Jowell, R., and Mohler, P. (2002). The European Social Survey: One survey in two dozen countries. Paper to be presented at the International Conference on Improving Surveys, Copenhagen. http://www.european
socialsurvey.org/docs/about/ESS_background_one_survey.pdf
The Sampling Expert Panel of the ESS (n.d). Sampling for the European Social Survey Round II: Principles and requirements. European Social Survey, GESIS.
The ESS Sampling Expert Panel (2016). Sampling guidelines: Principles and implementation for the European Social Survey. ESS ERIC Headquarters.
Tuli, F. (2011). The basis of distinction between qualitative and quantitative research in social science: Reflection on ontological, epistemological and methodological perspectives. Ethiopian Journal of Education and Sciences, 6 (1), pp. 97-108. https://doi.org/10.4314/ejesc.v6i1.65384
Viswesvaran, C., Barrick, M. R., and Ones, D. S. (1993). How definitive are conclusions based on survey data: Estimating robustness to nonresponse. Personnel Psychology, 46, pp. 511-567. https://doi.org/10.1111/j.1744-6570.1993.tb00884.x
Zillmann, D., Schmitz, A., Skopek, J., and Blossfeld, H. P. (2014). Survey topic and unit nonresponse. Quality & Quantity, 48 (4), pp. 2069-2088. https://doi.org/10.1007/s11135-013-9880-y