Have a personal or library account? Click to login

A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey

Journal of Official Statistics
Volume 36 (2020): Issue 3 (September 2020)
By:
Open Access
|Jul 2020

References

  1. AAPOR, American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition.
    Search in Google ScholarBack to article
  2. Bethlehem, J.G. 2002. “Weighting Non-response Adjustment Based on Auxiliary Information.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.
    Search in Google ScholarBack to article
  3. Beullens, K. and G. Loosveldt. 2012. “Should High response Rates Really be the Primary Objective?” Survey Practice 5(3): 1–5. DOI: https://doi.org/10.29115/SP-2012-0019.10.29115/SP-2012-0019
    Search in Google ScholarBack to article
  4. Beullens, K., G. Loosveldt, C. Vandenplas, and I. Stoop. 2018. “Response Rates in the European Social Survey: Increasing, Decreasing, or a Matter of Fieldwork Efforts?” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2018-00003.
    Search in Google ScholarBack to article
  5. Brick, J.M. and M.E. Jones. 2008. “Propensity to respond and nonresponse bias.” METRON – International Journal of Statistics, LXVI(1), 51–73.
    Search in Google ScholarBack to article
  6. Brick, J.M. and D. Williams. 2013. “Explaining rising Non-response Rates in Cross-sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. DOI: https://doi.org/10.1177/0002716212456834.10.1177/0002716212456834
    Search in Google ScholarBack to article
  7. Brick, J.M. and R. Tourangeau. 2017. “Responsive Survey Designs for Reducing Non-response Bias.” Journal of Official Statistics 33(3): 735–752. DOI: http://dx.doi.org/10.1515/JOS-2017-0034.10.1515/jos-2017-0034
    Search in Google ScholarBack to article
  8. Cobben, F. 2009. Non-response in Sample Surveys. Methods for Analysis and Adjustment. PhD thesis, The Hague: Statistics Netherlands. Available at: https://hdl.handle.-net/11245/1.312964 (accessed June 2020).
    Search in Google ScholarBack to article
  9. Cornesse, C. and M. Bosnjak. 2018. “Is there an association between survey characteristics and representativeness? A meta-analysis.” Survey Research Methods 12(1): 1–13. DOI: https://doi.org/10.18148/srm/2018.v12i1.7205.
    Search in Google ScholarBack to article
  10. De Heij, V., B. Schouten, and N. Shlomo. 2015. RISQ manual 2.1. Tools in SAS and R for the computation of R-indicators, partial R-indicators and partial coefficients of variation. RISQ Project. Available at: www.risq-project.eu.
    Search in Google ScholarBack to article
  11. De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Non-response: A Longitudinal and International Comparison.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.
    Search in Google ScholarBack to article
  12. Ernst Stähli, M., D. Joye, M. Sapin, A. Pollien, M. Ochsner, and A. van den Hende. 2018. “Non Response Surveys (NRS): ESS 2006, EVS 2008, ESS 2010, MOSAiCH 2011, ESS 2012, ESS 2014.” [Dataset]. Distributed by FORS, Lausanne. https://doi.org/10.23662/FORS-DS-697-1.
    Search in Google ScholarBack to article
  13. ESS Round 5: European Social Survey 2016: ESS-5 2010 Documentation Report. Edition 4.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/docs/round5/survey/ESS5_data_documentation_report_e04_2.pdf (accessed June 2020).
    Search in Google ScholarBack to article
  14. ESS Round 5: European Social Survey Round 5 Data. 2010. Data file edition 3.4. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/download.html?file=ESS5CH&c=CH&y=2010.
    Search in Google ScholarBack to article
  15. Groves, R.M. 2006. “Non-response Rates and Non-response Bias in Household surveys.” Public Opinion Quarterly 70, Special Issue: 646–675. DOI: https://doi.org/10.1093/poq/nfl033.10.1093/poq/nfl033
    Search in Google ScholarBack to article
  16. Groves, R.M. and S. Heeringa. 2006. “Responsive design for household surveys: tools for actively controlling survey errors and costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3): 439–457. DOI: https://doi.org/10.1111/j.1467-985X.2006.00423.x10.1111/j.1467-985X.2006.00423.x
    Search in Google ScholarBack to article
  17. Groves, R.M. and E. Peytcheva. 2008. “The Impact of Non-response Rates on Non-response Bias – A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. DOI: https://doi.org/10.1093/poq/nfn011.10.1093/poq/nfn011
    Search in Google ScholarBack to article
  18. Groves, R.M. and M.P. Couper. 1998. Non-Response in Household Interview Survey. New York: John Wiley & Sons.10.1002/9781118490082
    Search in Google ScholarBack to article
  19. Groves, R.M., J.M. Brick, M.P. Couper, W. Kalsbeek, B. Harris-Kojetin, F. Kreuter, B.-E. Pennell, T. Raghunathan, B. Schouten, T. Smith, R. Tourangeau, A. Bowers, M. Jans, C. Kennedy, R. Levenstein, K. Olson, E. Peytcheva, S. Ziniel, and J. Wagner. 2008. “Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools.” Survey Practice 1(3): 1–6. DOI: https://doi.org/10.29115/SP-2008-0013.10.29115/SP-2008-0013
    Search in Google ScholarBack to article
  20. Gummer, T. and J.E. Blummenstiel. 2018. “Experimental Evidence on Reducing Nonresponse Bias through Case Prioritization: The Allocation of Interviewers.” Field Methods 30(2): 124–139. DOI: https://doi.org/10.1177/1525822X18757967.10.1177/1525822X18757967
    Search in Google ScholarBack to article
  21. Kreuter, F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645(1): 23–35. DOI: https://doi.org/10.1177/0002716212456815.10.1177/0002716212456815
    Search in Google ScholarBack to article
  22. Kreuter, F. and K. Olson. 2011. “Multiple Auxiliary Variables in Nonresponse Adjustment.” Sociological Methods & Research 40(2): 311–332. DOI: https://doi.org/10.1177/0049124111400042.10.1177/0049124111400042
    Search in Google ScholarBack to article
  23. Lin, I.-F. and N.C. Schaeffer. 1995. “Using Survey Participants to Estimate the Impact of Nonparticipation.” Public Opinion Quarterly 59(2): 236–258, DOI: https://doi.org/10.1086/269471.10.1086/269471
    Search in Google ScholarBack to article
  24. Little, R.J.A. and S. Vartivarian. 2005. “Does Weighting for Non-Response increase the Variance of Survey Means?” Survey Methodology 31(2): 161–68. DOI: https://doi.org/10.2307/1403140.10.2307/1403140
    Search in Google ScholarBack to article
  25. Little, R.J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review / Revue Internationale de Statistique 54(2), 139–157. JSTOR. DOI: https://doi.org/10.2307/1403140.10.2307/1403140
    Search in Google ScholarBack to article
  26. Little, R.J.A. and D.B. Rubin. 2014. Statistical Analysis with Missing Data. London, UK: John Wiley & Sons.
    Search in Google ScholarBack to article
  27. Luiten, A. and B. Schouten, 2013. “Tailored Fieldwork Design to increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): 169–189. DOI: https://doi.org/10.1111/j.1467-985X.2012.01080.x.10.1111/j.1467-985X.2012.01080.x
    Search in Google ScholarBack to article
  28. Matsuo, H., J. Billiet, G. Loosveldt, F. Berglund, and Ø. Kleven. 2010. “Measurement and Adjustment of Non-response Bias based on Non-response Survey: the Case of Belgium and Norway in the European Social Survey Round 3.” Survey Research Methods 4(3): 165–178. DOI: http://dx.doi.org/10.18148/srm/2010.v4i3.3774.
    Search in Google ScholarBack to article
  29. McCaffrey, D.F., G. Ridgeway, and A.R. Morral. 2004. “Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.” Psychological Methods 9: 403–425. DOI: https://doi.org/10.1037/1082-989X.9.4.403.10.1037/1082-989X.9.4.40315598095
    Search in Google ScholarBack to article
  30. Moore, J.C., G.B. Durrant, and P.W. Smith. 2018. “Data Set Representativeness during Data Collection in Three UK Social Surveys: Generalizability and the Effects of Auxiliary Covariate Choice.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181(1): 229–248. DOI: https://doi.org/10.1111/rssa.12256.10.1111/rssa.12256
    Search in Google ScholarBack to article
  31. Nagelkerke, N.J. 1991. “A Note on a General Definition of the Coefficient of Determination.” Biometrika 78(3): 691–692. DOI: https://doi.org/10.1093/biomet/78.3.691.10.1093/biomet/78.3.691
    Search in Google ScholarBack to article
  32. Nishimura, R., J. Wagner, and M. Elliott. 2016. “Alternative Indicators for the Risk of Non-response Bias: A Simulation Study.” International Statistical Review 84(1): 43–62. DOI: https://doi.org/10.1111/insr.12100.10.1111/insr.12100487131627212786
    Search in Google ScholarBack to article
  33. Olmos, A. and P. Govindsamy. 2015. “A Practical Guide for using Propensity Score Weighting in R.” Practical Assessment, Research & Evaluation, 20(13). Available at: http://pareonline.net/getvn.asp?v=20&n=13. (accessed June 2020).
    Search in Google ScholarBack to article
  34. Olson, K. 2006. “Survey participation, Non-Response Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. DOI: https://doi.org/10.1093/-poq/nfl038.
    Search in Google ScholarBack to article
  35. Olson, K. 2013. “Do Non-response Follow-ups Improve or Reduce Data Quality?: A Review of the Existing Literature.” Journal of the Royal Statistical Society: Series A: Statistics in Society 176(1): 129–145. DOI: http://doi.org/10.1111/j.1467-985X.2012.01042.x.10.1111/j.1467-985X.2012.01042.x
    Search in Google ScholarBack to article
  36. Peytchev, A., E. Peytcheva, and R.M. Groves. 2010. “Measurement Error, Unit Nonresponse, and Self-Reports of Abortion Experiences.” Public Opinion Quarterly 74: 319–327. DOI: https://doi.org/10.1093/poq/nfq002.10.1093/poq/nfq002
    Search in Google ScholarBack to article
  37. Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Non-response Bias in Survey Estimates.” Journal of Official Statistics 25: 193–201. Available at: https://www.researchgate.net/publication/282119961_Using_Variation_in_Response_Rates_of_Demographic_Subgroups_as_Evidence_of_Nonresponse_Bias_in_Survey_Estimates (accessed June 2020).
    Search in Google ScholarBack to article
  38. Roberts, C. and C. Vandenplas. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2): 303–334. DOI: https://doi.org/10.1515/jos-2017-0016.10.1515/jos-2017-0016
    Search in Google ScholarBack to article
  39. Roberts, C., C. Vandenplas, and M. Ernst Stähli. 2014a. “Using Register Data to assess the Impact of Response Enhancement Methods on the Risk of Non-response Bias.” Survey Research Methods 8(2): 67–80. DOI: http://dx.doi.org/10.18148/srm/2014.v8i2.5459.
    Search in Google ScholarBack to article
  40. Roberts, C., N. Allum, and P. Sturgis. 2014b. “Non-response and Measurement error in an online panel: Does additional Effort to Recruit Reluctant Respondents Result in Poorer Quality Data?” In Online Panel Research: A Data Quality Perspective. edited by M. Callegaro, R. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas. Hoboken: Wiley, Survey Methodology Series.
    Search in Google ScholarBack to article
  41. Sakshaug, J.F. and M. Antoni. 2018. “Evaluating the Utility of Indirectly Linked Federal Administrative Records for Nonresponse Bias Adjustment.” Journal of Survey Statistics and Methodology 7(2): 227–249. DOI: https://doi.org/10.1093/jssam/smy009.10.1093/jssam/smy009
    Search in Google ScholarBack to article
  42. Särndal, C.-E. and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36(2): 131–144.
    Search in Google ScholarBack to article
  43. Schouten, B. 2018. “Statistical inference based on randomly generated auxiliary variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80(1): 33–56. DOI: https://doi.org/10.1111/rssb.12242.10.1111/rssb.12242
    Search in Google ScholarBack to article
  44. Schouten, B. and N. Shlomo. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85(1): 143–163. DOI: https://doi.org/10.1111/insr.12159.10.1111/insr.12159
    Search in Google ScholarBack to article
  45. Schouten, B. and F. Cobben. 2007. “R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes.” Statistics Netherlands. Discussion Paper, 07002, CBC, Voorburg. Available at: http://hummedia.manchester.ac.uk/institutes/cmist/risq/schouten-cobben-2007-a.pdf (accessed June 2020).
    Search in Google ScholarBack to article
  46. Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35(1): 101–113. Available at: https://pdfs.semanticscholar.org/aa59/4bf03a7cc219ccc6da01d1e3cb14a125d67a.pdf (accessed June 2020).
    Search in Google ScholarBack to article
  47. Schouten, B., F. Cobben, P. Lundquist, and J. Wagner. 2016. “Does more Balanced Response imply less Non-response Bias?” Journal of the Royal Statistical Society, Series A (Statistics in Society) 179(3): 727–748. DOI: https://doi.org/10.1111/rssa.12152.10.1111/rssa.12152
    Search in Google ScholarBack to article
  48. Schouten, B., J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80(3): 382–399. DOI: https://doi.org/10.1111/j.1751-5823.2012.00189.x.10.1111/j.1751-5823.2012.00189.x
    Search in Google ScholarBack to article
  49. Schouten, B., N. Shlomo, and C. Skinner. 2010. “Indicators for Representative Response.” Paper presented at Quality in Official Statistics Conference 2010, Helinski, Finland. Available at: https://q2010.stat.fi/ (accessed June 2020).
    Search in Google ScholarBack to article
  50. Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27(2): 231–253. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/indicators-for-monitoring-and-improving-representativeness-of-response.pdf (accessed June 2020).
    Search in Google ScholarBack to article
  51. Stoop, I.A. 2004. “Surveying Nonrespondents.” Field Methods 16(1): 23–54. DOI: https://doi.org/10.1177/1525822X03259479.10.1177/1525822X03259479
    Search in Google ScholarBack to article
  52. Stoop, I., J. Billiet, A. Koch, and R. Fitzerald. 2010. Improving Survey Response: Lessons Learned from the European Social Survey. London, UK: John Wiley and Sons Ltd.10.1002/9780470688335
    Search in Google ScholarBack to article
  53. Tancreto, J.G. and M. Bentley. 2005. “Determining the Effectiveness of Multiple Non-response Follow-up Contact Attempts on Response and Data Quality.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2005. 3626–3632. Minneapolis, MN: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2005f.html (accessed June 2020).
    Search in Google ScholarBack to article
  54. Vandenplas, C., D. Joye, M. Ernst Stähli, and A. Pollien. 2015. “Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland.” Survey Research Methods 9(3): 141–158. DOI: https://doi.org/10.18148/srm/2015.v9i3.6138.
    Search in Google ScholarBack to article
  55. Voogt, R.J.J. and W.E. Saris, 2005. “Mixed Mode Designs: Finding the Balance Between Non-response Bias and Mode Effects.” Journal of Official Statistics, 21(3): 367–387. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/-mixed-mode-designs-finding-the-balance-between-nonresponse-bias-and-mode-effects.pdf (accessed June 2020).
    Search in Google ScholarBack to article
  56. Wagner, J. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76(3): 555–575. DOI: https://doi.org/10.1093/poq/nfs032.10.1093/poq/nfs032627697530538342
    Search in Google ScholarBack to article
  57. Williams, D. and J.M. Brick. 2017. “Trends in U.S. Face-To-Face Household Survey Nonresponse and Level of Effort.” Journal of Survey Statistics and Methodology 6(2): 186–211. DOI: https://doi.org/10.1093/jssam/smx019.10.1093/jssam/smx019
    Search in Google ScholarBack to article
  58. Yan, T., R. Tourangeau, and Z. Arens. 2004. “When less is more: Are reluctant respondents poor reporters?” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2004. 4632–4651. Toronto: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2004/files/Jsm2004-000169.pdf (accessed July 2020).
    Search in Google ScholarBack to article
DOI: https://doi.org/10.2478/jos-2020-0034 | Journal eISSN: 2001-7367
Journal RSS Feed
Language: English
Page range: 675 - 701
Submitted on: Jul 1, 2018
Accepted on: May 1, 2020
Published on: Jul 24, 2020
Published by: Sciendo
In partnership with: Paradigm Publishing Services
Publication frequency: 4 times per year
Keywords:
Nonresponse,
R-indicator,
propensity score weighting,
nonresponse survey
Related subjects:
Mathematics,
Probability and statistics

© 2020 Caroline Roberts, Caroline Vandenplas, Jessica M.E. Herzing, published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Previous articleVolume 36 (2020): Issue 3 (September 2020)Next article