Ratio Edits Based on Statistical Tolerance Intervals

Derek S. Young; Thomas Mathew

doi:10.1515/jos-2015-0004

Abstract

The role of statistical tolerance intervals for developing ratio edit tolerances in a parametric setup is investigated. The performance of the methodology is assessed for the normal and Weibull distributions. The numerical results show that in terms of Type I and Type II errors, statistical tolerance intervals exhibit better performance compared to other ratio edit procedures available in the literature. The methodology is illustrated using 2010 and 2011 data from the Annual Survey of Manufacturers.

References

Aggarwal, C.C. 2013. Outlier Analysis. New York: Springer. Bain, L.J. and M. Engelhardt. 1981. “Simple Approximate Distributional Results for Confidence and Tolerance Limits for the Weibull Distribution Based on Maximum Likelihood Estimators.” Technometrics 23: 15-20. DOI: http://dx.doi.org/10.1080/ 00401706.1981.1048623110.1080/00401706.1981.10486231
Search in Google Scholar
Barnett, V. and T. Lewis. 1994. Outliers in Statistical Data, 3rd ed. Wiley Series in Probability and Mathematical Statistics. Chichester: John Wiley & Sons.
Search in Google Scholar
Belcher, R. 2003. “Application of the Hidiroglou-Berthelot Method of Outlier Detection for Periodic Business Surveys.” In Proceedings of the Survey Methods Section: Statistical Society of Canada Annual Meeting, June, 2003. 25-30 Halifax, Nova Scotia, Canada. Available at: http://www.ssc.ca/survey/documents/ SSC2003_R_Belcher.pdf
Search in Google Scholar
Chawla, S. and A. Gionis. 2013. “k-means-: A Unified Approach to Clustering and Outlier Detection.” In Proceedings of the 2013 SIAM International Conference on Data Mining: Society for Industrial and Applied Mathematics, May, 2013. 187-197 Austin, Texas, USA. Available at http://epubs.siam.org/doi/pdf/10.1137/1.9781611972832.2110.1137/1.9781611972832.21
Search in Google Scholar
Cornett, E., J.F. McLaughlin, and C. R. Hogue. 2006. “A Comparison of Two Ratio Edit Methods for the Annual Survey of Government Finances.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August, 2006. 2878-2883 Seattle, WA, USA. Available at: http://www.amstat.org/sections/SRMS/ Proceedings/y2006/Files/JSM2006-000199.pdf
Search in Google Scholar
Franklin, S., S. Thomas, and M. Brodeur. 2000. “Robust Multivariate Outlier Detection Using Mahalanobis' Distance and Modified Stahel-Donoho Estimators.” In Proceedings of the Second International Conference on Establishment Surveys (ICES-II), Survey Methods for Businesses, Farms, and Institutions, June, 2000. 697-706 Buffalo, NY, USA. Available at: http://www.amstat.org/meetings/ices/2000/proceedings/S33.pdf
Search in Google Scholar
Ghosh-Dastidar, B. and J.L. Schafer. 2006. “Outlier Detection and Editing Procedures for Continuous Multivariate Data.” Journal of Official Statistics 22: 487-506.
Search in Google Scholar
Hidiroglou, M.A. and J.-M. Berthelot. 1986. “Statistical Editing and Imputation for Periodic Business Surveys.” Survey Methodology 12: 73-83.
Search in Google Scholar
Hido, S., Y. Tsuboi, H. Kashima, M. Sugiyama, and T. Kanamori. 2011. “Statistical Outlier Detection Using Direct Density Ratio Estimation.” Knowledge and Information Systems 26: 309-336. DOI: http://dx.doi.org/10.1007/s10115-010-0283-2 Hoaglin, D.C., B. Iglewicz, and J.W. Tukey. 1986. “Performance of Some Resistant Rules for Outlier Labeling.” Journal of the American Statistical Association 81: 991-999.10.1080/01621459.1986.10478363
Search in Google Scholar
DOI: http//dx.doi.org/10.1080/01621459.1986.10478363 Iglewicz, B. and D.C. Hoaglin. 1993. How to Detect and Handle Outliers, vol. 16. Milwaukee, WI: American Society for Quality Control.
Search in Google Scholar
Kokic, P.N. and P.A. Bell. 1994. “Optimal Winsorizing Cutoffs for a Stratified Finite Population Estimator.” Journal of Official Statistics 10: 419-435.
Search in Google Scholar
Krishnamoorthy, K. and T. Mathew. 2009. Statistical Tolerance Regions: Theory, Applications, and Computation. Hoboken, NJ: Wiley. Journal of Official Statistics Latouche, M. and J.-M. Berthelot. 1992. “Use of a Score Function to Prioritize and Limit Recontacts in Editing Business Surveys.” Journal of Official Statistics 8: 389-400.
Search in Google Scholar
Montgomery, D.C. 2013. Introduction to Statistical Quality Control, 7th ed. Hoboken, NJ: Wiley. R Development Core Team. 2014. R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria. Available at: http://www.R-project.org/ ISBN 3-900051-07-0 (accessed February 13, 2015) Rais, S. 2008. “Outlier Detection for the Consumer Price Index.” In Proceedings of the Survey Methods Section: Statistical Society of Canada Annual Meeting, May, 2008. 1-10 Ottawa, Ontario, Canada. Available at: http://www.ssc.ca/survey/documents/ SSC2068_5_Rais.pdf.
Search in Google Scholar
Rivest, L.-P. and M. Hidiroglou. 2004. “Outlier Treatment for Disaggregated Estimates.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August, 2004. 4248-4256 Toronto, Ontario, Canada. Available at: http://www.amstat.stat.org/sections/SRMS/Proceedings/y2004/files/Jsm2004-000149.pdf
Search in Google Scholar
Rousseeuw, P. and A.M. Leroy. 2003. Robust Regression and Outlier Detection. Hoboken, NJ: Wiley Series in Probability and Mathematical Statistics.
Search in Google Scholar
Shewhart, W.A. 1939. Statistical Method from the Viewpoint of Quality Control. Washington, DC: Dover.
Search in Google Scholar
Sigman, R.S. 2002. “Statistical Methods Used to Detect Cell-Level and Respondent-Level Outliers in the 2002 Economic Census of the Services Sector.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August 2002. 3566-3573 Minneapolis, MN, USA. Available at: https://www.amstat.org/ sections/SRMS/Proceedings/y2005/Files/JSM2005-000465.pdf
Search in Google Scholar
Tambay, J.-L. 1988. “An Integrated Approach for the Treatment of Outliers in Sub-Annual Economic Surveys.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 229-234. Available at: http://www.amstat.org/ sections/SRMS/Proceedings/papers/1988-040.pdf
Search in Google Scholar
Thompson, K.J. 1999. “Ratio Edit Tolerance Development Using Variations of Exploratory Data Analysis (EDA) Resistant Fences Methods.” In Proceedings of the 1999 Federal Committee on Statistical Methodology Research Conference, November 1999. 1-10. Arlington, VA, USA. Available at: https://fcsm.sites.usa.gov/ files/2014/05/VII-B_Thompson_FCSM1999.pdf.
Search in Google Scholar
Thompson, K.J. 2007. “Investigation of Macro Editing Techniques for Outlier Detection in Survey Data.” In Proceedings of the Third International Conference on Establishment Surveys (ICES-III), Survey Methods for Businesses, Farms, and Institutions, June 2007. 1186-1193. Montreal, Quebec, Canada. Available at http://www.amstat.org/meetings/ ices/2007/proceedings/ICES2007-000071.pdf.
Search in Google Scholar
Thompson, K.J. and S.A. Adeshiyan. 2003. “Data Quality Effects of Alternative Edit Parameters.” Journal of Data Science 1: 1-25.10.6339/JDS.2003.01(1).102
Search in Google Scholar
Thompson, K.J. and R.S. Sigman. 1999. “Statistical Methods for Developing Ratio Edit Tolerances for Economic Data.” Journal of Official Statistics 15: 517-535.
Search in Google Scholar
Tukey, J.W. and D.H. McLaughlin. 1963. “Less Vulnerable Confidence and Significance Procedures for Location Based on a Single Sample: Trimming/Winsorization 1.” Sankhyā: The Indian Journal of Statistics (Series A) 25: 331-352.
Search in Google Scholar
Young, D.S. 2010. “Tolerance: An R package for Estimating Tolerance Intervals.” Journal of Statistical Software 36: 1-39. Available at: http://www.jstatsoft.org/v36/i05/ (accessed February 13, 2015) Yuen, K.-V. and H.-Q. Mu. 2012. “A Novel Probabilistic Method for Robust Parametric Identification and Outlier Detection.” Probabilistic Engineering Mechanics 30: 48-59. DOI: http://dx.doi.org/10.1016/j.probengmech.2012.06.00210.1016/j.probengmech.2012.06.002
Search in Google Scholar

Ratio Edits Based on Statistical Tolerance Intervals

Abstract

Paradigm

My account