Preparing for the 2000 Census : Interim Report II [1 ed.] 9780309591003, 9780309058803

Citation preview

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

i

Preparing for the 2000 Census Interim Report II

Andrew A. White and Keith F. Rust, Editors

Panel to Evaluate Alternative Census Methodologies Committee on National Statistics Commission on Behavioral and Social Sciences and Education National Research Council

NATIONAL ACADEMY PRESS Washington, D.C. 1997

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ii

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self- perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council. This study was supported by Contract No. 50-YABC-5-66005 between the National Academy of Sciences and the Bureau of the Census, U.S. Department of Commerce. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. Additional copies available from: Committee on National Statistics National Research Council 2101 Constitution Avenue, N.W. Washington, D.C. 20418 Printed in the United States of America Copyright 1997 by the National Academy of Sciences. All rights reserved.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

iii

PANEL ON ALTERNATIVE CENSUS METHODOLOGIES KEITH F. RUST (Chair), Westat, Inc., Rockville, Maryland RONALD F. ABLER, Association of American Geographers, Washington, D.C. ROBERT M. BELL, RAND, Santa Monica, California GORDON J. BRACKSTONE, Statistics Canada, Ottawa, Ontario JOHN L. CZAJKA, Mathematics Policy Research, Inc., Washington, D.C. MICHEL A. LETTRE, Maryland Office of Planning, Baltimore D. BRUCE PETRIE, Statistics Canada, Ottawa, Ontario NATHANIEL SCHENKER, Department of Biostatistics, School of Public Health, University of California, Los Angeles STANLEY K. SMITH, Warrington College of Business Administration, University of Florida, Gainesville LYNNE STOKES, Department of Management Science and Information Systems, University of Texas, Austin JAMES TRUSSELL, Office of Population Research, Princeton University ALAN M. ZASLAVSKY, Department of Health Care Policy, Harvard Medical School ANDREW A. WHITE, Study Director MICHAEL L. COHEN, Senior Program Officer AGNES E. GASKIN, Senior Project Assistant

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

iv

COMMITTEE ON NATIONAL STATISTICS NORMAN M. BRADBURN (Chair), National Opinion Research Center, University of Chicago JULIE DAVANZO, RAND, Santa Monica, California WILLIAM F. EDDY, Department of Statistics, Carnegie Mellon University JOHN F. GEWEKE, Department of Economics, University of Minnesota, Minneapolis JOEL B. GREENHOUSE, Department of Statistics, Carnegie Mellon University ERIC A. HANUSHEK, W. Allen Wallis Institute of Political Economy, Department of Economics, University of Rochester RODERICK J.A. LITTLE, Department of Biostatistics, School of Public Health, University of Michigan CHARLES F. MANSKI, Department of Economics, University of Wisconsin WILLIAM NORDHAUS, Department of Economics, Yale University JANET L. NORWOOD, Urban Institute, Washington, D.C. EDWARD B. PERRIN, Department of Health Services, University of Washington PAUL ROSENBAUM, Department of Statistics, Wharton School, University of Pennsylvania KEITH F. RUST, Westat, Inc., Rockville, Maryland FRANCISCO J. SAMANIEGO, Division of Statistics, University of California, Davis MIRON L. STRAF, Director

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

CONTENTS v

Contents

Executive Summary 1

1 Introduction 3

2 Application of Sampling Procedures 6

3 Addresses Linked to Geography: Cornerstone of the 2000 Census 13

4 Improved Survey Methods: Making It Easier to Respond 24

5 Sampling for Nonresponse Follow-Up: Achieving Adequate Precision at Acceptable cost 30

6 Integrated Coverage Measurement: Tackling the Differential Undercount 46

7 Administrative Records: Looking to the Future 62

References 79

Appendix: Sampling in the 2000 Census: Interim Report I 85

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

CONTENTS vi

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

EXECUTIVE SUMMARY

1

Executive Summary

The Bureau of the Census is completely redesigning the census to achieve a modern, efficient, integrated, and accurate approach to counting the U.S. population. The Bureau has conducted an extensive program of research, testing, and evaluation of its methods in anticipation of the 2000 census. The goals of this effort are fourfold: to reduce costs, to reduce respondent burden, to do a better job of counting traditionally undercounted groups, and to improve data quality. The Census Bureau asked the Committee on National Statistics to form the Panel to Evaluate Alternative Census Methodologies, which is charged with evaluating the Bureau's plans and current research on the design of the 2000 census and identifying short- and long-term research issues. Many of the Bureau's plans and the directions of its research are in accord with findings and recommendations made in two predecessor reports: Counting People in the Information Age (Steffey and Bradburn, 1994) and Modernizing the U.S. Census (Edmonston and Schultze, 1995). In 1995 the Census Bureau conducted a census test using the new methodologies planned for the 2000 census. The panel is continuing its review of the Bureau's research plans for the 2000 census and the results of the 1995 test, including the sampling design for nonresponse follow-up, the methodology (including sampling) for integrated coverage measurement, and the use of administrative records. The panel's first interim report, Sampling in the 2000 Census: Interim Report I (White and Rust, 1996), discussed conceptual issues of accuracy of census counts achieved through the use of sampling procedures. The panel concluded in that report that a census of acceptable accuracy and cost is not possible without the use of sampling procedures. We reiterate that conclusion in this report. The panel further concludes that the Census Bureau's research and planning are going in the right direction to ensure an efficient and accurate census. The panel does recommend refinements in several areas that need more attention or in which research in different directions is needed: plans and research in the use of sampling for nonresponse follow-up and plans to introduce integrated coverage measurement that uses sampling; the Bureau's geographic work in compiling the Master Address File and in developing cooperation with local governments; plans and testing of new survey methods (e.g., multiple response modes, respondent friendly questionnaires); and plans for administrative records. Sampling. In addition to considering specific plans to use sampling in nonresponse follow-up and integrated coverage measurement, the panel addressed three general concerns: How will sampling affect public confidence and participation in the census? Will individuals mistakenly think that sampling means they do not have to respond by mail? Will sampling introduce variability at small geographic levels, large enough to compromise the redistricting process? The panel concludes that in the absence of an organized negative publicity campaign, that sampling is likely to increase confidence and is not likely to decrease mail return rates. Using block-level data, sampling should have, at worst, a neutral effect on congressional redistricting, and it should yield more accurate redistricting when blocks are aggregated into meaningful legal and political units.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

EXECUTIVE SUMMARY

2

On the basis of its review of the Census Bureau's plans and research regarding the use of sampling for nonresponse follow-up, the panel recommends direct sampling, appropriate sampling rates, an adequate budget, and the use of housing units as the sampling unit during nonresponse follow-up. For integrated coverage measurement, the Bureau has recently decided to use dual-system estimation. The panel agrees with the decision and recommends combining the best aspects of alternative procedures, carrying out a total error analysis, and calculating the effects on state estimates. Geographic Work. For the Census Bureau's geographic work, the panel finds that the Bureau's plans and progress to develop the most accurate Master Address File and TIGER map data base ever used in a decennial census are outstanding, both in terms of the goals and the proposed methodologies. The panel offers recommendations regarding the development of a cost-benefit model for the updating process, the establishment of explicit criteria for targeting updating checks, the development of a communication plan with local partners, and quality control checking. Survey Methods. The panel finds that the Census Bureau is proceeding well in improvements in survey methods for the 2000 census, including respondent-friendly questionnaires, the availability of forms, targeted Spanish-language questionnaires, a formal promotional campaign, service-based enumeration, and, especially, improved public relations. The panel offers recommendations regarding criteria for distribution of Spanishlanguage forms, evaluation of the formal promotional campaign, and continuing research into the best ways to enumerate people with no usual residence. Administrative Records. Looking at the Census Bureau's plans and research to date on the use of administrative records in the census process, the panel agrees with the Bureau's recent decision to limit the use of administrative records in 2000 and offers recommendations regarding further evaluation and research.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTRODUCTION

3

1 Introduction

This second interim report of the Panel to Evaluate Alternative Census Methodologies, simply put, is about change. The Bureau of the Census is in the process of overhauling and updating the census in response to external change and congressional mandate (see White and Rust, 1996). It is completely redesigning the census process to achieve a modern, efficient, integrated, and accurate approach to counting the U.S. population (Bureau of the Census, 1996). Changing, updating, and adapting the census methods is a proven and desirable course of action. Change is not the enemy of an accurate and useful census; rather, not changing methods as the United States changes would inevitably result in a seriously degraded census. The history of census methods has been one of continuous change, innovation, and evaluation. The U.S. population, culture, knowledge, and technology are vital, active, and ever-changing. Census methods have changed not only to keep up with these changes, but also to take advantage of scientific and technical advances to produce more and better information from the census. From the first decennial in 1790 through the latest in 1990, census methods have evolved and improved to accommodate developments in communications, transportation, patterns of living, migration, work force characteristics, information needs, and many other factors. Proposed change often meets with resistance, and the proposed changes in the decennial census are no exception. Reasonable doubt in itself is a good thing: it results in appropriate research to weed out weaknesses and strengthen plans. In effect, it contributes to the likelihood of successful change. To avoid all risk by resisting reasonable change can be detrimental in a dynamic environment. It is also antithetical to the very essence of the dynamic U.S. society and the American spirit. The Census Bureau has conducted an extensive program of research, testing, and evaluation of its survey methods since the 1990 census. The goals of this effort have been fourfold: (1) (2) (3) (4)

to reduce costs, to reduce respondent burden, to do a better job of counting traditionally undercounted groups, and to improve data quality.

Many of the Census Bureau's plans were responsive to reports of predecessor panels of The Committee on National Statistics (Steffey and Bradburn, 1994; Edmonston and Schultze, 1995). The Census Bureau asked the Committee on National Statistics to form the Panel to Evaluate Alternative Census Methodologies to review and evaluate the Bureau's plans and current research on the design of the 2000 census and to identify short- and longterm research issues. In 1995 the Census Bureau conducted a field test of a new integrated approach,

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTRODUCTION

4

combining the use of sampling and estimation with traditional mail and physical enumeration (Chapter 4 discusses other innovations included in the test). Sampling was used at two points in the census process: in follow-up interviews for a sample of nonrespondents to mail questionnaires and in a coverage measurement survey at the end of nonresponse follow-up, with results incorporated into the census test. The 1995 census test was conducted in three locations: Paterson, New Jersey; Oakland, California; and six parishes in northwest Louisiana. The Census Bureau carried out extensive analyses of the census test results and has documented the research in a series of memoranda (Bureau of the Census, 1995). Number 46 in this series (Vacca, Mulry, and Killion, 1996) is a very useful compilation of results and decisions. That work was reviewed for this report. The Census Bureau has also conducted a smaller census test, called the 1996 Community Census, in six census tracts in Chicago, Illinois, and in the pueblo of Acoma Reservation in New Mexico and the Fort Hall Reservation and Trust Lands in Idaho. The 1996 test was primarily concerned with evaluating refinements in integrated coverage measurement (ICM) procedures (Whitford, 1996). The results from this test were not available for this report. The Bureau has also released a comprehensive plan for conducting the redesigned 2000 census (Bureau of the Census, 1996) and has conducted a series of public presentations around the country to engage local interests and gain feedback. This report evaluates information from the 1995 census test, analyzing a variety of issues and test results that bear on the success of the 2000 decennial census. The Census Bureau has developed a detailed and dynamic research agenda (Killion, 1996a) for preparing for the decennial census. We applaud this effort and believe that resources invested in research now can have a big payoff in a more efficient, accurate, and operationally smooth census in 2000. The panel's first interim report (White and Rust, 1996) discussed issues of accuracy of census counts achieved through the use of sampling procedures. The report pointed out that, in addition to the significant cost saving in field data collection, the use of sampling for nonresponse follow-up offers other potential benefits. The report described the ways in which the use of sampling for nonresponse follow-up could reduce the level of nonsampling error that arises in the process of collecting census data. The next chapter of this report reviews some additional issues related to the use of sampling procedures as a part of census operations that were not addressed in the panel's first interim report, particularly the implications of sampling for public confidence in the census, mail return rates, and the accuracy of small-area data. Chapter 3 discusses plans and procedures for building an accurate Master Address File (MAF) and for updating the Census Bureau's Topologically Integrated Geographic Encoding and Referencing (TIGER) system database. An accurate geographic representation of the territory to be enumerated and a complete list of addresses referenced to their correct geographic location are crucial to the success of the proposed 2000 census methodology. In Chapter 4 the panel briefly discusses several new methods tested or under consideration for the decennial census, including respondent-friendly questionnaires,

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTRODUCTION

5

multiple modes of response, Spanish-language area targeting, targeted promotional campaigns, and new approaches to enumerating people with no usual residence. These topics were discussed in detail in previous panel reports (Steffey and Bradburn, 1994; Edmonston and Schultze, 1995); the emphasis here is on information and insight gained from the 1995 census test. Chapter 5 considers sampling for nonresponse follow-up in some detail. The panel discusses the major design decisions and alternative implementation plans. Although sampling for nonresponse follow-up will improve efficiency and yield some improvements in quality, it is the integrated coverage measurement component of the 2000 census that will address long-standing issues of undercoverage and accuracy. Chapter 6 reviews the history of under coverage and coverage measurement and it looks at experimental results for two methods proposed for integrated coverage measurement. Finally, in Chapter 7, the panel reviews the Census Bureau's progress in research on the use of administrative records in the 2000 census. The Census Bureau has investigated a variety of uses for administrative data, and the panel considers the implications of the research to date for each use. Specific recommendations are made throughout the report.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

6

2 Application of Sampling Procedures

NEED FOR SAMPLING In its first interim report, the panel reviewed the possible uses of sampling procedures in the census, both as a means of completing the enumeration of households (sampling for nonresponse follow-up) and as a means of increasing the accuracy of the count by identifying people missed in the census enumeration and adjusting population figures accordingly (integrated coverage measurement). In that report the panel discussed the potential benefits of sampling for both uses and concluded that, in both cases, there is clear potential for improvements in census data through sampling. The report also concluded that it appeared at that time that the Census Bureau would be able to implement those sampling procedures for 2000 in a way that would realize the potential benefits. However, the panel noted that the Census Bureau would need to apply substantial thought, planning, and diligence between then and 2000 if sampling procedures are to be implemented in a way that will realize their full potential to produce a census of higher quality with controlled cost. At this time, the panel finds that the Census Bureau has made substantial progress in developing procedures for conducting nonresponse follow-up on a sample basis and in developing a large-scale sample survey for integrated coverage measurement (see Chapters 5 and 6). It is clear, however, that the Bureau faces additional work to guarantee that the 2000 census will be a well-managed and cost-effective census that meets its constitutional mandate and is of uniformly high quality throughout the nation. The fact that sizable challenges remain in the development of a census that uses sampling methods does not change the panel's assessment that a census without sampling will almost certainly be unacceptable in terms of both quality and cost. The panel concludes that there is no reasonable ''fall-back position" for the 2000 census. As we indicated in our previous report, echoing the message of the National Research Council's two previous panels on the 2000 census (Steffey and Bradburn, 1994; Edmonston and Schultze, 1995), we do not believe that a census of acceptable accuracy and cost is possible without the use of sampling procedures, for both nonresponse follow-up and integrated coverage measurement. The potential for improved data quality through the use of sampling for nonresponse follow-up derives from two main features. First, the use of sampling will reduce the field workload and may result in more timely completion of the nonresponse follow-up procedures in the field. This increased timeliness will increase data quality because respondents will typically be giving information to enumerators closer to census day than they would if nonresponse follow-up without sampling were implemented. There will be fewer recall errors and less use of poor-quality "last-resort" information obtained from indirect sources in the final stages of field data collection. In addition, the

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

7

more timely completion of nonresponse follow-up will permit the coverage measurement survey to be implemented in the field closer to census day, thereby decreasing recall errors and the effect of people who move on the process of coverage measurement and correction. Second, the use of sampling will make it possible to use better qualified and more highly trained personnel to conduct the nonresponse follow-up work. We stress again, as we did in our first interim report, that it would not be feasible to implement the intensive procedures that would be needed to significantly improve census coverage without the use of sampling. The initial census process itself will use the best methods possible to identify every household in the country and the best techniques available so that householders can provide information about all residents of the household. Yet experience in recent censuses clearly shows that the number of people missed as a result of missed dwellings and of people missed in enumerated dwellings is too great and, in particular, too inequitably distributed to be ignored if a census is to be of adequate quality. Thus, to conduct an adequate census, sampling procedures must be used and the results integrated into the final population counts. Although sampling procedures to "complete the count" have not been used previously in a decennial census (either for nonresponse follow-up or for coverage measurement), the procedures proposed are well established in the production of official statistics and in the conduct of scientific research. Indeed, there is widespread public confidence in the data collected by the census long form, which is on a sample basis from 1 in 6 households in 1990. Such sampling for expanded information began in the 1940 census. Probability sampling procedures are acknowledged to be an objective method of collecting data from which it is possible to obtain valid measures of the level of variability introduced as a result of using a sample. The exact procedures for implementing sampling and its associated estimation procedures, both for nonresponse follow-up and for coverage measurement, must be scientifically established on the basis of available evidence. The panel finds that the Census Bureau is using such an approach to develop its procedures. The final procedures to be used must be shared with knowledgeable data users and other interested parties and must be clearly established prior to the conduct of the census. With this approach, the use of sampling in enumerating the population will be demonstrably free from influence aimed at achieving a particular result in a given geographic area. CONCERNS ABOUT THE USE OF SAMPLING There are several objections that have been raised about the use of sampling as part of census procedures. Some concerns are legal: we do not attempt to address those, which are largely outside the panel's area of expertise. We do note, however, that like a previous panel (Edmonston and Schultze, 1995) we have not seen evidence of any prevailing or significant legal opinion that the use of sampling, in the manner contemplated by the Census Bureau, would result in a census that did not fulfill constitutional requirements.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

8

In the remainder of this chapter, we discuss three concerns about the quality of results from a census that uses sampling: 1. Will uncertainties in the population counts, due to sampling variability, undermine public confidence in the results? Will state and local participation in the census process decline as a result?| 2. Will people assume that they do not need to respond by mail because the use of sampling means that their participation makes no difference to the results? 3. Will the use of sampling for nonresponse follow-up compromise the accuracy of small-area data used for redistricting, since estimates for small areas may have substantial sampling variability? To ensure clarity, a brief review of sampling terminology precedes the discussion. The Terminology of Sampling Variability Throughout this report, the panel refers to "sampling variability," "sampling error," "confidence interval," ''standard error," and "coefficient of variation." These terms all refer to the measurable uncertainty introduced by sampling. For example, the media may report that 58 percent of the population drink coffee and qualify the statement by adding "plus or minus 3 percent." The "plus or minus 3 percent" is one way to express that there is sampling variability, or sampling error, inherent in the estimate from the particular sample design used to gather the data. The two terms, sampling variability and sampling error, have a general connotation and are used to express the fact that estimates made from samples have some known and measurable uncertainty (or variation). In contrast, "standard error", "confidence interval", and "coefficient of variation" refer to mathematically precise expressions of this variability for a particular estimate from a particular sample design. Both the sample size and other aspects of the sample design (respondent selection scheme) affect the amount of uncertainty, which can be expressed as a standard error, coefficient of variation, or a confidence interval. When an estimate is made from sample data, whether it is a percentage, a count, or some other measure, the standard error of the estimate is expressed in the same units of measurement (percentages, counts, etc.). As a general rule in statistics, one can be 95 percent confident that the process of drawing a sample and computing a range defined by first subtracting and then adding twice the standard error of the estimate to the estimate itself, will yield a range that includes the true value. The coffee drinking example expressed one such 95 percent confidence interval: that 58 percent plus or minus 3 percent of the population drink coffee and, thus, that one can be confident at the 95 percent level that the true value lies between 55 and 61 percent. The standard error is roughly half of 3 percent, or 1-1/2 percent. Or consider an example of estimating the population size of a small city instead of percent of coffee drinkers. A sample might yield an estimate of say, 100,000 people. The standard error of this estimate, dictated by sample design parameters, will be expressed in the same unit. If the standard error is 2,300 people, a 95 percent

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

9

confidence interval would be 100,000 plus or minus 4,600 (2 times 2,300), or from 95,400 to 104,600 people. In order to compare the relative effect of sample designs on different kinds of estimates, one has to express the sampling variability in a standardized or comparable manner, the "coefficient of variation." The coefficient of variation is a relative measure, in contrast to the standard error, which is expressed as a percentage of the estimate itself. Thus, in the coffee example, the standard error of 1.5 percent becomes a coefficient of variation of .023 (1.5/58) or 2.3 percent. Coefficients of variation are always expressed as a percentage and are therefore directly comparable. A coefficient of variation can be converted to a 95 percent confidence interval by subtracting and adding to an estimate a quantity equal to twice the coefficient of variation (percentage) of the estimate itself. For example, 2.3 percent of 100,000 is 2,300, so twice that or 4,600 is subtracted and added to 100,000 to get an interval of 95,400 to 104,600 people. When comparing the costs and other relative advantages and disadvantages of alternative sample designs, it is useful either to hold the expected coefficients of variation constant and look at the differences in cost or to hold the cost constant and look at the difference in coefficients of variation. In sum, while sampling variability and sampling error are general terms expressing the fact that sample estimates have some measurable variability, standard error and confidence interval are specific measures of the variability of a specific estimate, and the coefficient of variation is a unit-free relative measure that expresses the standard error as a percentage of the estimate. This unit-free measure is easy to compare across different types of estimates and for alternative sample designs. Sampling Error and Public Confidence Knowledgeable census data users, especially state and local officials, are aware that the 1990 and earlier census data were not error-free at the local level. In fact, in some cases state and local officials found the results of these past censuses very much lacking in credibility. This problem was exacerbated by the fact that there was no procedure for quantifying the possible size of the error in a given jurisdiction. Under these circumstances, a census that combines sampling with other procedures to improve the quality of the count has the potential to increase public confidence in the result. The fact that sampling error can be measured will provide confirmation that sampling errors for relatively large geographic areas are actually small. Furthermore, the panel believes that sampling will actually reduce nonsampling error because census resources can be strategically targeted to improve quality. The Census Bureau must do a careful job in communicating to census users what sampling involves and what the resulting numbers will and will not mean. The idea of sampling as part of the procedure for obtaining population counts is novel, that it is important that its role be explained carefully and often. If this is done successfully, however, there is good reason to think that, through the use of sampling, together with other enhancements to census procedures, public confidence in census results can be increased.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

10

Sampling and the Mail Return Rate The issue of whether the use of sampling for nonresponse follow-up might have an adverse effect on mail return rates is an important one. The Census Bureau has undertaken a number of enhancements aimed at increasing the rate of return by mail. It would certainly be counterproductive if the introduction of sampling were to have the unintended consequence of reducing the mail response rate. The arguments suggesting that sampling for nonresponse follow-up might reduce mail response take two lines. The first is that households will make the calculation that if they fail to return the form by mail, there will be no in-person follow-up to collect a form. Thus, the household can minimize its expected total burden by failing to complete and return the form that it receives in the mail. The logic of this argument assumes that (in the absence of nonresponse follow-up sampling) households know that failing to return their forms by mail is certain to result in a visit from an enumerator to obtain it. The second line of argument that mail response will go down as a result of sampling has the reverse underlying assumption. This argument is that, in the absence of sampling, householders will assume that an accurate count depends on their returning the form by mail; that if they do not return the form, their household will not be counted. If there is widespread public awareness of the use of sampling, however, then some householders will determine that the result of the count is unaffected by whether they return the form by mail or not, and so will not bother. Ironically, it is actually when no sampling is used that the result of the census is largely unaffected by whether a household returns its form by mail (setting aside prohibitive logistical and cost barriers to contacting every single nonresponding household). Thus, in the absence of any research about householder behavior in a real census, one might just as well argue that the absence of any sampling for nonresponse follow-up has traditionally held down mail return rates, since respondents expected in-person follow-up and might not have perceived added benefit from returning the form by mail. The panel has seen no evidence that households go through the kind of logical calculations hypothesized above when deciding whether to respond by mail. The concern that the introduction of sampling procedures will have a noticeable effect on the rate of mail return is speculative at this point, with no evidence that such an effect will occur. There is an absence of empirical research on this topic. However, sampling was used in the 1995 census test, and although there was no control group, mail return rates below those that would have been expected without sampling were not observed. Frankly, in the absence of an organized negative publicity campaign, we consider it unlikely that any significant proportion of households will make any connection between their decision whether or not to return the form by mail and the use of sampling for nonresponse follow-up. Other factors, some under the control of the Census Bureau and others not, are much more likely to significantly affect the mail return rate: public perception of the importance of the census, trust in and respect for government, the clarity of the instructions, and the use of reminders and replacement forms, are all much more likely to affect the mail return rate than is the use of sampling for nonresponse follow-up.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

11

However, the panel does believe that it is important for the Census Bureau to communicate clearly to local authorities and knowledgeable users of census data what the plans for nonresponse follow-up sampling are, how they might vary from area to area (depending on the rate of mail return, for example), and the importance of obtaining a high rate of mail return for a successful census. A constant and clear message on these points, which does not oversimplify the issues involved, needs to be an important component of the Census Bureau's plans for 2000. These issues do highlight a very crucial aspect of the sample design for nonresponse follow-up. As we discuss in detail in Chapter 5, it is important that the sample design does not have the consequence that areas with high mail return rates have counts subject to greater levels of sampling variability than areas with low mail return rates. An increase in the mail return rate should (at the least) lead to no increase in the level of sampling error. In fact, a design in which there is a modest improvement in the level of sampling error as the mail return rate increases might give incentives to local governments and interest groups to increase mail return rates. Such a plan might counter the notion that the use of sampling will lead to a decrease in public participation in the census. The Accuracy of Small-Area Data The mathematics of sampling and estimation are such that, for a given sample design, the level of sampling error, relative to the size of the population to be counted, will increase as one moves to smaller and smaller geographic units. In general, other sources of census error tend to remain constant on average across units, relative to population, as one considers increasingly smaller geographic units. Consider, for example, the errors that result from housing units being missed from the Master Address File. If they are missing at an average rate of 1 percent per block, then for a large geographic area that consists of perhaps 10,000 blocks (such as a congressional district), the rate of missing addresses will also be about 1 percent. But the coefficient of variation due to sampling error will only be one-hundredth the size for the congressional district that it is for the blocks in that district. With any reasonable sampling strategy that the Census Bureau might adopt, the level of sampling error would be very small for a large geographic area, such as an urban county. In fact, as pointed out in the panel's first interim report and above, at such a geographic level there is a good case to be made that the introduction of sampling procedures can lead to an overall net reduction in errors because the errors that can be reduced when sampling is used outweigh the sampling error introduced. Inevitably, however, this does not hold true at very small levels of geography. With any reasonable sampling scheme that might be used nationally, there will be some levels of aggregation (for example, census blocks) for which the census count will be less precise on average and would arguably not be an improvement over what could be obtained without the use of sampling. The important point to note here is that for the counts for census blocks, the level

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPLICATION OF SAMPLING PROCEDURES

12

of sampling error is, relatively speaking, not an appropriate criterion for judging the quality of the census. Although block counts may contribute to the congressional redistricting process, for example, it is important to keep in mind that the results in a redistricting process are the counts for the congressional districts that are eventually created (and to a lesser extent, the counts for districts that were, or conceptually might have been, considered but were discarded). For these kinds of counts, the level of sampling error will be modest because the larger the number of observations used for an estimate, the smaller its sampling error will be. Thus, in the panel's view, the important considerations for evaluating whether the amount of sampling error present in the census process is acceptable are not those that relate to counts for very small units, such as blocks. It is clear that at that level, sampling error may be substantial in some cases (again, relative to the size of the block). The evaluation of sampling error should take place for the geographic level counts that have important legal, political, or financial implications. For such levels, a census that uses sampling can achieve results that are at least as good as those from a more time-consuming and expensive effort to obtain a completed form for every household. In summary, then, as we have stated before, the panel concludes that the use of sampling and statistical estimation are important components of the plans for the 2000 census. Both sampling for nonresponse follow-up and sampling for integrated coverage measurement are key to the successful conduct of an affordable enumeration of adequate quality in all parts of the country. Although each type of sampling improves both efficiency and quality, sampling for nonresponse follow-up will make the greatest contribution to cost savings, while integrated coverage measurement contributes more to improved accuracy. The Census Bureau needs to carry out further research to develop the specific details of how each of these components is to be conducted and how they are to be integrated. The Census Bureau must also be careful to inform knowledgeable users of the methods to be used and the reliability of the counts that will be obtained. If sound procedures are developed by the Census Bureau and communicated to users, the panel believes that it will be possible for the Bureau to address all reasonable potential objections to the use of sampling and to satisfy users that the use of sampling has added to the soundness and quality of the 2000 census, rather than detracting from it.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

13

3 Addresses Linked to Geography: Cornerstone of the 2000 Census An accurate geographic representation of the territory to be enumerated, in both cartographical and digital form, and a complete list of addresses referenced to their correct geographic location are crucial to the success of the proposed methodology for the 2000 census. A census that counts people at their usual place of residence requires a geographic framework that delineates the territory where dwellings are located, includes the addresses of all dwelling units, and ensures that these dwelling units are referenced to their correct geographic location. This is essential for accurate census counts and for statistical data for small geographic areas. Both the association of census responses with their correct locations and the inclusion of locations within the correct tabulation areas depend on the accuracy of the geographic framework and the address list. In addition, several of the important methodological innovations being introduced in the 2000 census assume and depend on an accurate geographic framework and an accurate list for their success. The degree of precision required in the geographic framework for the census has evolved over time as the need for finer and finer small-area data has increased. For the 2000 census, dwellings need to be pinpointed very precisely in order to allow the aggregation of census counts and statistical data for many geographic areas, including those user-defined areas that do not respect the boundaries of the standard geographic areas. With the introduction of the Topologically Integrated Geographic Encoding and Referencing (TIGER) system in the 1990 census, the framework for identifying the precise geographic location of each dwelling was established. While TIGER provides the geographic framework within which every dwelling (as well as many other physical features) can be located, it does not provide a list of dwellings. Such a list of dwellings, the Master Address File (MAF), linked to the correct geographic location in TIGER, is crucial to the successful implementation of the 2000 census methodology. Prior to 1970, when censuses were conducted completely by direct enumeration, the interviewers created the list of dwellings as they covered their territory. With the introduction of a mail census, a list of dwellings was needed prior to the actual census as the basis for mailing questionnaires. Up to and including the 1990 census, these initial lists were created anew for each census, using commercial and other sources. Dwellings might be added to these initial lists during the census enumeration and follow-up activities. For the 2000 census, the plan is to create the initial MAF, not from scratch, but from the final 1990 census dwelling list, updated through a variety of sources, including the U.S. Postal Service (USPS), local governments, and, possibly, private companies. The goal is to develop an updating system that will eventually ensure the availability of a current MAF at all times. The case for such a continuously updated MAF does not rest on census considerations alone, but also on the potential uses of such a list for other

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

14

statistical purposes (see Steffey and Bradburn, 1994). Thus, the plan and schedule for MAF have been designed to satisfy not only the needs of the 2000 census, but also the continuous measurement needs of the American Community Survey.1 From the viewpoint of the census alone, there are several benefits of a continuously updated MAF: it provides a smoother or more level work flow, avoiding huge precensus peaks that may be difficult to staff; it provides the opportunity for continuing partnerships with census participants and users, especially local governments; and it facilitates the use of quality control programs that should lead to a higher quality list. With the innovations in methodology proposed for the 2000 census, the fundamental importance of an accurate MAF and TIGER database to the success of the census is significantly increased. As subsequent chapters illustrate, the offering of alternative means of response, the use of sample nonresponse follow-up, and the plans for integrated coverage measurement all depend on having a reliable dwelling list, referenced to correct geographic locations. For example, the matching of information coming from "Be Counted" forms to the census database requires complete and unambiguous address and geographic area identification. The success of both nonresponse follow-up and integrated coverage measurement is contingent on an accurate MAF with clearly identified dwelling units linked to the correct geographic locations. Although the quality of the initial lists was always important for the initial mail-out of census form, it takes on added importance with the introduction of methodology for the 2000 census. The quality of the initial MAF and its linkages to TIGER may well be the single most important factor determining the success of the 2000 census. Parallel updating of the representation of geographic features in TIGER and of dwellings in MAF is not only necessary but efficient. Information sources that generate new addresses will often simultaneously generate geographic features, especially roads and streets, that must be added to TIGER, and vice versa. But equally as important as the current individual updating of TIGER and MAF is the currency of the linkage between them. In updating these sources in parallel, it is important to carefully manage their linkages and vintages. Since some census-taking tools (e.g., maps) are derived from TIGER, while others (e.g., address listings) are derived from MAF, the two databases must be properly linked. Addresses or features reflected in one but not the other, or maps and address lists coming from asynchronous versions of the two databases, can lead to confusion and loss of confidence by local government officials who are helping to improve these databases and by census field staff who are relying on them to take the census.

1 The American Community Survey (ACS) is designed to update regularly the information about communities that the U.S. census has traditionally produced once a decade. "Communities" include geographic areas of all sizes and demographic subgroups (Alexander, 1996).

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

15

NEED FOR AN ACCURATE MAF The completeness of the initial MAF and its accurate referencing to TIGER will play a crucial role in the success of the 2000 census. Although dwellings can be added to the MAF during the enumeration process, the more dwellings that can be accurately included in the initial MAF, the smoother and more efficient will be the enumeration process. Therefore, investment in an accurate MAF, with integrated linkage to TIGER, should have a very high priority in Census Bureau work. The challenge faced by the Census Bureau is to develop the best possible initial MAF given available resources. This challenge involves assuring not only a high average quality level, but also a uniformly high quality level. Since much of the output of the census is data for local areas, the census cannot afford to have areas for which the coverage of dwellings is deficient, even if on average it is very good. Yet it is recognized that whatever methods are used for creating the initial MAF, there are bound to be some areas for which it is not possible to create an adequate initial version. In such areas, census methodology will have to be able to compensate for these deficiencies through special enumeration efforts or other procedures. Five general aspects of the challenge should determine the approach to be used. First, because the territory to be covered is vast and local knowledge is an important ingredient in creating and maintaining an accurate geographic framework with a linked address list, the Census Bureau cannot expect to do this task alone. Partnership is necessary. There are many potential partners who have an interest in a strong geographic framework and a complete address list for the census, including local governments as users of census data, data marketers, and other organizations, such as the USPS, which also requires accurate geographic representation of the country. Second, the difficulty of keeping TIGER and MAF up to date varies greatly across the country. Some relatively stable areas present few problems, while high-growth areas or areas of urban renewal may be very challenging. Urban areas with precise street addresses but with many multiple-unit buildings present different problems from those in rural areas, where addresses may not provide geographic precision. This heterogeneity requires methods for identifying areas that require different kinds of attention and then targeted approaches for dealing with the different circumstances. Third, the magnitude of the task dictates automation almost everywhere. Input from other agencies will normally be in an automated, though not necessarily consistent, format. Flexibility in being able to accept and process different input formats is essential. Some manual entry of inputs or updates is inevitable. Fourth, the development of TIGER and MAF has to proceed in an integrated fashion. It is important to keep the linkages between the two databases current so that the products derived from them at any stage of the census process are consistent with each other. Fifth, the ability of local governments to provide accurate information in a timely manner varies enormously. Thus, plans for working with local governments need to be flexible and adaptable. With these general aspects in mind, we assess the Census Bureau's plans for TIGER and MAF.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

16

CENSUS BUREAU PLANS The current Census Bureau plans, as we understand them, are set out in an unpublished document (Bureau of the Census, 1996b). Overall, we found this plan to reflect a sensible approach to the challenging yet crucial task of providing timely and high-quality TIGER and MAF databases for the 2000 census. As we elaborate below, our concerns center on the feasibility of achieving the plan's objectives in the face of budgetary and human resource constraints and on the procedures for dealing with those parts of the country for which up-to-date geographic and address representations will be difficult to create and maintain. The overall approach embraces three broad strategies. First, to the extent possible, it makes use of automated matching of files and updating of databases in the office, as opposed to field enumerations and checks. Second, it promotes the establishment of partnerships to achieve results. Third, it aims to make maximum use of inputs from USPS files, while taking account of any quality weaknesses, outdatedness, or inconsistency in those files across the entire country. Use of USPS Address Lists At the core of any updating strategy for a national address file, there has to be a nationwide source of current addresses. While the strategy of cooperative updating programs with local governments is essential, their coverage can never be complete and sufficient, and updating processes need to be developed for areas where no local contributions are available. With or without local cooperation, it is necessary to have some nationwide source of updating to ensure at least a minimum consistent level of quality across the country. The USPS provides a nationwide source of addresses that is frequently updated, by its Delivery Sequence File (DSF). However, it is not a panacea. For example, there may be time delays in making updates, unevenness of quality across carrier routes, rural addresses that may not reveal precise geographic locations,2 and addresses that do not have mail delivery that do not appear on the list. In addition, the 1995 test clearly revealed problems with the classification of vacant units on the DSF (Green and Vazquez, 1996). Nevertheless, it still represents the most valuable source of new addresses for the entire country. The Census Bureau is working appropriately to evaluate and understand the properties of this file and to incorporate it as a central component of the MAF updating process. The Census Bureau plan calls for using DSF three or four times each year to update the MAF. The plan ensures that an update using DSF occurs immediately before each major use of the MAF to produce address lists, for independent reviews, for survey frames, or for census mail-outs. Clearly the quality of MAF will be highly dependent on the quality of the DSF. During the 1995 census test, the Census Bureau found that

2 Although this has begun to change, it is clear that this process is not happening fast enough to seriously affect the 2000 census procedure.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

17

updating the MAF with the DSF improved the deliverability of addresses (Corteville, 1996). However, as a rule of thumb, the USPS flags an address as vacant only if it is unoccupied for at least 90 days, so even a very late match to the DSF may not provide a comparable substitute for a manual "census address check" operation conducted by the USPS. Many of the other elements of the MAF updating strategy are designed to supplement the DSF contribution to MAF and to compensate for its weaknesses. Updating TIGER, MAF, and the Linkage Between Them Two programs are addressing the issue of TIGER updates and the linkage of TIGER to MAF. In February 1995 the Census Bureau began an operation called Master Address File Geocoding Office Resolution (MAFGOR) that aims to resolve discrepancies between address range information in TIGER and the individual addresses in MAF. Addresses in MAF that cannot be assigned to an address range in TIGER are resolved using address reference files obtained by regional offices or by information supplied by participants in the second updating program, the TIGER Improvement Program (TIP). As a result of this reconciliation, corrections are made either to TIGER or to MAF so that the problem addresses can be matched in any subsequent geocoding operations. Although originally scheduled for completion in 1998, MAFGOR will now be extended through the 2000 census due to the large number of addresses that are not geocoded. TIP is a partnership program under which local governments are invited to provide the Census Bureau with geographic files and other materials that can be used to update TIGER. TIP began in June 1995 and is intended to be an ongoing program. The MAFGOR/TIP operations have three critical stages. The first stage ends with the production of the maps and lists needed for the local update check of addresses (LUCA) program that provides local governments with a final precensus opportunity to review the Census Bureau's address lists and suggest changes (see below). The second stage adds subsequent improvements, from the LUCA program or elsewhere, to produce the MAF that will be used for the 2000 census mail-out. The third stage adds later arriving changes that can be used for supplemental mail-outs or enumerations. A third program, the Program for Address List Supplementation (PALS), is another partnership program that invites local governments to supply the Census Bureau with their address lists for matching with, and updating of, MAF. Address standards have been specified in the Federal Register, and software for matching lists has been developed. Invitations were sent out in summer 1996, and files provided in response to the invitation are to be verified against the standards and then matched to MAF and TIGER. The Census Bureau intends to feed back information on the disposition of all addresses submitted. For addresses that cannot be matched to TIGER, location information will be sought. All these programs implement the partnership approach with local governments that was advocated by Congress (Bureau of the Census, 1996). But there are potential problems. If the response by local governments is very high, the Census Bureau's resources may be taxed in trying to respond to all submissions, and failure to respond would probably create public relations problems. Yet a very low response rate by local

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

18

governments would put more burden on the other approaches necessary for updating the geographic databases. At this time the panel does not have the information to judge whether the resources available to the Census Bureau for these programs are sufficient for the work. Recommendation: The Census Bureau should develop a cost-benefit model for the planned MAFTIGER updating processes to allow analysis and monitoring of its quality and to assess budgetary implications of different levels of participation or investment in those processes. Targeted Updating An important change of strategy from the 1990 census relates to what used to be known as the precanvass check. Prior to the census mail-out, the Census Bureau used to conduct a complete (100%) check of the address list (which in previous censuses had been based primarily on commercial sources). With the new approach to address list updating through MAF, this complete check is being replaced with a series of targeted checks at various points in the evolution of the MAF. For example, targeted checks are planned for multiple-unit buildings where there is a street address match between the DSF and the MAF but a significant difference in the number of individual dwelling units indicated at the address. A second targeted check is planned in areas with high potential for structure conversions from single-unit structures to multiple-unit structures. The LUCA program reviews can also be thought of as targeted checks. A key targeted check is planned for January 2000 and involves a USPS check of MAF addresses in areas with substantial housing growth. This check would identify addresses to which USPS is delivering mail but that do not appear on the MAF. The goal of this check is to compensate for the time lag in the DSF by identifying new construction (since mid-1999). The strategy of devoting resources to areas where the quality of MAF is thought to be deficient, rather than conducting blanket checks that make little difference in many areas, is clearly sensible if one is able to predict accurately where those deficient areas are. While the various targeted checks planned in the MAF-TIGER updating program focus on the obvious problem areas, the panel has not yet seen a plan for or demonstration of effective criteria for identifying specific kinds of problems. Recommendation: The Census Bureau should develop and make explicit the criteria it will use to determine which areas to include in each of its targeted updating checks. Rural Areas Rural areas present a particular problem of address list maintenance because the mailing address of a housing unit frequently does not reveal its precise geographic

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

19

location. The Census Bureau plans to construct a comprehensive list of rural housing units through a canvassing operation in the first quarter of 1998. This operation will identify the physical location of each dwelling unit on a map. It is timed to feed into the use of MAF for the continuous measurement survey program (the American Community Survey) later in 1998. These address lists may be subsequently updated through PALS or LUCA. The updated versions will then be used as the basis for an ''update/leave" operation immediately before census day. Address lists and maps will be updated and questionnaires left for return by mail. With the trend of extending the use of city-style addresses into rural areas, it may be possible to convert some rural areas to the mail-out part of the MAF before the census. Cooperation with Local Governments: Final Stages As described above, Census Bureau plans invite broad local government participation on a continuing basis in the updating and preparation of the geographic framework and address list for the 2000 census. These partnership programs supplement the regular updates through the DSF, as well as some of the targeted checks that the Census Bureau will be conducting. Local government involvement in the preparation of MAF culminates in the LUCA program, which provides a last opportunity for local governments to review the census address list and point out any needed additions, deletions, or changes. The Census Bureau would then review the suggested changes and update MAF as necessary. For those local governments that choose to participate in TIP or PALS, the LUCA program can be seen as the final confirmation stage of a continuing process. For those that did not participate, it is the chance to provide input into one of the prime determinants of the quality of the 2000 census. As a result of testing the LUCA program in the 1995 census test (Moohn, 1995a, 1995b; Barrett, 1995), several changes in the approach have been incorporated into the Census Bureau plans for 2000, focusing primarily on timing and communication. There are three key constraints on the timing of LUCA. First, the MAF that is supplied for local government review should be as accurate and up to date as possible. It would be wasteful to have local governments pointing out omissions that the Census Bureau would have found later; it would be better if local governments concentrated their effort on those areas where it is difficult for the Census Bureau to obtain good information. Also, sending a clearly deficient MAF to local governments may undermine the recipients' confidence in the census and leave the impression that the Census Bureau is not doing its job. The second constraint is to leave the local governments sufficient time to conduct their reviews thoroughly, given their resource constraints. To give them too little time is to belittle the importance of their contribution. Third, there must be sufficient time after receipt of the suggested changes for the Census Bureau to incorporate them, provide feedback, handle appeals, and update the MAF before the cutoff date for the census mail-out. These conflicting constraints have led to the following schedule: the materials for LUCA are to be prepared and distributed in January 1999; local governments review those materials and provide feedback in February and March 1999; and the Census

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

20

Bureau assesses the proposed changes and provides feedback to local governments in March-June 1999, in time for the cutoff date of July 1999 for preparing the main census mail-out. This is a tight schedule. While it may represent a reasonable compromise between the conflicting time constraints, many local governments may not have the time, expertise, or resources to do more than a perfunctory check of the address list. Ultimately, the effectiveness of the LUCA program depends on the rate of participation by local governments and the extent and quality of the changes they propose. The tight schedule leads to the risks that the Census Bureau will not receive the information they need from local governments, will not be able to incorporate the information, or will not be able to provide the promised feedback on the proposed changes within the time available in the census preparation schedule. These risks should be reflected in the content of the Census Bureau's communications with local governments so as not to raise expectations that cannot be met. In fact, a well-planned communication program with local governments on LUCA will be crucial to its success. The communication plan should cover at least the following elements: • • •

• •

the importance to local governments of a high-quality census, given its effects on revenue distribution, local planning, and redistricting; adequate advance warning of upcoming activities, time constraints, the nature of the work expected of local governments, and the process of review, feedback, and appeal; a clear explanation of the potential impact of the local government's input on the quality and cost of the 2000 census, including, for example, the negative impact of either not checking the lists thoroughly in the expectation that the gaps will get fixed during a census enumeration, or flooding the Census Bureau with doubtful addresses in the mistaken belief that this strategy will maximize the area's eventual census count; emphasis on the fact that LUCA supersedes the former local review of the pre census address list--in other words, local governments will not get another chance after LUCA to review dwelling counts; and how much flexibility the Census Bureau has in accommodating different formats of input and output in sending and receiving materials.

To the extent that the Census Bureau is aware of particular weaknesses in the MAF extract it is sending to a local government, it should point out where the local authority might concentrate its review effort. Recommendation: The Census Bureau should develop a communication plan for its contacts with local governments on the LUCA program, emphasizing the importance of this program to a high-quality census and the implications of their participation in it. The Census Bureau should invite the active participation of all state agencies that are members of the Federal-State Cooperative Program for Population Estimates. These agencies can help the Census Bureau contact the appropriate people in local government

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

21

and help to publicize and encourage participation in the various stages of the local review process. Although not directly related to MAF, a potentially serious communications issue with local governments concerns the plan not to offer local governments a chance to review the final census dwelling counts. While we recognize that offering local governments the opportunity to participate in the updating of the address lists throughout the preparation of the census is a much more constructive approach, we are concerned that the abandonment of a post census local review of the type conducted in 1990 may be seen in isolation as the removal of an appeal right with potential financial implications for local areas. It is most important to ensure that local areas realize that LUCA has replaced this local review and are persuaded of the benefits of this change. Multiple Unit Structures As noted above, multiple-unit structures represent a problem in the updating of MAF due to ambiguity in the delineation of labeling of units within the structure. These structures are also a problem for nonresponse follow-up and integrated coverage measurement operations, which depend on clear identification of the individual units chosen for follow-up or reinterview. There seem to be three kinds of cases in a multiple-unit structure, each requiring a different approach in the census: • units that are easily distinguishable and well labeled: these typically occur in modern apartment buildings and present no problems beyond those experienced in covering single-unit structures; • units easily distinguishable, but without clear or unique labels (e.g., basement, ground, upper, or A, B, C): sensible rules are needed for deducing equivalency, with more emphasis being put on labels that incorporate physical descriptions (e.g., which floor, front or back); • units that are not clearly distinguishable: different visitors to the same structure might count different numbers of units; in some cases the residents may not make the distinctions that the Census Bureau is trying to impose. Both the LUCA program and field checks should pay special attention to structures in the latter two categories. The inclusion in the targeted check of multiple unit buildings that are treated as a single drop for mail delivery purposes, as identified in the DSF, should help to clarify the MAF for many structures in the last two categories. However, there will inevitably remain structures for which unambiguous units cannot be delineated. In these cases, it might even be in the best interests of census coverage to treat the whole structure as the "dwelling unit" for the purposes of MAF, nonresponse follow-up, and integrated coverage measurement, in order to get all the people properly covered. That would imply the flagging of such addresses on the MAF, their omission from the mail-out, and the use of an update-leave or a list-enumerate procedure, where "list" means listing the units within the basic street address. Any subsequent nonresponse follow-up and integrated coverage measurement operations involving the address would

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

22

also be applied to the whole structure. Thus, if any forms were not returned by mail from this structure, the structure would be treated as a single sampling unit for nonresponse follow-up. Though this would represent a significant change from current plans and some added initial cost, it is a change that might alleviate a substantial source of cost, confusion, and complexity in subsequent enumeration and estimation operations. It merits consideration in the context of enumeration approaches for difficult-to-enumerate urban areas. ASSESSMENT The development of a comprehensive, accurate, and timely address list linked to the TIGER system is critical to the success of the 2000 census. This operation is the cornerstone of the entire census process. It is the foundation on which the implementation of the 2000 census methodology rests. Through the TIPS, PALS, and LUCA programs, the TIGER and MAF are reviewed by local governments prior to the census. Local governments' assessment of the completeness and accuracy of these databases will, at a minimum, affect their perception of how good a census the Census Bureau can conduct. The MAF then serves as the basis for the mailout of the census questionnaires. The success of the mail delivery of questionnaires will affect local government, public, and media perceptions of the orderliness of the census process. The success of the subsequent "Be Counted," nonresponse follow-up, and integrated coverage measurement procedures is constrained by the quality of the address list and its geographic referencing to the correct census tabulation blocks. Finally, the published census counts and statistical data for small areas are dependent on completed questionnaires and sampled nonresponse follow-up forms being referenced to the correct geographic areas. In addition to enhancing the TIGER database and integrating it more fully with the MAF, the Census Bureau's plans for improving the quality of the address list are based primarily on input from USPS, and local governments and on targeted field checks. These may be supplemented, in ways not yet clearly defined, by information from commercial lists and from administrative records. While we agree with the general strategy being followed by the Census Bureau, we remain concerned about whether the resources available are adequate for the work to be done, at a high level of quality. We recognize that each of the primary updating sources--the USPS files and local governments--has weaknesses and that a program of targeted checking aims to address these weaknesses. The impact that private commercial lists will have on the quality of the MAF is still unknown because the solicitation of such lists took place only at the end of 1996. The panel has not seen any estimates of the expected quality of the address list after updates from these sources and therefore of the extent of dependence on targeted checking for the completeness of the final list. We are concerned about how precisely, and on the basis of what information, areas to be targeted can be identified. In the absence of a complete but expensive pre census field check, the success of targeting becomes a critical determinant of the quality of the MAF. The Census Bureau has not provided a cost-quality tradeoff analysis that

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADDRESSES LINKED TO GEOGRAPHY: CORNERSTONE OF THE 2000 CENSUS

23

demonstrates that the proposed updating strategy, coupled with targeted pre census checking, would perform better than an approach incorporating a final universal pre census field check. In the absence of such an analysis, it might be wiser to reverse the burden of proof and develop criteria for exempting from an otherwise universal pre census check those areas for which an acceptably high level of accuracy from the updating processes can be confirmed. This approach would put the onus on demonstrating accuracy, rather than on spotting inaccuracy. Since developing an accurate MAF is the first step in taking the census, we reiterate its critical role in the decennial census. Recommendation: The Census Bureau should develop and maintain on the MAF a system of quality indicators at or below the census tract level and should use these indicators as a means of identifying areas requiring special pre census field checking to bring them up to an acceptable level of quality.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

24

4 Improved Survey Methods: Making It Easier to Respond

In order to achieve its goals--to reduce costs and respondent burden, to do a better job of counting traditionally undercounted groups, and to improve data quality--the Census Bureau tried five new survey methods in the 1995 census test: (1) development of respondent-friendly questionnaires; (2) provision of multiple modes for response; (3) mailing Spanish-language forms, in addition to English ones, to targeted areas; (4) targeted promotional campaigns; and (5) new approaches to enumerating people with no usual residence, service-based enumeration. Evaluation of these methods showed that some were successful, some showed promise but needed changes, and others did not appear to have any positive effect on either the response rate or data quality. The goals also sometimes proved to be conflicting, making decisions about implementation more difficult. RESPONDENT-FRIENDLY QUESTIONNAIRES Improvement in the census questionnaire has the potential to reduce costs by increasing mail response, by far the least expensive way to count household members. In past censuses, the questionnaire's design was based on a need for easy data capture, without much concern for its effect on respondents. Over the last couple of decades, however, considerable research has shown that the appearance of a survey questionnaire and the form of the appeals made for cooperation can substantially increase response rates. This approach, called the total design method, has a carefully prescribed series of steps for presentation of a questionnaire to respondents. The Census Bureau began small-scale tests and focus group evaluations of these methods in 1992, and the new procedures were implemented in the 1995 census test (Treat, 1995a,b). Research in this area by the Bureau is continuing. One of the major ways that the total design method differs from previous census operations is that it makes multiple mail contacts with a household. Plans for the 2000 census now include sending an advance letter a few days before, and a reminder postcard a few days after, the census questionnaire arrives by mail. In addition, a replacement questionnaire will be sent a couple of weeks after the initial questionnaire mailing to those households that have not yet responded. The test results indicate that this is an especially fruitful strategy. Evaluations in the early tests estimated that the prenotice letter and post mailing postcard increased the mail response rate by about 12.7 percent, and the replacement questionnaire produced an additional gain of about 10 percentage points. If both effects persist in the actual census, they would produce a cost savings of about half a billion dollars (Edmonston and Schultz, 1995), minus the added cost of the

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

4 IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

25

additional mail contacts.3 Savings estimates based solely on the 1995 census test are probably too optimistic, since the effect of any questionnaire improvements are likely to be less in 2000 than in the 1995 census test because the initial response rates will be higher. At higher response rates, every additional percentage point of improvement is more costly to achieve. There would also be considerable operational complexity in sending replacement questionnaires to nonresponding households over the entire United States, as would be required in 2000. If this constraint proves prohibitive, one possibility is to target only areas with low response rates for replacement questionnaires. These areas could be identified either in advance, based on 1990 block characteristics, or on early mail response rates in 2000. Another feature of the total design method is to take account of the general appearance and form of the survey questionnaire, as well as to follow guidelines on the wording and order of the questions themselves. These guidelines were followed in the development of the questionnaire for the 1995 census test. Analysis of the results showed some shortcomings of the questionnaire items, such as apparent confusion about the meaning of some coverage questions, whose purpose was to encourage respondents to assess whether they had correctly enumerated all household members. The Census Bureau is continuing to pursue research on questionnaire design, including research into improvement of the long form.4 Since research on questionnaire design is relatively inexpensive and has high potential payoff, we believe that this effort is well placed. In addition to research to design respondentfriendly forms, the Bureau has recently announced the transfer of five items from the short to the long form. This shortening of the form may act to increase mail return rates, although at the cost of reduced small-area data. MULTIPLE MODES FOR RESPONSE The plans for the 2000 census provide a number of new ways for respondents to be enumerated. The innovations were intended to reduce respondent burden and to improve coverage of traditionally undercounted groups. One of these changes is the introduction of the "Be Counted" forms: these forms are placed in public locations so that people who did not receive a census form or do not believe they were counted can easily report their information. This change in the census design was adopted from a suggestion presented to Congress in a report from the U.S. General Accounting Office (1992). The forms were available in the 1995 census test in five languages and at a

3

We do not have data to estimate the added costs, but they are likely to be considerably less than the estimated savings. The long form sent to 1 of 6 sample of households in 1990, collects additional data beyond the minimum required for reapportionment and redistricting. These data are of intense interest to many census users. 4

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

4 IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

26

information by telephone to a toll-free number. Individuals could also request by telephone a census form in any available language. Unlike the "Were You Counted" campaigns of the 1980 and 1990 censuses, which took place after nonresponse follow-up and did not include reporting by telephone, the "Be Counted" campaign is concurrent with the census. The 1995 census test suggests that the "Be Counted" forms did not achieve the goal of improving coverage of groups who have traditionally been undercounted in the census. The number of people responding by these forms was quite small, less than 1 percent of the 1990 population in all three 1995 census test sites: 0.8% for Oakland and Paterson, 0.5% for Louisiana. In addition, the hard-to-enumerate demographic subgroups--including people younger than 30 years old and blacks and Hispanics--tended to be underrepresented in the "Be Counted" population in comparison with their numbers in the 1990 census in those sites. Thus, as a method of reducing the differential undercount, the "Be Counted" campaign does not seem promising. However, most of the time a ''Be Counted" form was received, it was the only form received for that housing unit. Only 15.2 percent of the households with a person who sent in the form had another census form (mail or enumerator-returned) (Ammenhauser and Lucas, 1996). In addition, the public relations benefits from such a campaign might be substantial. For the 1995 census test, a wide variety of types of sites were used for distribution of the forms, including city halls, motor vehicle offices, and community centers. Only a few types of site resulted in a significant number of used forms. It seems that some reduction in cost could be realized by reducing the number of sites used for distribution of the "Be Counted" forms. The toll-free telephone number was an especially effective mode for collecting the "Be Counted" data in the 1995 census test. Over 42 percent of the people enumerated through the forms initiated the interview by telephone in the two urban test sites. In a related development, there is now discussion within the Census Bureau of allowing reporting electronically, through the Internet. Allowing response by Internet will surely not increase participation by historically undercounted groups, but it might also provide the benefit of improved public relations. There are some technical complexities of processing "Be Counted" forms that are not yet resolved and need further research (Ammenhauser and Lucas, 1996). One is that respondents are required to report on the form whether their response is for a whole household or a partial household. Exclusion of the address from the nonresponse followup universe is dependent on the answer to this question. However, given the difficulty that respondents appear to have with the concept of usual residence, as indicated in the analysis of the coverage question on the mail-back census form, it seems unlikely that the information received for this item is very accurate. Thus, this additional complexity in the processing procedure seems error prone. A simpler procedure would be to include either none or all of the addresses in the nonresponse follow-up universe: if none, it would leave unreported people in households with a "Be Counted" person to be estimated by integrated coverage measurement; if all, it would increase the nonresponse follow-up workload, but not by much because of the small number of forms. Either approach would require careful thought about sample design and estimation issues. For example, if the households are not included in the

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

4 IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

27

nonresponse follow-up universe, it might be advantageous if they constituted their own set of strata for integrated coverage measurement estimation. If all the households are included in the nonresponse follow-up universe, they might best be sampled at a different rate than other nonresponding households. Furthermore, auxiliary information from the "Be Counted" forms themselves--which would be available for both sampled and nonsampled households--could be used in estimation. To our knowledge, these issues have not yet been considered by the Census Bureau. Whenever more than one questionnaire is made available to a household, as will occur with either "Be Counted" forms or replacement questionnaires, there must be some means of ensuring that people who are counted more than once can be accurately "unduplicated." The 1995 census test showed that the number of duplicated responses resulting from replacement forms was small, and their unduplication was a feasible procedure (Ammenhauser, 1996; Hill and Leslie, 1996). In the actual census, there might be more motivation for people to try to inflate the count intentionally through use of multiple forms, especially the "Be Counted" forms. The Census Bureau should be prepared for this possibility. TARGETING FOR SPANISH-LANGUAGE QUESTIONNAIRES The Census Bureau has conducted two tests of the effectiveness of mailing Spanish-language questionnaires, in addition to English ones, to households in areas with high linguistic isolation in 1990, defined as areas in which more than 30 percent of the households include no one over 14 who speaks English very well or at all. The goal of mailing Spanish-language questionnaires to these areas was to increase the response rate and decrease the differential undercount. The Spanish Forms Availability Test, conducted in fall 1993, showed that offering a Spanish-language form increased the completion rate by an estimated 2-6 percentage points. However, the households that completed the Spanish questionnaire showed significantly higher item nonresponse rates than the households that completed an English questionnaire, thus appearing to reduce progress towards the stated goal of improving data quality. In addition, the majority of Hispanic households that returned a questionnaire returned an English form. The 1995 census test investigated the effect of sending two questionnaires, one in English and one in Spanish, to targeted areas in a census environment, where greater publicity may reduce the differential nonresponse rates. In this test, approximately 60 percent of the Hispanic households in the targeted areas returned Spanish questionnaires. This proportion remained constant whether the targeted area had a moderately high (between 15% and 30%) or high (more than 30%) level of linguistic isolation, based on 1990 data. Thus, the linguistic isolation variable does not appear to be a particularly good predictor of the usefulness of a Spanishlanguage questionnaire. This result could have occurred because the 1990 data were out of date by 1995, which suggests that they will be even less useful in 2000. The result is important because the Census Bureau is planning to expand the non-English forms distribution to include non-English languages in addition to Spanish.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

4 IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

28

Recommendation: The Census Bureau should monitor the use of Spanish-language and other nonEnglish language forms and the 800 telephone number during the 1998 dress rehearsal to refine criteria for forms distribution in 2000. TARGETED PROMOTIONAL CAMPAIGNS The Census Bureau is planning a major shift in the focus of its marketing plan for the 2000 census. The major change is that geographic areas and demographic subgroups will be targeted with different publicity and outreach campaigns, rather than the uniform approach used in previous censuses. The goal of this approach is to reduce the differential undercount. Another change is that the campaign will use paid advertising, rather than public service announcements, as in previous censuses. Since the Census Bureau lacks in-house expertise in this area, the plan is to contract out this program. Although this new approach seems reasonable, the panel has not seen evidence of its effectiveness. In fact, the only analysis undertaken to date has been through focus groups, whose results are mixed. It may not be possible to obtain experimental evidence of the effectiveness of the planned promotional campaign in advance of the census, but we believe it is important that measures of its success be specified and data collected to assess its performance for use in future censuses. Recommendation: The Census Bureau should establish measures and specify the data to be collected to assess and evaluate the performance of the various aspects of its promotional program. SERVICE-BASED ENUMERATION The Census Bureau has designed the service-based enumeration program to enumerate clients in emergency shelters and soup kitchens. This program replaces the Snight enumeration of the homeless, which was developed for the 1990 census in collaboration with experts in the field of homelessness. The central feature of the new program, in the 1995 census test, was to enumerate clients at all service locations. Efforts were first made to obtain service provider cooperation, and enumerators who were familiar with the service locations were used. Enumeration in this environment proved to have many difficulties, and the 1995 test identified a number of procedures that could be simplified. One of the more problematic aspects of the 1995 test program was the reinterview attempted at a sample of provider locations. The purpose of this reinterview was to provide data that would allow estimation of coverage, using a dual-system methodology. Respondents were confused about the reason that they were being reinterviewed and were therefore reluctant to complete the questionnaire again, which resulted in high refusal and item nonresponse rates. This in turn caused difficulty with matching to the initial questionnaire. Alternative methods of estimation of coverage were also attempted. One of these, known as multiplicity estimation, required data from only a single visit, but

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

4 IMPROVED SURVEY METHODS: MAKING IT EASIER TO RESPOND

29

relied on information provided by the respondent about how many days in the previous seven that services had been used. The methodology for the multiplicity estimator is similar to that of the Politz-Simmons estimator (Politz and Simmons, 1949, 1950). Both the dual-system estimator and the multiplicity estimator rely on several assumptions that are questionable in this application. These coverage measurement methods were examined further in the 1996 test, but results from it were not available for this report. Recommendation: The Census Bureau should continue research to determine the best means to enumerate people with no usual residence. This effort should be part of the Census Bureau's research agenda, with specific goals for both pre-2000 research and to learn from 2000 for the 2010 census.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

30

5 Sampling for Nonresponse Follow-Up: Achieving Adequate Precision at Acceptable Cost The 1990 census was conducted as a mail-out/mail-back operation with field follow-up to any address that did not return the census form by mail. Although the majority of households returned the form by mail, the most expensive part of the census was the follow-up for nonresponse: it accounted for about 20 percent of the $2.6 billion cost (over 10 years) (Bureau of the Census, 1992; U.S. General Accounting Office, 1992). In addition, completion of nonresponse follow-up took much longer than scheduled, delaying and potentially reducing the quality of the Post-Enumeration Survey (PES), designed to measure the differential undercount. For the 2000 census, the Census Bureau plans to use sampling to reduce costs, to improve management of field operations, and to speed completion of nonresponse follow-up. Chapter 2 of this report discusses this panel's support for the concept of using sampling for nonresponse follow-up. In this chapter we address the major design decisions that are required to implement it successfully. Although decisions about the nonresponse follow-up design need to take into account estimation plans, we defer most discussion of estimation to our final report. We defer discussion of integrated coverage measurement to Chapter 6, except to the extent that details of that process are needed here. ALTERNATIVE DESIGNS FOR NONRESPONSE FOLLOW-UP The Census Bureau will provide more ways to be counted in the 2000 census than ever before. Data will be collected about nongroup housing units by several modes:5 (1) initial census forms returned by mail; (2) return of replacement forms mailed to units that do not respond within 2 weeks of the initial mail-out or mailing of other forms (e.g., in a foreign language) requested by residents; (3) forms completed by telephone; (4) "Be Counted" forms with valid addresses; and (5) forms returned by the postmaster as undeliverable because the unit is vacant. Although modes (2) through (5) might all be considered forms of nonresponse follow-up, we refer to housing units enumerated by all five modes as initial respondents. For this discussion, all of those responses are excluded from the main nonresponse follow-up universe. (We discuss handling of postmaster returns below.) For units in the nonresponse follow-up universe, data collection modes will include: in-person enumeration during nonresponse follow-up; vacancies identified by field enumerators; and close-out procedures (e.g., interviews of neighbors) when

5 Plans for service-based enumeration aimed at counting people without a usual residence are discussed in Chapter 4 and are not discussed further in this chapter.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

31

attempts to contact residents are unsuccessful. In February 1996 the Census Bureau officially introduced its plan to use sampling for nonresponse follow-up in the 2000 census. That plan calls for a two-stage design. During stage 1, the Census Bureau would conduct follow-up by enumerators in the field to raise the response rate to at least 90 percent of housing units on the MAF in each census tract.6 Once the 90 percent mark has been reached, stage 2 follow-up would be conducted on a 1in-10 random sample of the remaining housing units. Characteristics for the remaining nonsampled housing units would be estimated on the basis of the random sample. Below, we describe more fully this design and two alternatives that the Census Bureau has considered. During stage 1 of nonresponse follow-up, the Census Bureau would begin personal visits to nonresponding housing units in the vast majority of tracts, much as in 1990. Field enumerators would visit housing units in the nonresponse follow-up universe, asking residents the questions on the census form or helping them to fill out the form. Residents who were initially mailed the long form would be asked to fill out the long form. Units where no one was contacted on the initial visit would be scheduled for revisits. In contrast to 1990, however, this stage of nonresponse follow-up would cease in a tract once the response rate (initial response plus this work) had reached 90 percent: that is, when a completed census form or confirmation that the dwelling was vacant had been obtained for 90 percent of the addresses to which a census form had been originally mailed. In a tract with a 90 percent or higher initial response rate--which would be rare— no stage 1 nonresponse follow-up would be conducted. Because the goal of stage 1 nonresponse follow-up is to reach 90 percent as quickly and inexpensively as possible, those responses would not necessarily represent the whole nonresponse follow-up universe. For example, the overall response for a tract could reach 90 percent before any follow-up had occurred in certain blocks. Also, stage 1 responses might tend to exclude housing units with difficult access or where residents were seldom at home. To allow unbiased estimation for the final 10 percent of housing units in each tract, the Census Bureau's announced plans for stage 2 of nonresponse follow-up include a 1-in-10 random sample of the remaining nonresponding housing units. Consequently, there would be direct enumerations for at least 91 percent of housing units in each tract at the end of stage 2 nonresponse follow-up. In contrast to stage 1, in stage 2 data would be collected for each sampled unit. Enumerators would make repeated visits to a unit-ideally at different times of the day and different days of the week--until a resident

6 The Census Bureau originally planned to use counties as the geographic unit for controlling completion of stage 1 nonresponse follow-up. That choice drew wide criticism from the Census Bureau's advisory groups and other interested parties: the concern was that many counties are so heterogeneous that a countywide response rate of 90 percent could be achieved while large areas of the county trailed far behind (White and Rust, 1996). In September 1996 the Census Bureau announced a revised plan to reach 90 percent in each tract. Because the country's 65,000 census tracts, which average 4,000 people and 1,500 housing units, are much smaller than the average county, the revised plan will greatly smooth out response rates at the completion of stage 1 nonresponse follow-up.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

32

responds or they determine that the unit is vacant. For units for which these visits do not yield a response before the final days of nonresponse follow-up, enumerators would conduct closeout procedures--interviewing neighbors to procure proxy information or estimating the number of residents from what they know about the neighborhood. We refer to this plan as the 90-percent truncation design. In this design, as well as others under consideration, characteristics of the nonsampled housing units would be estimated on the basis of the random sample. The Census Bureau is also considering two alternative plans for nonresponse follow-up (Killion, 1996b). Under the first alternative, time truncation, the stage 1 of nonresponse follow-up would be truncated at a given time. Instead of ending when the response rate reaches 90 percent in a census tract, this stage would last a specified period of time, such as 3 weeks, to allow making at least one visit to every housing unit in the nonresponse follow-up universe. The Census Bureau would then draw a large enough random sample of the remaining units to reach 90 percent response in each tract after completion of stage 2. In tracts that had already reached 90 percent or higher in stage 1, a 1-in-10 sample would be drawn. Under a second alternative, referred to as direct sampling to 90 percent, sampling for nonresponse follow-up would begin immediately after the mail-back period, with no stage 1. Nonresponse follow-up sampling rates would be set to achieve a final response rate of 90 percent in each census tract. For example, a 1-in-2 sampling rate for nonrespondents would be used in tracts with initial response rates of 80 percent, while a 5-in-6 sampling rate would be used in tracts with a 40 percent response rate. For all tracts with a initial response rate of 90 percent or higher, a 1-in-10 sampling rate would be used. Figure 1 shows a general schematic that covers all three design alternatives. "A" represents the percentage of housing units that respond by mail or are otherwise excluded from the nonresponse follow-up universe. The initial response rate (A) does not depend on the nonresponse follow-up design and will vary greatly from tract to tract. The Census Bureau has predicted an average initial response (excluding postmaster returns) of 67 percent across all tracts in the United States (Killion, 1996b), but with great variation across tracts. The percentage of notsampled housing units (D) would be approximately the same (9 or 10 percent) in each tract for each of the three designs under consideration.

FIGURE 1. Stages of enumeration. NRFU, nonresponse follow-up.

The designs differ mainly in the allocation of nonresponse follow-up between B and C, where B denotes the percentage of households for which data are collected in stage 1, and C denotes the percentage for which data are collected in stage 2. In the sampling

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

33

design in the 90 percent truncation design, almost all the effort is in the stage 1 part (B = 90 - A) and sets C at 1 percent of the population; the time truncation design is a compromise between B and C, with the relative effort varying among tracts. Under each of the design alternatives, the current assumption is that the Census Bureau will use characteristics and counts from the sampled housing units to estimate corresponding characteristics and counts for the nonsampled units. For example, under 90 percent truncation design, with a 1-in-10 sampling rate, data from each sampled household will be imputed to nine nonsampled housing units. This estimation method produces unbiased estimates of the results that would be achieved using complete nonresponse follow-up. Estimation for any of the designs will essentially result in weighting each sampled housing unit by the inverse of the sampling rate for the tract, with this sample weighting implemented by means of imputation. ASSESSMENT OF THE BASIC DESIGN ALTERNATIVES In this section we assess the above three nonresponse follow-up designs, and others, three using three main criteria: operational feasibility, cost, and size and equity of sampling variation. In addition, we consider how public reactions might differ for design variations. The choice of sampling unit (housing unit or block), a major design component, is discussed in the next section. Operational and Costs Issues Nonresponse follow-up is a massive logistical operation with few, if any, peacetime rivals. In order to complete nonresponse follow-up within a 6-week schedule, the Census Bureau estimates a need for 550 temporary offices with 260,000 to 300,000 enumerators (Bureau of the Census, 1996a, 1997). Although nonresponse follow-up was also scheduled for 6 weeks in 1990, only 72 percent of the workload was completed on time, and some district offices did not complete nonresponse follow-up until the 14th week (U.S. Government Accounting Office, 1992). Timely completion of nonresponse follow-up is critical in 2000 so that integrated coverage measurement can begin in time for production of reapportionment counts by December 31, 2000. In addition, as we discuss in Chapter 2, there is reason to expect that reducing the time to completion would improve the accuracy of enumerations during the latter stage of nonresponse follow-up; Ericksen et al. (1991) found that the rate of erroneous enumeration increased with time to completion in 1990. Despite the massive number of enumerators, nonresponse follow-up will need to be conducted in waves. Enumerators will receive their assignments in sets of about 40 units at the beginning of the nonresponse follow-up period; when an assignment is completed, or nearly so, the enumerator receives a new set. It is anticipated that the average completion time per set will be about 1-1/2 weeks, although times will vary by neighborhood characteristics, enumerator productivity, and enumerator work hours (part

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

34

time for most enumerators). Conducting nonresponse follow-up in waves allows for flexible assignment of workload without requiring enumerators to report daily. Timely communication between supervisors and their staffs is critical to the successful completion of the field work on time. There is no way to schedule the enumeration workload accurately. Despite the short length of the nonresponse follow-up operation, it will require a very large number of replacement enumerators in many areas: during the 1995 census test, the average enumerator tenure was just 2 weeks, including training time. Consequently, some areas are bound to fall behind schedule as enumerators cut back on their scheduled hours, complete a temporary employment period, or resign unexpectedly. If there is timely feedback on completion rates, there may be the opportunity to catch up by transferring staff from nearby areas that are ahead of schedule. While the logistical challenges will be great for any of the three nonresponse follow-up designs being considered, the two-stage designs face additional problems. Compared with direct sampling, the truncation designs place greater importance on timely communication for two reasons. First, reaching 90 percent response in all tracts during stage 1 may depend on not "overshooting" in some tracts. This requires up-to-date data on tract-level completion rates and the ability to modify enumerators' assignments quickly. Second, the use of a two-stage design increases the importance of not falling behind schedule in any area (in either stage 1 or stage 2) because there is little time to catch up. In initial discussions of the truncation designs, Census Bureau field managers anticipated that the transition from stage 1 to stage 2 of nonresponse follow-up would lead to a 1-week hiatus in enumeration operations, resulting in substantial staff loss, more than occurs during operations. The lost field time and staffing problems could make it impossible to complete the workload within the required schedule. On the basis of this anticipated break in operations, the Census Bureau estimates that the 90 percent truncation design would cost about $300 million more than direct sampling (total census costs over 10 years of $4.0 and $3.7 billion, respectively). Due to the need to visit every nonresponding housing unit, the Census Bureau estimates that the time truncation design would cost $700 million more than direct sampling ($4.4 billion total cost). The panel remains somewhat skeptical of the cost disadvantages attributed to the truncation designs, especially the time truncation design. In theory, those designs might be less costly than anticipated by the Census Bureau because they cover more of the easier-to-reach households in stage 1 and take as little as a 1-in-10 sample of the hardest-to-reach units. In contrast, direct sampling draws equally from easy- and hard-to-enumerate units in the nonresponse universe. Also, it might be possible to stagger the field operations to avoid down time of more than a couple days, if any, for most enumerators. As noted above, nonresponse follow-up enumeration needs to occur in waves. Consequently, stage 1 in some tracts could be completed while stage 1 in other nearby tracts continues or, perhaps, before it even begins. While stage 1 enumeration continues in "late" tracts, the Census Bureau could select stage 2 samples and distribute assignments for tracts (or parts of tracts) where stage 1 has been completed, eliminating the need for a break in field operations.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

35

Careful assignment of initial workloads would help to facilitate this process. Despite our skepticism on certain points, the panel still recognizes the substantial advantages of a onestage procedure, such as direct sampling, in terms of training, scheduling, and monitoring. Sampling Variability Associated with Alternative Designs Table 1 compares coefficients of variation (CV, see Chapter 2) of estimated population counts under three different nonresponse follow-up designs for a range of initial response rates. We use a constant tract size (1,500 housing units) and assume that the ratio of the standard deviation of household size to mean household size is a constant 0.67 during each stage of nonresponse follow-up.7 The table accounts only for variation due to nonresponse follow-up sampling. 8 Under these assumptions, the 90 percent truncation design (with 1-in-10 sampling of the final 10 percent) produces a constant coefficient of variation of 1.64 percent for all initial response rates up to 90 percent (first section of Table 1). For a tract with a population of 4,000, that translates into a standard error of 66 people, implying that there is a two-thirds probability that the estimated population falls between 3,934 and 4,066. In contrast, for the great majority of tracts that have initial response rates of 83 percent or less, direct sampling to 90 percent would cut the coefficient of variation by half or more. However, this benefit of direct sampling to 90 percent in each tract would not be realized in tracts with high response rates. In fact, that design would have the somewhat perverse property of producing the least accurate results in areas with initial response rates of close to 90 percent (the coefficient of variation would gradually fall for tracts that achieved rates above 90 percent). The final section of Table 1 presents a direct sampling design with sampling rates set to equalize the coefficient of variation among tracts of the same size, but with different initial response rates. The overall workload under this modified direct sampling design was chosen to match or fall slightly below that for direct sampling to 90 percent. The result is a uniform reduction in the coefficient of variation by a factor of 2.1 (a 4.5-fold reduction in the variance), compared with the 90 percent truncation plan. We do not include the time-truncation design in Table 1 because the results depend critically

7 The ratio of 0.67 is based on using the national distribution of household size (Bureau of the Census, 1990) and assuming that 10 percent of housing units are vacant at each stage of nonresponse follow-up. The correct value might be higher, to the extent that vacant units are more likely to fall into the nonresponse follow-up sample, or lower, to the extent that within-tract variation in household size is less than the overall variation. 8 We use the following formula: CV2 = 100 (D + D2/C) (0.67) 2 / n , where C and D are measured in percents (Thibaudeau and Navarro, 1995), and n = the number of HUs in a tract. If estimation for nonsampled units is done by imputation (weights restricted to integers), this formula slightly underestimates the coefficient of variation when the sampling rate, C/(C+D), is not the inverse of an integer.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

36

TABLE 1 Coefficient of Variation (CV) for Population Counts and Percentage of Housing Units Reached for an AverageSized Census Tract Under Three Alternative Designs, by Initial Response Rate 90% Truncation, 1-in-10 Sample of Last 10

Direct Sampling, to 90%

Direct Sampling, Equal CVs

Initial Response Rate (%)

Percent Reached

CV (in %)

Percent Reached

CV (in %)

Percent Reached

CV (in %)

40

91

1.64

90

.60

85.1

.77

50

91

1.64

90

.61

85.8

.77

60

91

1.64

90

.63

86.7

.77

70

91

1.64

90

.67

88.1

.77

75

91

1.64

90

.71

88.9

.77

80

91

1.64

90

.77

90.0

.77

85

91

1.64

90

.95

91.5

.77

90

91

1.64

91

1.64

93.4

.77

NOTES: The table assumes: a constant tract size of n = 1,500 housing units; a constant ratio of 0.67 for the standard deviation of household size divided by the mean household size; and sample data that are weighted by the inverse of the sampling rate.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

37

on response rates in stage 1, which are unknown. Results would tend to fall between the two extremes for 90 percent truncation and direct sampling to 90 percent. Because the computations for Table 1 rest on assumptions that cannot be verified until the 2000 census, the absolute levels of the coefficients of variation should be interpreted cautiously. However, we believe that relative comparisons among the designs are fairly accurate. Similar tables could be constructed for tracts of other sizes or for smaller geographic units, such as block groups or even individual blocks (assuming that 90 percent was reached uniformly within tracts under the truncation design). For smaller areas, the coefficients of variation would grow in inverse proportion to the square root of the number of housing units. Consequently, the relative advantages of the direct sampling designs would not change from those shown in Table 1. For larger geographic areas, such as congressional districts and states, sampling variability due to nonresponse follow-up sampling would shrink to 0.1 percent or less under direct sampling. At this level of geography, variability due to nonresponse followup would be negligible compared with sampling variability due to integrated coverage measurement--about 0.5 percent for most congressional districts (Killion, 1996b). Consequently, the choice among these three nonresponse follow-up designs will not significantly affect the precision of the population counts for those larger geographic units. The inefficiency of the 90 percent truncation design stems from the low sampling rate of 1 in 10, which means that each sampled unit gets counted (weighted) ten times. In contrast, no units gets counted more than about three times in the direct sampling design aimed at equalizing the coefficients of variation. We note that there are alternative estimation methods (e.g., ones that incorporate data from the initial or stage 1 respondents) for which the relative disadvantage of two-stage designs would not be so large. However, those methods would introduce further complexity or reliance on assumptions. The design alternatives also need to be evaluated in terms of how well they achieve equitable accuracy across areas. Of course, the first criterion is reduction of the differential undercount, which is why we believe that integrated coverage measurement is essential. However, we do not expect that sampling for nonresponse follow-up will have a significant impact on this goal (positive or negative), and we do not see any evidence that the design options differ in this regard. In contrast, the options do differ substantially in how well they equalize precision across tracts. We believe that the appropriate goal is that tracts of equal size have equal sampling variability (Thibaudeau and Navarro, 1995). Because the accuracy of the data is key, we believe that this measure of equity should prevail over others (such as equal final-response rate which penalizes areas with high initial response under direct sampling to 90 percent, or equal effort, which penalizes areas with low initial response). An argument could be made for a plan under which tracts with the highest initial response rates are sampled enough to produce the lowest sampling variability, in order to motivate greater mail response. However, until research efforts provide evidence that such a plan would improve the overall response, we are forced to assume that such a policy would not draw any less resistance than one that penalizes areas with high initial response. An exception to the equity goal might apply to very small political jurisdictions,

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

38

many of which are smaller than the tract size specified in Table 1. Because of the specific uses of the population counts for those jurisdictions, the Census Bureau might well choose to use higher sampling rates than for tracts in larger jurisdictions. Such a policy would be analogous to the higher rate of 1990 long-form sampling in jurisdictions with less than 2,500 population. The additional cost of this would be small compared with the overall nonresponse follow-up budget because the total population of these areas is fairly small. Other Criteria Under any nonresponse follow-up design, some housing units in the nonresponse sample will require closeout procedures after multiple visits fail to yield a response from a resident. It is difficult to know how much systematic or random error the closeouts add to the final estimates, but it is clear that closeouts are less desirable than direct responses. Although 90 percent truncation would probably require fewer closeouts than direct sampling, those cases would receive more weight under the former design and thus actually contribute as much (or more) of the final imputed data on which counts will be based. For example, consider a tract containing 50 housing units for which it was impossible to get a response under any design (the census day residents have moved or they just do not answer the door): one would expect to get about 5 of them in the 90 percent truncation sample and perhaps 20-30 with direct sampling, and they would receive approximately the same total weight in the final estimate. For the reasons noted above, direct sampling would reduce the sampling variability associated with those units. There is also some reason to expect that the weighted number of closeouts might be higher under the twostage designs. The likelihood of getting a response from a sampled unit increases with the number of knocks on the door and the time span between the first and last knock. The shorter sampling period under the truncation designs may increase the number of closeouts because of census day residents who move early in the nonresponse follow-up period. Also, if the two-stage designs are less efficient operationally, they might produce fewer visits to each sampled unit before the need to use closeout procedures. However, the larger concern is that some tracts will fall well short of the 90 percent goal at the scheduled end of stage 1 nonresponse follow-up. If the response rate in a tract is still only 60 or 70 percent, the Census Bureau will need to choose among arbitrarily closing out a large number of units, delaying the beginning of stage 2, or modifying the sampling plan design. Direct sampling also offers the advantage of improved control over where data are collected relative to that for truncation designs. Under the 90 percent truncation plan, response rates for different parts of a tract could differ greatly at the end of stage 1 even if the initial response rate was constant. Because enumerators will work on different days, the tractwide goal of 90-percent truncation before sampling could easily be reached by achieving 95 percent for the part of the tract with mainly single-family residences, but only 70 percent in a large apartment building. Although tract-level counts and characteristics would achieve the desired accuracy, results for apartment renters in the tract would have greater-than-average sampling error. Similarly, without adequate

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

39

controls, some enumerators might choose to improve their ''yield" by systematically avoiding units that were assigned the more time-consuming long form. In those areas, the accuracy of long-form data would suffer disproportionately. Under direct sampling, neither of these problems can occur because there is strict control of the units that provide nonresponse follow-up data. Another advantage of direct sampling is that it shortens the time period between census day and the beginning of the sampling operation, which reduces the number of errors due to mobility and some respondent memory problems. However, the Census Bureau anticipates public perception problems with any direct sampling design. Because public cooperation is vital to the success of the census, it is important to maintain public confidence that the Census Bureau is trying to count everyone and that the final counts can be believed. In focus groups conducted prior to the 1995 census test, residents of the test sites expressed particular concern about the use of sampling that begins before an effort to reach everyone but seemed more willing to accept sampling after response had already reached 90 percent (Research/Strategy/Management and Beldon & Russonello, 1995). The Census Bureau also anticipates opposition to any plan that fails to reach at least 90 percent of households in every tract (such as the third design in Table 1). The panel believes, however, that such focus group information may be largely anecdotal and that accurately predicting public perceptions or responses to specific sampling schemes is not yet possible (see below). Although public perception appears to favor the 90 percent truncation design, we caution against putting too much faith in that assessment. The focus groups described in Research/Strategy/Management and Beldon & Russonello (1995) covered eight questions, only two of which focused on sampling designs. Consequently, the negative reaction to direct sampling may be based on a superficial presentation of the options, and some people may distrust any use of sampling. However, we believe that 90 percent truncation would fare no better than direct sampling in public reactions if its drawbacks were made clear. Conclusions About Design Alternatives To place the various alternatives on equal footing, we have considered only designs with roughly comparable workloads. We see no reason that our conclusions in this section would not apply equally well for sets of alternatives that involved either a heavier or lighter workload. We believe that the best design involves direct sampling with sampling rates designed to equalize variances for tracts of equal size. Direct sampling is the most cost-effective use of resources during nonresponse follow-up. Relative to direct sampling, either truncation design being considered causes additional logistical challenges, is likely to increase costs, and would produce more variable estimates. Truncation might also produce greater inequity of precision among tracts. Although the discussion in this chapter concentrates on sampling at the census tract level, in Chapter 2 we point out the appropriateness of evaluating sampling error at geographic levels where counts have important legal, political, and financial implications.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

40

We note that the relative accuracy of alternative sampling schemes will hold at any geographic level. Recommendation: The Census Bureau should endorse direct sampling for nonresponse follow-up, using a design aimed at equalizing coefficients of variation for tracts of the same size. In comparing alternative designs, the evaluation of sampling error should take place at the geographic levels at which counts have important legal, political, and financial implications. The Census Bureau should begin immediately to educate the public and census users on the general merits of sampling for nonresponse follow-up and the specific merits of direct sampling. We recognize that a variety of concerns may mitigate against our recommended nonresponse follow-up design. For example, should negative public reaction to final response levels below 90 percent necessitate setting a floor such as 90 percent for all census tracts, modifying the direct sampling design by setting a minimum sampling rate of at least 1 in 4 or 1 in 3 would avoid the worst inefficiency and ensure roughly equal accuracy across the range of initial response rates. A similar modification could ameliorate the worst statistical aspect of time truncation. Finally, we note that 85 percent truncation followed by 1-in-3 sampling would produce a constant coefficient of variation of 0.95, 42 percent below that for 90 percent truncation. The key to all of these options is to avoid a very low sampling rate in any tract. Recommendation: If the Census Bureau needs to select a design alternative other than direct sampling with equal coefficients of variation, it is imperative that they avoid using a very low sampling rate, such as 1 in 10, in any tracts. OTHER DESIGN ISSUES Determining the Appropriate Amount of Nonresponse Follow-Up Based on negative reactions to previous alternatives that involved final response levels below 90 percent (A +B+C in Figure 1, above), the Census Bureau is considering only options with final response levels of at least 90 percent. It is not clear to the panel that that is the right target level. For a particular nonresponse follow-up design (e.g., direct sampling to equalize coefficient of variation) with an average final response level in the neighborhood of 90 percent, the exact level involves a policy tradeoff of costs and sampling variability: How much nonresponse follow-up is the public willing to pay for? How much sampling error is the public willing to accept? Although we cannot answer those two questions, we can offer a perspective for considering the latter question. No census is without error. Even if complete nonresponse follow-up is conducted, small-area counts and data on population characteristics will include errors due to misinterpretation of residency rules, incorrect information collected from proxy respondents, and other problems. A critical problem in specifying the amount of nonresponse follow-up to conduct

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

41

is that there is no way to accurately forecast the size of the follow-up population. In 1990 the mail response rate was 7 percentage points lower than it had been in 1980, ballooning the nonresponse follow-up universe by about 37 percent (Edmonston and Schultze, 1995). This unexpected increase in workload was a primary factor in the cost and schedule overruns of the 1990 census. While numerous innovations hold promise to improve the mail response rate in 2000 (not necessarily relative to the rate achieved in 1990, but relative to what would be achieved in 2000 without the innovations), it is not possible to predict what that rate will be. Plans that set a specific target for the final response rate risk repeating the problems of 1990 if the initial response rate is too far below what was forecast. With the reduced time in 2000 to complete nonresponse follow-up, this approach risks a large increase in the number of closeouts in some places and uneven accuracy in the final results. Thus, a better approach would be a flexible policy that sets the total nonresponse follow-up workload within realistic time limits and allows the final overall response rate to follow the overall initial response rate. Another risk to the successful completion of nonresponse follow-up is insufficient funding. If a certain workload is mandated but funding is not adequate to hire qualified staff for the time needed for multiple followups to find residents, then the quality of the census results will suffer significantly. While nonresponse follow-up sampling offers the prospect for improving the quality of that operation at reduced costs, an attempt to extract too much cost savings for a given reduction in workload would be counterproductive. Recommendation: Whatever plan for nonresponse follow-up is adopted, it is imperative that it be executed with an adequate budget. A modest nonresponse follow-up design (i.e., one with a relatively small workload) that is executed as planned is far superior to a more ambitious design that runs short of time or resources. Unit of Sampling In the 1995 census test the Census Bureau evaluated two distinct methods for selection of the samples immediately after completion of the initial response period: a housing-unit sample, in which nonresponding units were sampled randomly from each block (possibly none in some blocks), and a block sample, in which a 100 percent nonresponse follow-up effort was conducted in a random sample of blocks (with no nonresponse follow-up in nonsampled blocks). Direct sampling was used in the test, but the same issues would arise for truncation designs and for direct sampling (with various sampling rates). In the absence of integrated coverage measurement, the most efficient choice would probably be a housingunit sample. Consideration of a block sample arises from the need for complete nonresponse follow-up in integrated coverage measurement blocks (and, perhaps, in surrounding blocks). A housing-unit sample would require use of different procedures in the blocks that are in and not in integrated coverage measurement. Three factors need to be considered in choosing between the two types of samples:

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

42

1. The potential cost advantages of a block design: Enumerators might be more productive in carrying out a block design due to reduced travel time, less time spent reorienting themselves to addresses, and the opportunity to find multiple respondents at the same time. 2. The design effect of a block sample. Systematic differences among blocks within the same sampling stratum in terms of counts and other characteristics for nonresponse follow-up housing units (positive intrablock correlation) would imply larger sampling variances for a block sample than for a housingunit sample of the same size. The ratio of the variance for a block sample to that for a housing-unit sample is known as the design effect (Kish, 1965). 3. The potential for differential coverage in integrated coverage measurement blocks in comparison with other blocks with a unit-sample design: integrated coverage measurement estimation assumes that previous census operations occurred uniformly in both blocks alike. If instead, coverage in integrated coverage measurement blocks had differed systematically from that in non-integrated coverage measurement blocks, that fact would bias estimates from integrated coverage measurement. If systematic differences occurred and varied by geographic area or population group, that interaction would affect the integrated coverage measurement correction of the differential undercount. Use of a housing-unit sample in blocks not in integrated coverage measurement would introduce nonuniform procedures, increasing the opportunity for systematic differences between the two types of blocks. The 1995 census test provided empirical evidence on each of these factors. Based on that evidence, the Census Bureau has decided to use a housing-unit sample in blocks without integrated coverage measurement. There was no evidence of a cost difference under 90 percent truncation and very little difference under direct sampling (Treat, 1996a). Given the proximity of adjacent blocks, that finding seems quite understandable. The Census Bureau used 1990 data for the 1995 census test sites to estimate the relative size of sampling error that would result from block sampling versus unit sampling for nonresponse follow-up. For the two urban sites, the ratios were approximately 3 to 1 (Vacca, Mulry, and Killion, 1996), substantially larger than any of the ratios among designs considered in Table 1. These ratios are comparable to those found by Schindler (1993), using 1990 census data for Connecticut. These results provide strong reason to prefer a unit design for nonresponse follow-up, unless the integrated coverage measurement blocks (a block sample) would experience systematically different coverage during nonresponse follow-up. Using 1995 census test data, the Census Bureau compared results for the block and housing-unit samples for five nonresponse follow-up characteristics: average household size, vacancy rate, whole household substitution rate, last resort rate, and distribution of household size. No statistically significant main effects occurred at the level of alpha = 0.2 (two-sided tests). However, the limited size of the 1995 census test does not rule out the possibility that substantial systematic coverage differentials exist for one or more of these outcomes. The conclusion that coverage differentials do not exist requires additional evidence, for example, about the nature of the field operations. Indeed, Census Bureau personnel believe that the enumerators in the 1995 census

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

43

test did not know what type of block they were in; they simply received lists of addresses to visit and went to those addresses. However, the stakes will be much higher in 2000. Under 90 percent truncation, just 1 percent of housing units in blocks without integrated coverage measurement would be visited during the stage 2 nonresponse followup, so that a block where an enumerator visits several units may stand out as unusual. One step to reduce identification of integrated coverage measurement blocks (Census Bureau field personnel may have better suggestions) might be to divide nonresponse follow-up sampling among multiple enumerators in integrated coverage measurement blocks. We note that the likelihood that integrated coverage measurement blocks would be compromised is greatly reduced with direct sampling. In the procedures for the 2000 census, there will be statistical power to detect even relatively minor coverage-differential main effects or interactions that the 1995 test could not detect. Current plans call for approximately 25,000 integrated coverage measurement blocks and about 3 million housing-unit sample blocks (those without integrated coverage measurement)--compared with 544 block-sample blocks and 1,485 unit-sample blocks in 1995. Consequently, the standard errors for estimated differentials will be about one-eighth as large as they were in the 1995 census test. Recommendation: We support the decision to use the housing unit as the sampling unit during nonresponse follow-up. However, the Census Bureau should consider procedures to disguise the integrated coverage measurement blocks from enumerators, residents, and others. Follow-Up of Units with Postmaster Returns During the 1995 census test, 6 percent of prenotice letters and 7 percent of initial questionnaires across the two urban sites were returned by postmasters for being undeliverable as addressed. About two-thirds of the postmaster returns identified the unit as vacant, rather than nonvacant (e.g., nonexistent or bad address). In 1990, all postmaster returns of the census form (there was no prenotice letter) were visited to verify that the unit was either vacant or nonexistent. The official plans for 2000 call for follow-up visits to a 1-in-10 sample of postmaster returns identified as vacant and estimation for the other 90 percent, in an operation separate from the main nonresponse follow-up. Analysis of data from the 1995 census test (Green and Vazquez, 1996) provides valuable information on whether to use postmaster returns from the prenotice letter or the initial questionnaire to identify likely vacant or nonexistent units, when to follow up returns, and how large a sample to follow up. Of the initial questionnaires returned from units identified as vacant, integrated coverage measurement confirmed that 66 percent of those in Oakland and 59 percent in Paterson were indeed vacant on census day; however, 28 and 35 percent, respectively, were identified as occupied. Rates for the units from returns of the prenotice letter were slightly worse and those for nonvacant units were substantially worse. Given these findings, it makes sense to handle vacant units from postmaster returns separately from the main nonresponse follow-up to avoid the cost of mailing replacement questionnaires

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

44

and to reduce the time between census day and the follow-up operation. It is not as clear to us how to handle other types of postmaster returns. Is 10 percent the best proportion of returns to follow up? Optimal design theory says that, given fixed resources, the ratio of sampling rates for two strata should be proportional to the ratio of the within-strata standard deviations divided by the square root of the ratio of the per-unit sampling costs in the two strata (Cochran, 1977). We propose an average sampling rate that exceeds 50 percent for the main nonresponse follow-up population. A 10 percent rate for the postmaster returns would be too low unless those cases are more costly to follow up (the reverse seems likely) or variation among the returns is very small compared with that in the nonresponse follow-up population. The latter might be true if very few of the returns are actually occupied. However, at an occupancy rate of about 30 percent, the variance of household counts for vacant units identified by postmaster returns would be roughly the same as that for the main nonresponse follow-up sample. Recommendation: The Census Bureau should base the sampling rate for postmaster identified vacant units on optimal allocation of resources across all follow-up activities. Role of Administrative Records Prior to December 1996, Census Bureau plans called for administrative records (or combinations of records) to substitute for field enumeration for selected housing units. These records were to be taken at face value--that is, there would be no field validation of those records. Administrative records deemed inadequate for substitution would not be used at all in nonresponse follow-up. The Census Bureau had not developed or tested criteria for specifying which administrative records might be used in such a way. Consequently, the Bureau set an apparently conservative goal of substituting for just 5 percent of nonrespondents (about 1.7 percent of all housing units). In December 1996 the Bureau announced that it was dropping plans to use administrative records data in this way. The decision resulted from the fact that the necessary research into such use of administrative records was not sufficiently advanced to permit such use by 2000. The panel questions whether the proposed use of administrative records would ever prove effective, as the plan seems to err in both directions. Given the problems with even the best administrative records, none should be accepted as truth without some form of validation during the census. At the same time, the plan ignores the potential for substantial information in records that are not judged adequate for substitution (even the lack of an address for certain records carries information). A more promising approach would seem to be to use administrative records in conjunction with field validation for a sample of housing units. Administrative records could then be used as inputs for estimating the attributes of housing units with no field follow-up (Zanutto, 1996). We understand that use of administrative records in this way has not been precluded for the 2000 census; however, difficulties in matching households

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

SAMPLING FOR NONRESPONSE FOLLOW-UP: ACHIEVING ADEQUATE PRECISION AT ACCEPTABLE COST

from administrative records and the census (see Chapter 7) make it unlikely that such an approach can be used effectively in 2000. .

45

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

46

6 Integrated Coverage Measurement: Tackling the Differential Undercount The problem of undercoverage by the census has been known and studied during the last five censuses. Differential under coverage has always been considered particularly important because it affects distributive accuracy of the census--how benefits such as congressional representation and funds are divided among areas. Much of what we know about historic trends in undercoverage is based on demographic analysis, which is a useful tool for obtaining aggregated coverage estimates (Robinson et al., 1993). In such analysis, population estimates for age cohorts, by sex and by race, are produced by the basic accounting equation: current population = births - deaths + immigration - emigration.9

Historical series of demographic analyses show that overall census coverage increased from 1940 to 1980 and then decreased in 1990. But there has been persistent and systematic undercoverage of certain groups, including blacks in general and, particularly, black males from about 20 to 40 years old. In addition, the coverage gap between blacks and whites has not closed over time (Passel, 1991; Fay, Passell, and Robinson 1988; Citro and Cohen, 1985). Demographic analysis has only a limited ability to produce subnational population estimates because of little knowledge about internal migration patterns. Therefore, survey-based coverage measurement is essential to estimate undercoverage in smaller areas, such as states and cities. A Post-Enumeration Survey (PES) was conducted after the 1950 census (Marks, Mauldin, and Nisselson, 1953; Bureau of the Census, 1960), and it is remarkable how many of the design features and issues that are important today were confronted in that early work, although in a different social, organizational and technological environment. 10 The 1980 Post-Enumeration Program (Fay, Passell, and Robinson, 1988) was primarily based on the Current Population Survey, while the 1990 coverage measurement effort was based on a separate Post-Enumeration Survey (Hogan, 1992, 1993). The intention of the 1990 effort was to measure coverage and then decide after the census whether to use the results of the postcensal analysis in calculating the final

9 For demographic projections from a census to a postcensal year, the base-year population would also appear on the righthand side of the equation; in this application, however, each cohort is followed from its birth year so the base population is zero. 10 Similarities between the 1950 PES and the 1995 Census Plus, an alternative integrated coverage measurement methodology, are discussed below.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

47

population counts. This procedure placed the Census Bureau in the position of having to decide whether or not to use the analysis when its effects were already known. The director of the Census Bureau recommended using that analysis to adjust the 1990 census, based on a technical assessment that this would improve the overall accuracy of the census counts, but the Secretary of Commerce did not accept the recommendation. Much valuable experience was gained from this first full-scale attempt to implement coverage measurement and adjustment in the census. INCLUSION OF INTEGRATED COVERAGE MEASUREMENT IN THE FINAL CENSUS ENUMERATION Current plans for the 2000 census call for full integration of coverage measurement into the census process (Bureau of the Census, 1996a). Major decisions about methodology for measuring coverage of the initial census operations (mail-out/mailback returns, nonresponse follow-up, "Be Counted" forms, etc.) and for including the results of coverage measurement in the final counts will be made before the census. Therefore, there will be a single set of census numbers (a "one-number census") rather than a decision on which of two sets of numbers to use after the census. To emphasize the interdependence of all aspects of the census process, the coverage measurement component of the procedure is called ''integrated coverage measurement." The Census Bureau's intention is to do an adequate evaluation of the methodology before the census so that the quality of the integrated coverage measurement can be determined before the census and the decision can be made in advance on whether to incorporate its results into the final enumeration. The advantages of this approach include better timing (no need to allow time in the schedule for making a decision between the initial census results and the release of the final counts), more efficient and focused census operations (no need to pursue two tracks simultaneously), and greater acceptability (because the main decisions on methods are made before their consequences for particular groups or areas are known). Integrated coverage measurement subsumes both the data collection operations that measure coverage and the estimation procedures that bring the results into the enumeration. Integrated coverage measurement is intended to measure and correct differences in census undercoverage across fairly broad domains. These domains could, for example, be major geographic regions or states; urban, suburban, and rural parts of a geographic region; or subgroups by race and Hispanic origin. These are the levels at which systematic differences in coverage have been found to occur and to persist over time. Integrated coverage measurement is not intended to measure and correct relatively local differences in coverage, such as those occurring in particular localities (except, perhaps, in a few very large cities that are comparable in size to states); the costs and limitations of technical resources do not permit extremely detailed local corrections. Allocation of population to these very small domains depends on the mail-out/mail-back responses together with the nonresponse follow-up. However, integrated coverage measurement domains can be much more detailed geographically than those for which reliable demographic analysis estimates can be obtained. A general discussion of

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

48

integrated coverage measurement and its relationship to nonresponse follow-up can be found in a previous National Research Council report (Steffey and Bradburn, 1994). The purpose of the decennial census is to obtain as accurate a count of the U.S. population as possible. The existence and persistence of systematic biases in conventional census methodology cannot be ignored. Although we cannot interpret the precise meaning of the term "enumeration" in the Constitution, as statisticians and demographers we interpret the problem of "enumeration" as one of determining the numbers of people in various areas, and we endorse the application of methods that do as well as possible at this, including the correction of long-established biases. Of course, the accuracy of the new census methods must be subject to evaluation, as was the accuracy of the old methods. If the traditional method of the census tends to over count one group or area relative to another, the overcounted areas do not have a "right" to additional benefits, and they are not being penalized if integrated coverage measurement corrects the enumeration to bring it closer to the truth. It should be noted, however, that even with integrated coverage measurement, there is still an incentive for local efforts to improve the count because integrated coverage measurement measures coverage (and corrects bias) only over broad areas; within those areas, the smaller areas that work for a better count will benefit from it. Integrated coverage measurement can be expected to reduce enumeration errors for most small areas. Recommendation: The Census Bureau should continue on the path toward incorporating integrated coverage measurement in the 2000 census. ALTERNATIVE STRATEGIES Any method of integrated coverage measurement will have to accomplish three goals. First, it will have to estimate the ratio of the true count to the initial count across various domains. Second, the measurement process will have to allow for the fact that the initial count includes both underenumerations and omissions (people who should have been counted at a particular place but were not), as well as overenumerations and erroneous enumerations (people who should not have been counted at a particular place but were). Third, if the estimates are based on some kind of field operation (which seems to be necessary), then the measurement process will have to project from a sample of cases to the entire population in the domain. The Census Bureau is currently considering two integrated coverage measurement strategies: a PostEnumeration Survey (PES) with dual-system estimation (DSE) and Census-Plus. (See Steffey and Bradburn, 1994, for a more detailed discussion of these methodologies and the associated evaluation issues and estimation procedures.) Coverage measurement has a fairly long history at the Census Bureau, although the current effort is more ambitious than those in previous censuses. The PES/DSE strategy was used in the 1990 census, and a dualsystem estimation procedure (using the Current Population Survey as the second survey source, rather than a specially conducted postenumeration

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

49

survey) was used to evaluate the 1980 census. A version of Census-Plus was used in evaluation of the 1950 census; it was unsuccessful in obtaining the desired level of coverage. An approach considered under the name "Census-Plus" during the 1980s is actually more similar to the "Super Census" idea that was a design option earlier in the current planning cycle (see National Research Council, 1993). Hogan (1989a, 1989b) and Mulry (1993) discuss a number of coverage measurement methodologies including these, with historical references. Hogan discusses, in particular, the conclusions of evaluations of some of these methods after past censuses. Each of these strategies, PES/DSE or Census-Plus, incorporates both an approach to field data collection and an estimation procedure. The two strategies include a number of common features, including: (1) conducting operations in a sample of blocks in which nonresponse follow-up is also conducted for 100 percent of the nonresponding units; (2) creating an independent listing of housing units for those blocks (i.e., one that does not depend on the address lists used for the initial census); (3) using an interview that is run separately from the main census data collection to collect information about households in those blocks; (4) comparing the results of the interview to the initial census to identify errors in the initial census; and (5) using the results to estimate a ratio of true-to-initial census population. The main difference in the field operation has to do with the way that features (3) and (4) are implemented. In the PES/DSE strategy (which is similar in broad outlines to the approach used in 1990, with one important difference, noted below), the second (PES) interview is conducted after the census and is made as independent of the census as possible. The results of the interviews are compared ("matched") to the census roster, and all discrepancies are noted. It is then necessary to do follow-up interviews to "resolve" cases for which the census and PES disagree. People in the census who were not in the PES are resolved as either correct enumerations or erroneous enumerations. People who were in the PES but not in the census are resolved as either census omissions or erroneous counts in the PES: this will not necessarily require a follow-up interview, since the PES interview can often collect enough additional information to make resolution possible without further follow-up. A key difference between the PES plan tested in 1995 and the 1990 plan is that the survey in 1995 was directed at finding out who was in the sample blocks on census day (what has been called "PES-A") rather than those resident at the time of the PES ("PES-B"). This approach requires improved abilities of the Census Bureau to trace people who move (Anolik, 1996). An important motivation for trying this approach is that it is more consistent with what is required for Census-Plus, which was tested simultaneously with PES in 1995 and 1996. Furthermore, the PES-B approach cannot be applied exactly as in 1990 if there is also sampling for nonresponse follow-up. Because most in-movers will have come from blocks with incomplete nonresponse follow-up, it will be impossible to directly check the completeness of their enumeration for mail nonrespondents, who are not included in the nonresponse follow-up sample. This will make it necessary either to use PES-A or develop some new strategy for movers. In the Census-Plus strategy, the first part of the reinterview is independent, much like the PES interview; the second part is intended to carry out the resolution of

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

50

discrepancies between the two interviews ("reconciliation"). The integrated coverage measurement interviewer uses a computer-aided personal interview (CAPI) instrument, a laptop computer that guides the interviewer through the sequence of questions and records the responses. The CAPI instrument is loaded with the census roster for the household that is being interviewed. (It is interesting that the 1950 PES also involved providing the interviewer with a census roster, which was hidden during the first part of the interview and uncovered in the second part, although obviously at that time it involved paper forms rather than a computer.) Matching is carried out in the field by software in the machine, and the interviewer continues with a series of questions that produces a final "resolved roster." No further follow-up is required. A major operational advantage of Census-Plus, if it can be successfully implemented, is that field work can be completed relatively quickly because only one reinterview is required for each household. Both the Census-Plus and PES methodologies were tested in the 1995 census test, using the same set of integrated coverage measurement interviews. This strategy was possible because the first, independent, part of the Census-Plus interview is essentially equivalent to the PES interview. Other PES field operations (follow-up) could be added later, without interfering with the reconciliation part of the integrated coverage measurement interview. The estimation methods for the two approaches differ slightly (see Steffey and Bradburn, 1994:122-124). In principle, the statistical assumption underlying the dual-system estimation is that the PES is statistically independent of the census, in the sense that within defined population groups, the probability of being included in the PES is the same whether or not a person was included in the initial census; perfect coverage by the PES is not required. The statistical assumption underlying Census-Plus is that the resolved roster is the truth for the sample blocks, that is, that it has perfect coverage. In practice, neither of these theoretical assumptions is true, but the mathematical differences between the two estimators are not as great as their differences in field procedure. The real issue is not whether one or the other of the assumptions is perfectly correct, but the effect on the quality of the estimates when the assumptions deviate from the truth. Research conducted after the 1990 census (Bell, 1993) and still under way at the Census Bureau used demographic analysis to estimate how far the independence assumption of the dual-system estimation in the PES was from the truth and the possible effect of these deviations on estimates. In fact, the assumptions of the Census-Plus estimator are stronger than in the PES, since complete coverage by one source implies statistical independence. Dual-system estimation, in assuming only statistical independence, does not in principle require that the PES have perfect coverage or even that it has better coverage than the census. Moreover, even if the independence assumption fails, there is a good basis for believing that dual-system estimation at least moves the estimates for each population group in the right direction. This reasoning suggests that PES/DSE estimates may be more robust against failures of their assumptions than Census-Plus estimates, and this argument was made by Marks (in Krotki, 1978) in support of a dual-system estimation rather than a Census-Plus strategy. (Marks was writing in part in reaction to the experience of the 1950 PES, which used an estimator similar to Census-Plus.) Viewed another way, Census-Plus attempts to improve on dual

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

51

system estimation by using field procedures that force the independence assumptions to hold--by having perfect coverage in the PES. If Census-Plus fails to achieve this objective, there is a strong likelihood that its results will be more in error than those from the PES/DSE. RESULTS FROM THE 1995 CENSUS TEST At this writing, evaluation of the integrated coverage measurement methodologies must be based on experience in the 1995 census test, the only comparative study to date. Integrated coverage measurement was carried out in each of the three test sites: in general, all integrated coverage measurement operations were carried out as planned and on schedule, and noninterview rates were not excessive. The CAPI instruments worked as intended, although there were some difficulties with the details of the CAPI interview (discussed below). The operations provided a wealth of information on the interview process, and the operational successes give grounds for general optimism about the integrated coverage measurement process. Thirteen evaluation studies were conducted; an excellent summary of their conclusions is given in Vacca, Mulry, and Killion (1996). The Census Bureau should be congratulated for completion of these evaluations, which have been extremely helpful in assessing the integrated coverage measurement methodologies and developing improved approaches. In the remainder of this section we consider some of the results of these studies, focusing on those that address the critical decisions and issues noted in earlier National Research Council reports. Face Validity of the Methodologies The numerical results of integrated coverage measurement in the 1995 test were examined for their face validity in comparison with patterns that have been consistently observed in the past. The Census-Plus results did not seem reasonable, while the PES/DSE results were consistent with past patterns. For example, Census-Plus estimated that there was an overcount in 11 of 14 poststrata for blacks in Oakland and in 13 of 14 poststrata for blacks in Paterson; the PES estimated that there was an undercount in all of these poststrata (Mulry and Griffiths, 1996). Thus, the Census-Plus results do not agree with past results from both demographic estimation and survey data indicating that blacks are among the most undercounted groups. Research comparing the 1995 Census-Plus and PES/DSE results with benchmarks from demographic analysis indicates that the latter was more successful in estimating coverage error. For example, Robinson (1996) reported that the demographic analysis estimates of undercoverage for ages 0-14, based on birth, death, and migration records, were 16.7 percent for Oakland and 18.7 percent for Paterson. The respective dual-system estimates for ages 0-17 were 12.8 percent and 13.8 percent, and the respective Census-Plus estimates were 3.8 percent and 3.4 percent. In Louisiana, the demographic

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

52

analysis undercoverage estimate was 2.2 percent, which was very close to the Census-Plus estimate of 2.1 percent (dual-system estimation was not performed in this area). Robinson (1996) also examined sex ratios (number of males divided by number of females) by two age groups for blacks in Oakland and Paterson: Census-Plus lowered the estimated sex ratios from those obtained before integrated coverage measurement, while dual-system estimation raised them. The latter results were again more credible, since undercoverage is higher for black males than it is for black females. Kohn (1996) also examined sex ratios and found that the dual-system estimation results were much more consistent with demographically plausible sex ratios than were the Census-Plus results (although the dual-system estimation results were still too low). Operational Problems: Incomplete Rosters We have considered possible reasons for the shortfall of the Census-Plus estimator. An operational problem that appears to have contributed substantially to the implausible results for Census-Plus is that many of the integrated coverage measurement interviews were conducted either with an incomplete initial census roster in the CAPI instrument or without a census roster at all. This deficiency was a result of late nonresponse follow-up interviews and other late-arriving information (such as the "Be Counted" forms) and processing difficulties that kept the final census roster from being ready on time for the integrated coverage measurement interview. Approximately 22 percent of integrated coverage measurement interviews in Oakland were conducted without any census roster in the CAPI instrument; this is a rough estimate based on a restricted sample used in the evaluation interview, described below (West and Griffiths, 1996). The census roster was unavailable in some other cases, even if it had been loaded into the CAPI instrument, because the interviewer might go to the wrong unit, especially in a multi-unit structure. This might have occurred in structures without clearly marked separate mailboxes, because the original census questionnaire was delivered to a unit other than the one coded in the address or because definitions of the units was unclear, making it difficult to choose the correct one. The design of the CAPI instrument did not allow the interviewer, after the first part of the interview, to match the household to the census household to which it actually corresponded. Instead, the interviewer had to attempt a match of the household being interviewed to the census roster for the other household which had been mistakenly called up, making the census roster worse than useless. In Oakland, it appears that approximately 1.6 percent of the census cases were not matched to the integrated coverage measurement interview at the address given, but were known to be at a nearby address; another 2.7 percent were completely unknown at the given address and might or might not have been nearby (D. Childers, Census Bureau, private communication). An apparent consequence of not loading rosters and loading incomplete rosters in the CAPI instruments was that a large number of people who were in the census but not in the independent integrated coverage measurement interview were ultimately not

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

53

included in the integrated coverage measurement resolved roster; thus, they were in effect considered to have been erroneously enumerated by the census. This problem resulted in Census-Plus estimates that were too low. Presumably, had complete census rosters been in the interviewers' machines, many people not named in the independent integrated coverage measurement interview would have been identified as correctly enumerated by the census. Treat (1996b) reported on an evaluation study that used matching between data used in Census-Plus and data used in dual-system estimation for Oakland to estimate how many people were missed in Census-Plus that were ultimately included in the E sample (census enumerations) for the dual-system estimate. It was estimated that 32,365 would be missed by Census-Plus in housing units where there was a complete or partial integrated coverage measurement interview. (This estimate is considered an upper bound because a nonsystematic clerical review indicated that there were some false nonmatches.) To put this in perspective, the Census-Plus estimate for the total population was 334,493 (Vacca, Mulry, and Killion, 1996). Analyses of the missed people by length of residence, sex, race or Hispanic origin, and age suggest that renters, males, blacks, and young people were missed in higher proportions by Census-Plus than by the census. This finding may explain the poor results observed for the Census-Plus in traditionally undercounted groups. Further analysis by Treat (1996b) of the people estimated to have been missed by Census-Plus indicates that 60 percent were from housing units for which census rosters had been loaded into the integrated coverage measurement CAPI instruments; 40 percent were from housing units for which rosters had not been loaded. Thus, large numbers of missed people in Census-Plus are estimated to occur in both of these situations. Insight into these numbers is given by the integrated coverage measurement evaluation interview (West and Griffiths, 1996). A sample of integrated coverage measurement housing units in Oakland where there had been complete or partial interviews was reinterviewed by especially skilled interviewers using techniques similar to the original integrated coverage measurement interview, but with the input rosters in the CAPI instruments supplemented with the nonmatches from the original integrated coverage measurement interview. In housing units with at least one match between the initial census roster and the integrated coverage measurement independent roster, 3 percent of the people who were determined by the evaluation interview to be residents were not included on the integrated coverage measurement resolved roster. In housing units for which no census rosters were loaded into the integrated coverage measurement CAPI instruments, the corresponding figure was 12 percent; for those where census rosters were loaded but there were no matches, the figure was 11 percent. These findings suggest that the chance of a person being missed by Census-Plus is much lower in a housing unit with a census roster that corresponds to that person's household. Nevertheless, a large percentage of the estimated missed people were from this type of housing unit because such units occur more frequently than units with whole-household nonmatches or with no census rosters. In this study, the weighted number of people in housing units with at least one match is 74 percent of the total study population. These evaluations illustrate the absolute necessity of having the census rosters in

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

54

the CAPI instruments prior to the integrated coverage measurement interview and making these rosters as complete as possible. Further analyses of data from the 1995 tests and from future tests would enhance these evaluations, perhaps making it possible to predict how much the coverage of Census-Plus could be improved if the roster problem is resolved. For example, it might be feasible to compare the probabilities that a person is included in the integrated coverage measurement resolved roster under three different conditions: that there is no census roster (or a roster for a different household), that there is a roster but the person is not included, and that the person is in the roster. Operational Problems: The Integrated Coverage Measurement Interview An important issue for the validity of Census-Plus is whether there was a tendency to bias the results of the interview in such a way that the resolved roster agrees with the initial (independent) part of the integrated coverage measurement interview. This question implies that names that appeared on the census roster but not on the integrated coverage measurement independent roster would tend to be resolved as nonresident and therefore be left off the resolved roster, reducing the Census-Plus estimates. This phenomenon is called "reconciliation bias." Studies of a nonrandom set of interviews that had been tape recorded did not reveal evidence of overt bias, such as interviewers trying to convince respondents not to include a census person or deliberately going back and changing the independent integrated coverage measurement results to conform to the census (Bates and Good, 1996). However, the absence of overt bias does not exclude the possibility that there is a more subtle tendency, involving both the interviewer and the respondent, to make the results consistent with the initial integrated coverage measurement roster. Evaluation of the extent and effect of reconciliation bias is related to the issues raised above about the consequences of incomplete or missing rosters. Further analyses might show whether there is statistical dependence between inclusion of an individual in the census and the integrated coverage measurement resolved roster, given that the person is determined in the evaluation interview to be a correctly enumerated resident. A study by Biemer and Treat (1996) used latent class models applied to Oakland data to estimate the rates at which the original census, the integrated coverage measurement interview, and the evaluation interview correctly or incorrectly classified cases as resident or nonresident on census day. (Because the methodology is highly model dependent, the results must be treated with some caution.) The best-fitting models selected in this exercise had to allow for an interaction between each of the pairs of sources: even within a race stratum, inclusion in the original census enumeration was related to inclusion in the Census-Plus resolved roster. It is not possible to determine from this study, however, to what extent this relationship was due to the problems with rosters described above. An implication of this model is that the evaluation interview appeared to be virtually error-free in classification of cases as resident or not (excluding cases classified as unresolved), but the original roster and the integrated coverage measurement resolved roster both missed substantial numbers of residents. According to

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

55

this model, 7.8 percent of actual residents (among blacks, Hispanics, and Asians and Pacific Islanders) were omitted, or 7.3 percent of all cases considered, while 2.6 percent of all cases considered were erroneously included in the resolved roster; again, this supports the view that the resolved roster substantially underestimates population. Another feature of the integrated coverage measurement interview process is that during the reconciliation phase, the interviewer "probed" the respondent to obtain information to determine the residency status of people on either the census roster or the integrated coverage measurement roster (but not both), as well as to confirm the status of matching people. Qualitative study of this part of the integrated coverage measurement instrument revealed that the process for checking the roster was awkward and slow, tending to wear out respondents' patience. Furthermore, the probes were slow and repetitious. This caused problems in the interview, in which the interviewer tended to skip over the details of the probes, or the respondent rushed an answer without listening to all the alternatives. Moreover, in the 1995 test, the CAPI instrument had no provision to accommodate the absence of a census roster. Instead, the respondent had to explain for each household member individually that they had been included on a form that was not available to the interviewer (Bates and Good, 1996). Interviewers had a choice of codes to describe the reason for each nonmatch. West (1996) reported on a descriptive study of the frequencies with which the various codes were chosen in the three census sites. A few interviewers tended to be "outliers" in that each chose a specific code much more frequently than a large majority of interviewers. The most striking result was the frequency with which most interviewers chose the code "Other," in which cases the CAPI instruments prompted the interviewers to include notes. For people on the integrated coverage measurement roster who did not match anyone on the census roster, the percentage of cases in which this code was used was 88 in Oakland, 94 in Paterson, and 76 in Louisiana. For people on the census roster who did not match anyone on the integrated coverage measurement roster, the corresponding percentages were 30, 29, and 36. Use of the "Other" code resulted in a case being reviewed clerically to determine residency status. The clerks, when debriefed, remarked that information in the accompanying notes suggested the interviewer could have selected one of the other codes. They also noted that "Other'' was apparently used to explain whole-household nonmatches and housing units for which there was no census roster. Some other possible concerns with respect to the frequency in which various codes were used are discussed in West (1996) and Vacca, Mulry, and Killion (1996). Some suggest errors that could lead to classifying residents as unresolved; others seem to indicate confusion among respondents regarding census residency rules. More positively, evaluation studies of matching for dual-system estimation indicated that the information in the integrated coverage measurement interview was adequate to code P-sample cases (i.e., people who were listed in the initial integrated coverage measurement roster) as resident or nonresident, with a high degree of accuracy (especially for those who were resolved, given a definite status as either resident or nonresident). The information from the integrated coverage measurement interview was also very useful in determining whether E-sample cases (those listed in the census roster) were in fact resident, as long as an integrated coverage measurement interview was

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

56

obtained with the same household (Childers, 1996). Weighting Procedures Another possible factor contributing to error in the integrated coverage measurement was the weighting system used for households that were in the sample but not interviewed. This system did not make use of information from the census: it assumed that the noninterview households had the same proportional composition as all integrated coverage measurement households for which an integrated coverage measurement interview was obtained; it did not attempt to match them to integrated coverage measurement households whose census data was similar to the missed households. A follow-up study (Gbur, 1996) of a sample of noninterviewed households from Oakland suggests that these households tended to be smaller than those contacted during integrated coverage measurement and that the match rates of people in these households to people in the corresponding census rosters tended to be lower than the dual-system estimation match rates for the block clusters in which the households were located. Thus, addition of the noninterview follow-up data to the data used for Census-Plus and the PES/DSE lowered the Census-Plus adjustment factor from 1.005 to 0.978 but raised the PES/DSE adjustment factor from 1.0866 to 1.1081 (Vacca, Mulry, and Killion, 1996). We do not know what the effect on estimates would have been of a more conditional weighting scheme--one that made use of more information about the census characteristics of households that were not contacted. Assuming, however, that the census would have indicated that such households were smaller, we expect that the effect of a more conditional weighting scheme would also have been to lower the Census-Plus estimates of the undercount. Part of the significance of this finding is that it suggests that the Census-Plus field procedure did even worse than the estimates show because the estimates were helped by the fact that the nonresponse weighting procedure raised the Census-Plus estimates. Comparability of Blocks Because integrated coverage measurement is conducted only in a sample of blocks, one must be able to generalize from the sample blocks to the remaining blocks in the census. For this generalization to be valid, there should be no systematic differences in coverage between blocks in and not in the integrated coverage measurement. There are two reasons that the blocks could differ. First, the integrated coverage measurement blocks are a block sample for nonresponse follow-up, while it appears likely that outside of those blocks there will be a unit sample. Every mail-back nonrespondent address in the integrated coverage measurement sample blocks is followed up in the field during nonresponse follow-up. This procedure is necessary because the integrated coverage measurement interviewers work independently of the nonresponse follow-up

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

57

process and cannot distinguish between nonresponse follow-up sample and nonsample households in the same block. Second, the integrated coverage measurement process itself could have some effect on the census response rates. For example, the independent listing operation for integrated coverage measurement could sensitize residents of integrated coverage measurement blocks to the census, or the nonresponse follow-up and integrated coverage measurement interviewers could run into each other in the field. These interactions are referred to as "contamination" of the census by integrated coverage measurement and would only be a problem if integrated coverage measurement and nonresponse follow-up overlap in the same blocks. Both of these issues can be quantitatively assessed by comparing indicators of coverage in blocks from which a nonresponse follow-up unit sample was drawn, nonresponse follow-up blocks that were included in a block sample but not in integrated coverage measurement, and integrated coverage measurement blocks (which are also in the nonresponse follow-up block sample). Comparisons among groups of integrated coverage measurement blocks are difficult because their numbers are small, although it would be possible to perform such comparisons in an evaluation in the 1998 dress rehearsal and in the 2000 census itself, in which the integrated coverage measurement sample will be much larger than in the 1995 test. Comparisons involving nonresponse follow-up sample blocks that are not in the integrated coverage measurement will not be possible if the unit sample design is adopted. Griffiths (1996) compared both kinds of blocks in Oakland, paired by predicted nonresponse rates, and tested differences within four integrated coverage measurement sampling strata: blocks with high concentrations of blacks, of Hispanics, of Asians and Pacific Islanders, and all other blocks. Several outcomes were considered that described the census process, of which the mail response rate was of primary interest, as it is believed to have a strong relationship with inclusion probability. The results for mail response were inconsistent across strata (with two showing higher rates and two showing lower nonresponse rates for the integrated coverage measurement blocks), and in only one stratum were they even marginally statistically significant. No differences could be detected for other variables. Since one might expect the effects of contamination at least to go in the same direction in most strata, we conclude that this study found no evidence for such effects, although with the caveat that the data are consistent with a small effect on the order of a few percentage points. Treat (1996b) also compared the two kinds of blocks in Oakland, as well as the nonresponse follow-up block and unit sample panels. As with the other study, most differences were not statistically significant, and few consistent patterns were found, but the standard errors were large enough so that small but important effects could have eluded detection. Census managers who are familiar with the field operation have expressed a strong view that census results are affected neither by contamination nor by differential coverage by nonresponse follow-up sample design. Their view is based on an assessment that the design of the nonresponse follow-up makes it very unlikely that the interviewers would be able to distinguish whether their assignments come from a unit or a block sample.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

58

Comparability of integrated coverage measurement blocks with other blocks is one of the critical issues for the validity of integrated coverage measurement, and we regard the currently available evidence as favorable but not conclusive. The worst possible outcome would be if the Census Bureau proceeded on the assumption that there are no contamination effects or differences between blocks in or not in integrated coverage measurement and was later taken by surprise when evidence appeared for these effects in the census. Recommendation: Quantitative evaluation of the comparability of integrated coverage measurement blocks with other blocks should be built into the 1998 dress rehearsal and the 2000 census, and plans for the census should include a statistical correction for any important differences found. NEXT STEPS The Census Bureau plans two changes in the integrated coverage measurement process to correct the problems with the interview experienced in the 1995 census test, which most strongly affected the Census-Plus results. These changes are being implemented in the 1996 test, which is under way as this report is being written. First, the Bureau intends to speed up processing of census returns (from mailed questionnaires and nonresponse follow-up interviews) so that a more complete census roster will be available for the integrated coverage measurement interviews. Second, the Bureau has redesigned the CAPI instrument for the integrated coverage measurement interview so that it is less dependent on the judgment and skills of the interviewer and better adapted to situations that commonly arise. For example, it is now easier for a respondent to confirm the residency of the household on census day without going through each household member individually. The questions used for resolution of residency status are more structured and do not require open-ended coding by the interviewer. Each type of discrepancy between the initial integrated coverage measurement interview and the census roster is handled with a distinct set of questions tailored to that type of case, and there is also a set of questions designed for whole household nonmatches (arising when the interviewer has the wrong roster or no roster or when a family moved). Preliminary tests of this new CAPI instrument show that the interviewers followed the desired question sequence much more consistently than in the 1995 test, especially prior to the final section of the interview in which the status of unmatched people in the census roster was probed. In addition, undesirable responses were recorded at much lower rates (Bates, 1996). As this report was being prepared, the Census Bureau decided to drop the Census-Plus option from consideration and use PES/DSE methods for integrated coverage measurement in 2000. The panel congratulates the Census Bureau on making this timely decision. It is essential to select methodology that is fairly certain to work well enough in the 1998 dress rehearsal so that only minor refinements would be necessary for the 2000 census. (In fact, it is unlikely that the Census-Plus methodology could have been

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

59

conclusively supported by the results of the small 1996 test.) This Census-Plus versus PES/DSE decision should not prevent consideration of strategies that combine some of the best features of the two approaches. For example, the PES/DSE procedure could be combined with aspects of the 1995 interview structure. In this plan, the initial (independent) phase of the integrated coverage measurement interview would be regarded as a PES interview. The reconciliation phase of the integrated coverage measurement interview would be used to collect as much information as possible about the discrepant cases, but because a dual-system estimation procedure is used, estimation does not depend on the assumption that a correct and complete resolved roster can be created in the field. In particular, the integrated coverage measurement interviewer might attempt to determine whether individuals who were listed on the census roster but not in the independent reinterview were actually census day residents, and likewise whether individuals who were listed in the independent roster but not in the census were residents on census day. Although some additional field followup might be necessary, it should involve fewer cases than with traditional PES methodology and therefore be completed more quickly. However, it is not clear whether the benefits of this combined interview would outweigh the delays and processing costs associated with loading census information in the CAPI instrument or whether the quality of the information obtained for these cases would be adequate to make a further interview unnecessary for a large fraction of them. For example, one issue is that the integrated coverage measurement interview only searches for matches to the census within the same housing unit, while PES procedures (at least as implemented in 1990) typically search over a larger area, such as the entire block or the block together with surrounding blocks. The 1996 census test should shed further light on these questions. Recommendation: The Census Bureau should consider integrated coverage measurement strategies that combine features of the current Census-Plus and PES dual-system estimation approaches, rather than restricting its options to using one approach or the other in its present form. OTHER ISSUES AND RESEARCH The Census Bureau is developing a detailed research agenda (Killion, 1996a) for the 2000 census. We applaud this effort and believe that relatively small investments in research now can have a big payoff in a more efficient, accurate, and operationally smooth census in 2000. At this time we note a few issues that we regard as especially important in light of our current understanding of the integrated coverage measurement methodology.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

60

Total Error Model Much of the discussion above deals with some specific issues with the integrated coverage measurement process in the 1995 test. A more general issue concerns the difficulties that we encountered in summarizing the effects of the various problems on the estimates. The calculations that lead from the initial census and integrated coverage measurement data to the final population estimates are quite complex, and it was often difficult to determine the quantitative effect of a particular type of error (measured or hypothesized) on the estimates. One of the most useful products of the evaluation of the 1990 Post-Enumeration Survey was a total error analysis (Mulry and Spencer, 1993). This analysis listed all of the various types of error that could occur in the census and PES, summarized what was known about their magnitude, and then summarized their contribution to the overall bias and uncertainty of the estimates. We recognize that this approach is a difficult undertaking, but we believe that it would be useful in interpreting the census tests. Most important, it would be an essential part of the presentation of the results of the 2000 census, including the integrated coverage measurement. This type of systematic presentation will be important to provide convincing evidence of the overall accuracy of the onenumber census. Recommendation: The Census Bureau should prepare to carry out a total error analysis for the 2000 census. Use of Administrative Records One approach that has been considered for improving the completeness of the integrated coverage measurement roster is to collect names from administrative records pertaining to each integrated coverage measurement address, load them in the CAPI instrument, and then check in the field on whether they correspond to census day residents at that address. Preliminary studies suggest that a substantial number of names could be added by this method, but it raises serious and possibly insuperable concerns about confidentiality. A discussion of administrative records is in Chapter 7. Sample Design and Estimation Methods We have focused on issues concerning the operational aspects of integrated coverage measurement because the 1995 and 1996 census tests were small-scale efforts that resemble the decennial census at the micro level but not at the macro level of sample design and estimation procedures. Nonetheless, choices made at the macro level will have an important effect on the usefulness of integrated coverage measurement results. A number of these issues are discussed in a previous National Research Council report (Steffey and Bradburn, 1994), and we anticipate consideration of these issues in our final

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

INTEGRATED COVERAGE MEASUREMENT: TACKLING THE DIFFERENTIAL UNDERCOUNT

61

report. At this time we summarize briefly some of the important questions that must be dealt with in order to plan the decennial design. Sample design and estimation procedures are intimately related to each other because the overall size and allocation of the sample in part determines what kinds of estimates will be feasible, while the estimation procedures conversely determine what the requirements are for the sample. One important issue is whether state population estimates will all be direct estimates, that is, estimates that use only data collected within that state. If this constraint is placed on the estimation procedures, then the sample must be designed so that every state has a large enough sample size to support a direct estimate of acceptable accuracy. Current plans call for a sample size of approximately 750,000 households for integrated coverage measurement. This figure is calculated (Navarro, 1994) to make possible direct estimates with coefficients of variation of no more than 0.5 percent for states and for some important substate areas, such as major cities. The calculations appear in a series of internal Census Bureau memoranda from 1994-1995, which consider several different sample allocations with differing tradeoffs between the criteria of controlling the maximum standard deviation of the estimated population (which requires larger samples in the larger states) and controlling the maximum coefficient of variation (which requires roughly equal sample sizes in all states). We have not yet reviewed these issues closely, but we hope that the Census Bureau will prepare a more detailed discussion of sample allocation in time for the panel to give this closer attention in our final report. The decision to allocate the sample size to support direct estimates for every state implies that the sampling rate--the ratio of sample size to state population--will be much lower in large states than in small ones, leaving little sample available for differential adjustment for substate domains (e.g., geographical regions in large states or urban compared with suburban and rural areas). This decision should be based on appropriate cost estimates. We also look forward to research on other features of the estimation procedure, such as the use of indirect estimates for substate domains. Recommendation: The Census Bureau should perform the calculations necessary to clarify the effect of using direct state estimates on the sample sizes required for state estimates for the 2000 census and the consequences of these requirements for the accuracy of other estimates affected by integrated coverage measurement. Fielding a large survey as part of the census will be a major challenge for the Census Bureau, but there is reason to believe that it will be possible. First, the management structure for the survey would be similar to that for the 1990 PES, which was successfully implemented by the regional offices. Second, the number of temporary staff in 2000 will be relatively lower than was required to implement the 1995 census test integrated coverage measurement (since the test had a much denser sample than is planned for the 2000 census), and the training and CAPI instrument will be better developed than in 1995. Since it was possible to do the necessary recruitment in 1995 even in the areas that are typical of those in which it is hard to recruit skilled personnel, it should be possible to do so in 2000.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

62

7 Administrative Records: Looking to the Future

Despite formidable obstacles to their use, administrative records offer a tantalizing potential for reducing the costs and respondent burden associated with many of the operations involved in conducting a decennial census. Some have envisioned the possibility of an "administrative records census," perhaps as soon as 2010 (Edmonston and Schultze, 1995). In "The Plan for Census 2000," the Census Bureau (1996) announced its intention to use administrative records in the following ways: • to update the Master Address File; • to assist with the enumeration of special population groups (such as American Indians on reservations and in large cities, Alaskan Natives, people in group quarters, and people in remote areas); • to provide indirect responses for about 5 percent of the nonresponding households prior to nonresponse follow-up; • to augment the household rosters used in integrated coverage measurement interviews; and • to impute missing items on the long form. In parallel with the 2000 census activities, the Census Bureau planned to experiment with the development of a complete administrative records census in several sites. Finally, in addition to these announced uses, the Census Bureau has supported research on the use of administrative records as a component of the estimation procedure that will be used to impute census records for the 9-10 percent of households that do not return questionnaires and are not enumerated during nonresponse follow-up. In December 1996 the Census Bureau announced that administrative records would not be used to derive the census count for nonresponding households--the third item on the list above. It cited a number of reasons for this decision, including the need for much additional research. All of the other planned applications would proceed, however, and research on the quality and coverage of administrative records would continue. Previous National Research Council panels have recommended that the Census Bureau make greater use of administrative records in preparing for and conducting the census (National Research Council, 1993; Steffey and Bradburn, 1994; Edmonston and Schultze, 1995), and we, too, endorse the Bureau's efforts in that direction. Efforts by the Census Bureau in its administrative records research for the 2000 census have included acquiring numerous files of administrative records data from federal, state, local, and commercial sources; using these data to build an administrative records database for each of the three sites in the 1995 census test; evaluating the quality of these administrative data; and assessing the usefulness of administrative records in applications related to addressing census nonresponse and improving coverage. This

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

63

chapter reviews those efforts in terms of the planned uses of administrative records for the 2000 census. The first section discusses research on the development of an administrative records database, which is a prerequisite to most of the planned uses; the second section discusses each of the prospective applications. BUILDING AN ADMINISTRATIVE RECORDS DATABASE Using administrative records for census activities requires that the files obtained from different national, state, and local sources be combined into a single database. The challenge in building such a database is to reduce multiple files that overlap in their coverage and content to a single record with one name, one address, and one set of demographic characteristics for each covered person. This involves considerably more than just "unduplicating" records. It requires addressing the uncertainty created by the possible presence of multiple addresses, alternative variants of names, and perhaps even different data on age, sex, race, and Hispanic origin for the same individual. The construction of an administrative records database involves several distinct steps, which include the identification, requisition, and acquisition of files from their respective sources; the reformatting, standardization, geocoding, and unduplication of each source file; the pooling of the resulting files into a single database; and the reduction of this database to a single record per unit, whether that be housing units, people, or both. How well this is done will affect the overall quality of the database. Ideally, the database construction will improve on the accuracy of even the best source files, since no file is perfect. This outcome is not automatic, however, and it may be very difficult to achieve. Privacy and Confidentiality It is important to note that all information collected by the Bureau of the Census is confidential. It is protected by Title 13 of the U.S. Code, which both requires people to respond to the census and protects their privacy. No one outside the Census Bureau can see data collection instruments or use them to link individual data with names and addresses. This protection covers any and all information gathered from administrative records, as well as that collected in the field. All of the information collected for research purposes for the census is also protected. The Census Bureau may use the data only for statistical purposes and may release it only in a format that protects privacy and confidentiality. The Census Bureau enforces Title 13 very strictly, and it is the overriding reason that agencies that do not usually share information (such as the Internal Revenue Service and the Social Security Administration) have agreed to do so for the purposes of the census. Neither the Census Bureau nor any outside group, including this panel, has ever proposed that any unique identifier--such as Social Security numbers--be collected in any U.S. census. The Census Bureau has been and remains dedicated to keeping individual data confidential and private, and we believe it has diligently enforced Title 13.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

64

File Acquisition In preparation for the 1995 census test, the Census Bureau requested numerous administrative files for the geographic areas covered by the three test sites. These requests were made to government agencies at the federal, state, and local levels. The mere acquisition of these files proved to be a significant undertaking. Requests for some files--most significantly, those from the Aid for Dependent Children (AFDC) program--were denied. Other files that were supplied had to be recreated or documented more adequately before they could be used. One of the more promising files--the Information Returns Master File from the Internal Revenue Service (IRS)--could not be used within the time frame of the test. Nevertheless, the Census Bureau obtained and used most of the files that it requested, and it also purchased a number of commercial files (Neugebauer, 1996). Perhaps the magnitude of the operational difficulties that would be encountered in trying to use so many files from so many diverse sources should have been anticipated. The lessons learned were valuable, nonetheless. Despite the potential coverage improvement to be gained by adding more and more files, the effort expended in obtaining and working with so many files made clear that "more data" are not necessarily better data. For the potential coverage improvement that they provide, additional files multiply the problems associated with basic processing and unduplication, and they risk creating false additions. The prospect of confronting these same difficulties many times over was indeed a daunting one. In response to these lessons and the mixed findings from the initial ("stage I") evaluation, the Census Bureau identified a small number of files that were judged to provide nearly as much value as the entire set of files requested for the 1995 test but that would require substantially fewer resources to acquire and process. Using only these nine administrative files in each test site, the Census Bureau conducted a second ("stage II") evaluation of the data obtained from administrative records (Wurdeman, 1996)11. The construction of this second database and the findings from the evaluation are what we address below. Processing of Source Files Each of the administrative files was reformatted, as necessary, to provide person-level records in a common layout, and the data were subjected to name and address standardization, geocoding, and Social Security number verification (Wurdeman, 1996). Name and address standardization were needed to facilitate the unduplication of records, both within and across files (see below). Geocoding included attaching census geographic

11 For stage II, the subset of files obtained included: the Social Security Administration NUMIDENT file, the Internal Revenue Service Tax Year 1993 Master File; two files from the Department of Housing and Urban Development--MultiFamily Tenant Certification System (MTCS) and Tenant Rental Assistance Certification System (TRACS); food stamps; Medicare; drivers' license; school enrollment; voter registration; and parolees/probationers.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

65

codes to each record and assigning a housing unit identification number, to be used in aggregating the personlevel records into households. Records with Social Security numbers were matched to the Social Security Administration (SSA) NUMIDENT file to verify that the numbers had indeed been issued to the individuals named in the records. Geocoding was limited to street addresses. A post office box could not be geocoded. A record with a post office box listed as an address was retained in the file, however, in the event that the record could be matched to another record that included the missing street address but lacked complete or correct information on other characteristics. Social Security number verification provided an opportunity to obtain age, sex, race, and Hispanic origin data if they were not present on the source file. All of these variables are reported on the NUMIDENT file, albeit with an important limitation: Hispanic origin is not recorded separately from race but, rather, as one of the possible values of race. For people identifying themselves as Hispanic, then, the race variable must be treated as missing or assigned the value ''other." It appears that the Census Bureau may have done the latter, which would contribute to the measured disagreement between the constructed database and the census data collected at the test sites. Combining Files The multiple files were merged and sorted by household unit identifications to create a preliminary household file. Person-level unduplication, which was done first within and then across households, reduced the database to one record per person. In the course of this unduplication, if a variable was found to be reported with two or more different values for the same person (addresses and spelling of names having the most discrepancies), only one value was selected to represent that variable. Except when data were missing, incomplete, or could not be coded, the Census Bureau assigned the value from the source file that carried the highest priority, according to a scheme established on the basis of empirical findings (see below) and other considerations. Other than assigning a "conflict code" when discrepant values of certain variables were observed, the Census Bureau did not preserve in the final database any of the alternative values recorded in the source files. One consequence of person-level unduplication was that some reported addresses were eliminated from the database. Quite obviously, the way in which an administrative records database is constructed can have major effects on the quality of the information it contains. For this reason, we contend that an administrative database that is built for research purposes should be designed and constructed in such a way that it can be used to evaluate alternative approaches to combining the information from the source files. While the Census Bureau's strategy of obtaining a single value for each variable is probably the only realistic approach for the 2000 census, the discrepant values represent information about uncertainty, and there may be ways to capitalize on this feature of administrative records in the future. For current analytic tasks, the file obtained by the strategy of assigning one value rather than multiple values is very useful. However, not preserving the multiple values in the final database limits its usefulness for research on improving its

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

66

construction. To test another variant of the choice algorithm, or to assess the consequences of not using one or more of the source files, a researcher must create an entirely new database. Recommendation: The Census Bureau should build an administrative database that preserves the values of important variables from more than one source file to allow flexibility in evaluating alternative matching rules. Evaluation of the Administrative Data The Census Bureau's evaluation of the administrative records data included separate assessments of the quality of data reported in the individual source files and in the merged database. The evaluation relied on a comparison of the administrative records data with the census test data collected at the three sites. However, differences in coverage between the census test and the administrative records assembled for the evaluation make it difficult to draw simple conclusions from these comparisons. Before reviewing the findings, we outline some of these differences and their implications. The 1995 census test was not a complete enumeration of any of the three test sites. The Census Bureau enumerated only a sample of the households that did not self-report by mail or telephone (i.e., the nonresponse follow-up households). The housing units that were enumerated by mail or in the field or identified as vacant constituted 59 percent of the total units in Paterson, 71 percent in Oakland, and 55 percent in Louisiana.12 Other factors complicate the evaluation of administrative records even further. The requests for administrative files failed to identify four zip codes from the Oakland test site and one from Paterson. The omitted Paterson zip code contained only four housing units, but the omitted Oakland zip codes contained 8,000 units. Having no administrative records for these areas, the Census Bureau excluded them from its matching so that they would not be counted as unmatched cases. However, the Census Bureau did not exclude the administrative records that could not be matched because the households to which they referred were not enumerated. In addition, one or more of the sites included only parts of some zip codes. It does not appear that the Census Bureau attempted to remove from the administrative records database any addresses that fell into those portions of zip codes that were not included in the test sites. We presume, however, that if an agency submitted a file that included any zip codes that were entirely outside the test sites, the Bureau would have first removed those zip codes before processing and analyzing these records. Finally, some of the administrative files-particularly the local ones--appear to be incomplete. For example, the voter registration file in Oakland contained fewer than 8 percent as many records as the IRS file, while it

12 These fractions were calculated from tabulations reported by Wurdeman (1996). To the panel's knowledge, neither in this nor in any other report has the Census Bureau reported the percentages that were ultimately enumerated or identified as vacant, so their accuracy is uncertain.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

67

was 50-75 percent as large as the IRS files at the other two sites. For some purposes, one would like to use the total number of households or people in the administrative records database or individual files as denominators for rates. In some cases, the Census Bureau calculated such rates. Because of the broader coverage of the administrative files, however, such calculations understate the relative frequency with which administrative records could be matched to census records. Without an accompanying explanation, these match rates may be misinterpreted as evidence of poor quality in the administrative records. Quality of the Individual Source Files To evaluate the quality of the short-form items on different source files, the Census Bureau matched person records from each source file to the census database in each of the test sites. This was done prior to combining the source files, and the results contributed to the development of the priority scheme used in combining the files. Person records were matched on the basis of name, age, and sex. Using only complete matches, so as not to bias the evaluation with even a small number of mismatches, the Census Bureau then compared the address, race, and Hispanic origin on the administrative files with the values collected during the census test. The proportion of people in the administrative files could be matched to census records is lowered not only by the restriction to complete matches on name, age, and sex, but also by the fact that the census test included only a sample enumeration of the nonresponse follow-up cases. The match rates, which were rather consistent across the three sites, ranged from 40 to 60 percent for most of the administrative source files: Medicare records had the highest match rate in every site, ranging from 53 to 61 percent; IRS records were generally second, with match rates between 40 and 49 percent. Given the exclusion of much of the nonresponse follow-up caseload, comparisons of the source files with respect to match rates may be misleading. There is ample evidence that nonresponse follow-up households are quite different from households that respond to the questionnaire or otherwise self-report. Therefore, source files with large numbers of nonresponse follow-up households or people would have artificially conservative match rates. Among matched person records, the Census Bureau found large differences across the sites and across the source files with respect to the quality of the data on address, race, and Hispanic origin. Address Because of their purposes, administrative records may include addresses that represent any number of things other than residences as defined by the Census Bureau, such as post office boxes or other types of mailing addresses, business addresses, or even addresses of tax preparers or accountants. They may include misspellings and other errors or lack unit numbers, which are often not needed for mail delivery. They may

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

68

correspond to where a person wants to appear to live rather than actually lives. Finally, they may be out of date. The relative frequency of these problems differ by source file, as administrative agencies use addresses in different ways and have different mechanisms and schedules for updating them. In evaluating the addresses on administrative records, one can ask the following questions about the addresses per se. Are they street addresses as opposed to post office boxes or delivery addresses? Are they residential? Do they correctly identify a street and number? Do they include unit numbers if required? If the addresses are good by all of these standards, one can then ask if they correctly identify the census day residences of the people with whom they are identified. The Census Bureau analyzed address data for the six major source files (Medicare, IRS tax returns, food stamps, voter registration, driver's license, and school enrollment). In Paterson, the percentage of matched administrative and census people with the same basic street address (that is, they could be located within the same building although not necessarily the same housing unit) ranged between 38 and 79 percent, with school enrollment the lowest and Medicare the highest. In Oakland, the frequency of agreement on basic street address varied between 14 and 53 percent; in Louisiana, the range was only 8 to 19 percent. The very low rates of agreement in Louisiana reflect the preponderance of rural addresses in that test site. In rural Louisiana the mailing addresses used in the administrative files--and by the postal service, for that matter--differ from the street addresses used by the census. In the other two sites, the rates of agreement indicate that there are problems with the addresses in addition to their timeliness. Nationally, about one in five housing units turns over during a 12month period. Except for Medicare data in Paterson, however, all of the files disagreed with the census address for substantially more than 20 percent of households. Two records may agree on a basic street address, but if the address happens to be a multi-unit building, the two records may differ in their unit numbers, or one may not have a unit number. In Paterson, for matched people with street addresses, when addresses included unit numbers in one or the other file (census or administrative), the unit numbers agreed only 28 percent of the time between the census data and the best administrative file. At the opposite extreme, the food stamp file in Paterson appears to have no unit numbers, for there were no matches to the census unit numbers. In Oakland, the addresses in multi-unit buildings agreed on the unit number between 67 and 85 percent of the time, depending on the administrative file. In Louisiana, agreement on unit number ranged from 17 to 70 percent across the administrative files. Clearly, there is enormous variability across files and sites in the quality of the unit numbers. In Oakland and Louisiana some of the files have good if not perfect reporting of unit numbers, but in Paterson none of the files contains very good unit identification. To explain the low match rates on unit number in Paterson, where multi-unit buildings are the dominant mode of housing (greatly compounding the problem), the Census Bureau noted that nearly two-thirds of the multi-unit buildings in Paterson contained only two to four units, while only one-third of the multi-unit buildings in Oakland were this small (Wurdeman, 1996). Such small buildings are more likely to have nonstandard unit identifications than larger buildings. In addition, residents may be less likely to report unit designations if the mail is delivered without them.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

69

Understanding the reason for the relatively low match rates in Paterson does not provide a solution, however. What it does provide is a basis for predicting comparably low match rates between administrative addresses and census addresses in other communities with similar housing stock. The problems encountered with administrative addresses in Louisiana attest to a different type of limitation of administrative record addresses, namely, that census-type addresses are not useful for most administrative purposes when the Postal Service uses an entirely different set of addresses for mail delivery. The only administrative databases that are likely to help with this problem are those that require identification of housing units per se. Utility companies would need such identification and would very likely maintain both property and mailing addresses--the latter for billing purposes. Whether addresses from this source are ultimately usable remains to be established. What is clear is that national and even state files are not going to provide census-type addresses of uniform quality across all geographic areas and types of communities. This implies a need to supplement the major national and state files if the use of property addresses for census enumeration is to be maintained. We suggest that the Census Bureau conduct research to explore such supplementation--perhaps even starting with some of the databases that were acquired for the 1995 census test but excluded from the stage II evaluation. Race and Hispanic Origin The Census Bureau examined both the presence of a race or Hispanic origin entry and how well it compared with what was reported in the census. Medicare files were the only files on which race and Hispanic origin were virtually always present (99 percent of the time at all three sites). For the IRS files, for which race and Hispanic origin were obtained by matching Social Security numbers to SSA records, race entries were present on 75 and 79 percent of the records in Oakland and Louisiana, respectively, but only 56 percent of the records in Paterson. The figures for Hispanic origin were comparable, except that Paterson had entries for 75 percent of the records. The other large files had race and Hispanic origin entries that ranged from 0 to 100 percent of their records, with the state and local files varying widely across the three sites. When race was reported on both an administrative record and a matched census record, the rate of agreement generally varied between 80 and 93 percent, with the files in Louisiana being at the high end and those in Oakland at the low end. Clearly, the racial and Hispanic composition of the local population was a factor, with match rates being lower when Asians and Hispanic people were relatively numerous. This relationship was attributed to how respondents used the "other" race category in both the administrative records and the census. In addition, Social Security card applications prior to 1980 included only "white," "black," and "other'' as options. To the extent that the administrative measure of race and Hispanic origin was obtained from SSA records, the agreement between census and administrative files would be expected to decline as the proportion of the population that was Hispanic or neither black nor white increased. Agreement between the census and administrative files with respect to Hispanic origin

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

70

was comparable to what was observed for race, for site and source file differences as well as overall magnitudes. Data on race and Hispanic origin are sometimes missing from SSA records. An earlier panel reported that race was missing from about 15 percent of the Social Security numbers issued between 1980 and 1991 and between 1 and 3 percent of the numbers issued earlier (Edmonston and Schultze, 1995). Race is now being obtained for infants who are issued numbers on the basis of information provided on the birth record, a mechanism that was established in the early 1990s and that now encompasses most newborns. However, deficiencies of the SSA records can account for no more than one-half of the missing race entries on IRS records in Oakland and perhaps only one-quarter of the missing entries in Paterson. Problems in matching taxpayers' Social Security numbers to SSA records must account for the balance of the missing race data, but this is inconsistent with the fact that fewer than 1 percent of the numbers that taxpayers report on returns fail IRS validation. Because of the importance that will be assigned to SSA records as a demographic supplement to other administrative records, the match rates obtained in the test sites bear further review. We believe the Census Bureau should include this on its administrative records research agenda. Recommendation: The Census Bureau should give further review to the Social Security number match rates from the 1995 census test. Discovering the reasons for apparent failure to match Social Security Administration (SSA) records will provide valuable guidance and ensure the utility of the SSA data in the administrative records database. Quality of the Database The findings reported by the Census Bureau do not address the quality of the administrative records--only the extent of their agreement with census values. To assess the comparative quality of administrative and census data would require revisiting a sample of households for which both were reported. In our view, this is not a short-term need, but something that should be considered for the evaluation of administrative records that is conducted in conjunction with the 2000 census. In combining the multiple source files, eliminating duplicates, and selecting among alternative reports of addresses and other characteristics, the Census Bureau could obtain data that are of higher quality than those in most, if not all, of the individual source files. For example, missing unit numbers could be filled in from addresses obtained from lower priority files. Likewise, post office box numbers could be replaced by actual street addresses. But if an old or otherwise inaccurate basic street address is reported on the highest priority file, this address would be retained. In the absence of additional information, there is no mechanism for determining that an address reported on a lower priority file is more correct than an address reported on a higher priority file. Similarly, incomplete reports of race or Hispanic origin can be replaced by more complete reports found on other files, but procedures for determining the most accurate information have not been developed.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

71

The Census Bureau's evaluation of the merged administrative records database did not include a replication of the methodology used in the evaluation of the separate source files. Matching between the administrative database and the census file was done first at the household level and then at the person level. Match rates of administrative addresses to census addresses were calculated with the census counts as denominators. This strategy compensates for the incompleteness of the census file, but match rates that are calculated with census numbers as denominators express the coverage of the database, not the quality of the data. For Paterson, 49 percent of the self-reported (mail-back) households and 37 percent of the nonresponse follow-up enumerated households matched to addresses in the database. For Oakland, these figures were 90 percent and 83 percent, respectively. For Louisiana, the match rates were 80 percent and 73 percent, respectively. Since the highest match rate to census addresses by any single source file in Louisiana was only 19 percent (see above), these latter rates suggest that the total number of administrative addresses in Louisiana must have been several times the number of census records. The results in Paterson continue to reflect the lack of consistent unit numbers on addresses in multi-unit buildings. Recommendation: Research aimed at developing more effective strategies for combining administrative data files should be a priority item in the Census Bureau's research agenda on administrative records. This work should include direct measurement of the quality as well as the coverage of addresses in the polled files. USES OF ADMINISTRATIVE RECORDS FOR 2000 With the above discussion of the development of an administrative records database as background, we review the Census Bureau's progress on seven possible uses of administrative records for the 2000 census. Master Address File Updating As Chapter 3 describes, the development of an up-to-date national address file through a listing operation that occurs in the year prior to the census has been replaced by the maintenance of a continuously updated Master Address File (MAF). The MAF will serve as the sampling frame for a number of Census Bureau surveys in addition to providing the accurate address list for the decennial census. To exploit an existing, high-quality source of information for updating the MAF, the Census Bureau has developed a partnership with the U.S. Postal Service (USPS) (see Chapter 3). Census Bureau plans do not include the use of any other source of administrative records to update the MAF.13 While the use of administrative records might help to

13

Historically, the Census Bureau has not defined USPS databases as administrative records.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

72

reduce one component of the undercount (housing units not listed correctly in the MAF), we acknowledge that the state of research on the development of administrative records databases is not yet sufficiently advanced to make such use efficient. It is feasible to screen only a small number of new addresses. Earlier research suggested that the administrative databases created for the three test sites contained substantially more unique addresses than the total number in the census MAFs; it is unclear to what extent this is still true for more recent research conducted with improved administrative databases. We encourage the Census Bureau to do some exploratory research aimed at understanding what the excess addresses represent. Enumeration of Special Population Groups Special population groups are so designated because their enumeration requires different procedures than those used with the rest of the population. The Bureau hopes that administrative records will be able to replace a number of advance listing operations used with special populations and will be able to provide data on people residing in groups quarters or other special housing. To date, however, the panel has received very little information on the development of plans and research for such applications. The 1996 Community Census will include an evaluation of procedures to enumerate residents of group quarters, as well as American Indians living on reservations (Whitford, 1996). The panel has not yet been briefed on the Bureau's plans for such uses of administrative records, and therefore is not certain how the research that is being conducted as part of the 1996 community census will further the Bureau's goals. Reduction of the Nonresponse Follow-Up Workload The proposed use of administrative records as a replacement for a portion of nonresponse follow-up was the subject of extensive research in the 1995 census test. A reevaluation of the 1995 findings was released in late December 1996. Further research is planned, following the 1996 community census. The Bureau's recently announced decision not to use administrative records for this purpose in the 2000 census has significant implications for the research agenda. To evaluate the potential for administrative records to be used to impute short-form items to nonresponding households prior to nonresponse follow-up enumeration, the Census Bureau matched the records from the administrative records database to the census data capture file in each site. For self-reported households--that is, those that returned census questionnaires--49 percent of the census households in Paterson, 90 percent of those in Oakland, and 80 percent of those in Louisiana could be matched to a database record on the basis of complete address. For the sample of nonresponse followup enumerated households, these match rates were 37 percent, 83 percent, and 72

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

73

percent, respectively.14 The match rates for nonresponse follow-up households were consistently lower but not dramatically so. To determine how often a database household would yield a correct imputation for a census household, the Census Bureau matched people within the matching households. If all of the people in both the census and administrative households matched, this was defined as a whole household match. For imputation purposes, it is important that these whole household match rates be higher than the rates for the address matches (given above). The Census Bureau intended to impute only 5 percent of the nonresponse followup workload but hoped to impute these few households correctly. To achieve this would require high match rates at the household level. For self-reported households, the whole household match rates among address matched households in Paterson, Oakland, and Louisiana were 29 percent, 32 percent, and 29 percent, respectively. For nonresponse follow-up households, however, the match rates for the three sites were only 7 percent, 8 percent, and 13 percent, respectively. In other words, if nonresponding households were imputed unconditionally in the three sites, the low frequencies indicate how the imputed result would match the census outcomes. These low match rates support the Bureau's decision not to use administrative records to impute nonresponse follow-up households in 2000. On the basis of these findings, we support that decision. Furthermore, in our view, too little time remains to dramatically improve these match rates. Before the decision, the Census Bureau had initiated research aimed at trying to predict which nonresponse follow-up households could be correctly imputed. We are not persuaded that this strategy would ever be successful. The low match rates that the Census Bureau obtained at the three test sites imply the need for an exceedingly powerful predictive model. Furthermore, the low match rates are in part an artifact of certain limitations of the 1995 study. Basing predictive models on the 1995 study, in any event, is not likely to produce findings that can be applied in other contexts. The striking contrast between the self-reported and nonresponse follow-up enumerated households demands further study. Can this difference be due entirely to the correlation between a household's probability of nonresponse to the census and its probability of having incorrect or out-of-date administrative data, or is there another explanation? In particular, can one totally rule out any effect of the census data collection mode on the quality of the census data? It can be argued that whole household matches to the data collected by the census represent too narrow a definition of what constitutes good household data from administrative records. Households in which all of the census people were matched by administrative records but the database contained other people accounted for numbers comparable to the whole household matches in Oakland and Louisiana, although not quite one-half of the whole household matches in Paterson. The combination of whole

14 It is difficult to reconcile this match rate for Louisiana to the very low match rates for the individual source files in that test site. Unless there is an error (perhaps in the rates for individual files), this result could be explained only if the administrative records in Louisiana had records for nearly all of the census units and had additional, bad addresses numbering about six times the number of good addresses.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

74

household matches and this particular type of partial household match, representing census households in which all people were matched by people in the corresponding administrative households, numbered 11 percent of the address-matched households in Paterson, 16 percent in Oakland, and 25 percent in Louisiana. For self-reported households, the corresponding percentages were 40 percent in Paterson and 50 percent in the other two sites. The fact that the 1995 test data were lacking identifying information for dependents listed on tax returns may have contributed in a major way to the low match rates observed at the test sites. The 1996 census test will demonstrate to what extent the addition of dependent Social Security numbers to the IRS files can improve the whole household match rate15. The highest whole household match rates, by far, were obtained for households of size one, as measured by the census. These rates varied from 17 to 27 percent of the households with address matches in the three test tests. Adding partial matches in which the administrative records database accounted for the census people but included other people raises these rates to between 28 and 50 percent. That they are not higher may be attributable to the above average mobility of single-person households. Nevertheless, these match rates provide an indication of the level of matches that might be achieved in larger households if dependents were added to the IRS records. The IRS records will not capture all dependents, but if they did, one might expect the match rates for households above size one to exceed those of size one because larger households tend to be more stable as to their place of residence. The single greatest improvement may be the addition of data on dependents, which the IRS collects but which were missing from the tax year 1993 data used in the 1995 evaluation. Recommendation: Efforts to improve the match between administrative data and census data should be given high priority. These efforts should include attempts to improve the quality of the addresses obtained from administrative records (and the identification and deletion of nonresidential addresses) and to improve the coverage of household members. Recommendation: The Census Bureau should defer any research directed at predicting which administrative records are most likely to match census household records--at least until the match rates between administrative records and census data are substantially improved. Recommendation: The Census Bureau should take several steps to evaluate the potential contribution to an administrative records database of dependent data captured on tax returns.

15 There are a number of differences between the census and the IRS data with respect to where dependents are recorded as residing. These differences must be resolved in some manner if the intent in using the IRS data is to replicate census residence.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

75

•

Determine what dependent Social Security numbers the IRS now captures electronically on a 100 percent basis and obtain these data on all future IRS files delivered to the Census Bureau. • Determine whether the IRS captures dependents' names in addition to their Social Security numbers and, if so, secure their inclusion on future IRS files delivered to the Census Bureau. • Determine whether the IRS captures residential status (living at "home" or away for dependent), individually or collectively, and, if so, take steps to ensure their inclusion on future IRS files delivered to the Census Bureau. • Design and carry out a research project using the 1994 tax year file with dependents' Social Security numbers to assess the potential improvement in whole household matches in the 1995 test sites. Adding these findings to those obtained in the 1996 community census will provide a better measure of the contribution of data on dependents. More importantly, the Census Bureau can proceed with the 1995 reevaluation while waiting for the 1996 community census data. Augmentation of Integrated Coverage Measurement Household Rosters The prospective use of administrative records as a source of roster names in the integrated coverage measurement interviews raises concerns about privacy and confidentiality that the Bureau has not fully addressed. Does such usage violate the confidentiality restrictions attached to the acquisition of such data? The state food stamp offices that provided data for the 1995 census test interpreted their protection of clients' confidentiality as not allowing the use of names from food stamp records in those households for which food stamps were the only administrative record source. Ultimately, the public and the agencies that lend their administrative records for the census may view such use as innocuous. It is important, however, that the Bureau lay the appropriate groundwork in terms of explaining the procedures and its rationale before committing to a course that could risk adverse effects on the census. Such groundwork should include not only the agencies that provide the records, but also organizations that are concerned about issues of privacy and confidentiality. The focus group research done to date has not addressed this particular use of administrative records, but focus group participants have expressed concerns about applications that appear to be much more innocuous (Aquirre International and Bates, 1996). The focus group results suggest that, at best, the Census Bureau may face a difficult task in convincing the public on the use of administrative records for the census, although the group participants generally have little familiarity with the issues and with Census Bureau and other agency uses of records. This situation suggests that public education campaigns have the potential to be very effective in developing public understanding and support. The use of administrative records in the integrated coverage measurement roster also raised statistical concerns. In small-scale tests in Oakland and Paterson, 18 to 32

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

76

percent of the people identified in the administrative records database who did not match individuals listed in the census interview as residents of the same household were found to be census day residents who were also missed by Census-Plus. This result suggests a significant potential to add people to census households by using data from administrative records to bolster the integrated coverage measurement rosters, but the magnitude of the additions at these two sites raises concern about possible biases that the methodology might introduce. Is there any tendency for respondents to overstate census day residents if presented with names identified as coming from government records? In addition, the length of time between census day and the integrated coverage measurement interviews adds a recall effect to the measured reaction to names from administrative records. These questions are important because if there is indeed any response bias, its effect will be multiplied by its occurrence in the integrated coverage measurement sample. Recommendation: The Census Bureau should address statistical measurement issues, along with the privacy and confidentiality concerns, before committing to the use of administrative records to augment integrated coverage measurement household records for the 2000 census. Addressing privacy and confidentiality issues should include making contact with appropriate organizations. Imputation of Missing Long-Form Items As with the enumeration of special population groups, the panel has yet to see either a plan or any Census Bureau research relating to the use of administrative records to impute missing items from the long form. Any plan to use administrative records to impute missing items on the long form seems rather unrealistic at this time. Obviously, when people can be matched between administrative records and the census, using administrative records to impute missing items sounds reasonable. But what items would be imputed from what sources? More importantly, when will the Census Bureau evaluate such imputations? Such use would require an administrative records database that provides good coverage of households and their members and good coverage of long-form items. With much work remaining to be done in the development of a satisfactory database, and the decision not to use administrative records to provide short-form data for part of the nonresponse follow-up workload, we see little prospect for the successful use of administrative records to impute missing long-form items in the 2000 census. Unless the Census Bureau can develop a sound plan in fairly short order, and demonstrate some encouraging research results with the 1996 test or even the 1995 test, we suggest that the Bureau not continue with its plans. Estimation for Nonrespondents Not Selected in the Nonresponse Follow-Up Sample The Census Bureau has supported research on the use of administrative records as a component in the imputation of nonresponding households that were not included in the

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

77

nonresponse follow-up sample (Zanutto, 1996). This is not one of the uses that the Census Bureau has announced for administrative records, but it may prove to be one of the most promising. The use of administrative records as a component in the estimation methodology should be less controversial than almost any other application: administrative records would not replace direct enumeration, but would instead provide some data for households whose characteristics would otherwise be estimated without the benefit of any reported information. (The Census Bureau has not discussed this use or the estimation methodology with the panel.) If it can be demonstrated that the introduction of administrative records can improve the imputation of nonsampled nonresponse follow-up households, then we believe that such use of administrative records should be given serious consideration. At this point the evidence is unclear. Improvements in the construction of an administrative records database are needed to address other applications, and the potential contribution of administrative records to the estimation of nonrespondents cannot be evaluated adequately until better administrative data become available (Zanutto, 1996; Zanutto and Zaslavsky, 1996a, in press). Experimentation with an Administrative Records Census The data requirements for a complete administrative records census are much more stringent than those of the other applications that have been discussed. Complete coverage of households is absolutely critical, for example, and all short-form items must be available for all households. To the extent that these requirements cannot be met, other ways to obtain the missing information must be developed. The Census Bureau's agenda and research are in the early stages of development, and it is premature to attempt any assessment of them. CONCLUSION The Census Bureau must be able to demonstrate in the 1998 census dress rehearsal that an administrative records methodology or set of methodologies can be implemented in real time and yield satisfactory results. After 1998 there will be no additional data for evaluation. Nor will the census schedule allow time to modify plans. The Census Bureau must also make certain that neither the content, structure, nor delivery dates of the administrative records needed for the 2000 census will change in ways that would invalidate the tested procedures or create excessive delays in file preparation. We cannot yet assess the planned application of administrative records to the enumeration of special population groups, which is being tested with data collected in the 1996 community census. Nor can we yet assess the prospects for successful experimentation with an administrative records census, although we believe that it is important not to lose the opportunity presented in 2000. Another of the originally planned uses--to reduce the nonresponse follow-up workload by imputing 5 percent of the nonresponding households--has already been rejected by the Census Bureau, and there are

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

ADMINISTRATIVE RECORDS: LOOKING TO THE FUTURE

78

no explicit plans to use records other than the U.S. Postal Service Delivery Sequence File for updating the Master Address File. We are skeptical for a number of reasons about the feasibility of using administrative records to augment the integrated coverage measurement household rosters and to impute missing long-form items. We are intrigued, however, by the prospective use of administrative records as a component of the estimation methodology for census nonrespondents who are not selected for nonresponse follow-up. To support this and possible other uses of administrative records, we recommend (above) continuing research on the development of strategies for pooling multiple administrative records files to enhance their coverage and the quality of the data they contain and a number of specific research tasks that will assist this effort.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

79

References

Alexander, C.H. 1996 Some Basic Technical Information About the American Community Survey. Unpublished draft document (June 12). Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Ammenhauser, S. 1996 Processing and Operational Results from the 1995 Census Test Be Counted Campaign, 1995 Census Test Results Memorandum No. 48. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Ammenhauser, S., and S. Lucas 1996 Making Census Questionnaires Widely Available: Profile of Persons and Housing Units Enumerated via the Be Counted Campaign. 1995 Census Test Results Memorandum No. 25. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Anolik, I. 1996 Results from the 1995 Census Test: The Integrated Coverage Measurement (ICM) Evaluation of Out-Movers: Project 10. 1995 Census Test Results Memorandum No. 39. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Aquirre International and N. Bates 1996 Focus Group Reports on Data Sharing and Notification of Data Sharing. 1995 Census Test Results Memorandum No. 49. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Barrett, D. 1995 An Evaluation of the Local Update of Census Addresses Program and the Master Address File--Urban Test Sites. 1995 Census Test Results Memorandum No. 10. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Bates, N. 1996 Reinterviews and Reconciliation Using CAPI: The Integrated Coverage Measurement (ICM) Interview. Unpublished presentation, InterCASIC Conference, San Antonio, TX. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Bates, N., and C. Good 1996 Integrated Coverage Measurement (ICM) Evaluation Project 2: ICM Behavior Coding, 1995 Census Test Results Memorandum No. 18. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.. Bell, W.R. 1993 Using information from demographic analysis in post-enumeration survey estimation. Journal of the American Statistical Association 88:1106-1118. Biemer, P., and J.B. Treat 1996 An Investigation of Latent-Class Models for Evaluating the Census Integrated Coverage Measurement Process. Unpublished report. Research Triangle Institute, Research Triangle Park, N.C. Bureau of the Census 1960 The Post-Enumeration Survey: 1950. Technical Paper No. 4. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

80

1990 Household and Family Characteristics: March 1990 and 1989. Current Population Reports, Series P20-447. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1992 1990 Census Cost Components. Year 2000 Research and Development Staff Memorandum Series, Book I, Chapter 30, No. 4. Memo from Jay Keller to Susan Miskura (August 6). Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1995 995 Census Test Results Memorandum Series, No.s 1-54 (1995-1996). Decennial Management Division, Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1996a The Plan for Census 2000. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. April 5, 1996. 1996b A Master Address File Decennial Production Schedule. Unpublished document, Geography Division (December 17). 1997 Memorandum from Robert W. Max to Mary Mulry, March 14. Childers, D. 1996 Integrated Coverage Measurement (ICM) Evaluating Project 6: Evaluation of Dual System Estimation. 1995 Census Test Results memorandum No. 26. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Citro, C.F., and M.L. Cohen, eds. 1985 The Bicentennial Census: New Direction for Methodology in 1990. Panel on Decennial Census Methodology, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Cochran, W.G. 1977 Sampling Techniques. Third edition. New York: John Wiley and Sons Corteville, J. S. 1996 An Evaluation of the Master Address File. 1995 Census Test Results Memorandum No. 32. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Edmonston, B., and C. Schultze, eds. 1995 Modernizing the U.S. Census. Panel on Census Requirements in the Year 2000 and Beyond, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Ericksen, E.P., L.F. Estrada, J.. Tukey, and K.M. Wolter 1991 Report on the 1990 Decennial Census and the Past-Enumeration Survey. Report submitted by members of the Special Advisory Panel to the Secretary of the U.S. Department of Commerce (June 21). Fay, R., J.S. Passel, and J.G. Robinson 1988 The coverage of the population in the 1980 census. Evaluation and Research Report No. PHC80E4. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Gbur, P.M. 1996 Integrated Coverage Measurement (ICM) Evaluation Project 3: Noninterview Follow-up. 1995 Census Test Results Memorandum No. 44. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

81

Green, S., and V. Vazquez 1996 Final Report of the Postal Identification of Vacant and Nonexistent Units, 1995 Census Test Results Memorandum No. 23. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Griffiths, R. 1996 Integrated Coverage Measurement (ICM) Evaluation Project 11: The Contamination Study. 1995 Census Test Results Memorandum No. 36. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Hill, J., and T. Leslie 1996 1995 Coverage Study Results. 1995 Census Test Memorandum No. 38. Bureau of the Census, U.S. Department of Commerce. Hogan, H. 1989a Nine years of evaluation research: What have we learned? Pp. 663-668 in Proceedings of the Survey Research Methods. Alexandria, Va.: American Statistical Association. 1989b Nine Years of Evaluation Research: What Have We Learned? Unpublished memorandum. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1992 The 1990 Post-Enumeration Survey: An overview. The American Statistician 46(4):261-269. 1993 The 1990 Post-Enumeration Survey: Operations and results. Journal of the American Statistical Association 48:1047. Killion, R.A. 1996a High Priority Research Agenda. Draft memorandum to the Census 2000 Committee on Statistical Policy (October 27). Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1996b What Factors Should We Consider in Choosing How to Reach 90 Percent at the Census Tract Level? Report to the Panel to Evaluate Alternative Census Methodologies (November 4). Bureau of the Census, U.S. Department of Commerce, Washington, D.C.. Kish, L. 1965 Survey Sampling. New York: John Wiley and Sons. Kohn, F. 1996 Integrated Coverage Measurement (ICM) Evaluation Project 13: Evaluation of Reduction in Differential Undercount Based on the Analysis of Sex Ratios. 1995 Census Test Results Memorandum No. 45. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Krotki, K.J. 1978 Recent Developments in PGE/ERAD/ECP. Alberta, Canada: University of Alberta Press. Marks, E.S., W.P. Mauldin, and H. Nisselson 1953 The Post-Enumeration Survey of the 1950 census: A case history in survey design. Journal of the American Statistical Association 48:220-243. Moohn, B. 1995a An Assessment of the Operational Effectiveness of the Local Update of Census Addresses Program Part 1--Urban Test Sites. 1995 Census Test Results Memorandum No. 3. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

82

1995b An Assessment of the Operational Effectiveness of the Local Update of Census Addresses Program Part 2--Rural Test Sites. 1995 Census Test Results Memorandum No. 19. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Mulry, M.H. 1993 Census coverage measurement methodology research: Past and present. Pp. 617622 in Proceedings of the Section on Survey Research Methods. Alexandria, Va.: American Statistical Association. Mulry, M.H., and R. Griffiths 1996 Integrated Coverage Measurement (ICM) Evaluation Project 12: Comparison of Census Plus and Dual System Estimates. 1995 Census Test Results Memorandum No. 42. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Mulry, M.H., and B.D. Spencer 1993 Accuracy of the 1990 census and undercount adjustments. Journal of the American Statistical Association 88:1080-1091. National Research Council 1993 A Census that Mirrors America. Interim report of the Panel to Evaluate Alternative Census Methods, Committee on National Statistics. Washington, D.C.: National Academy Press. Navarro, F. 1994 Coverage Measurement Research--Sample Size. Unpublished note to Henry Woltman, December 28. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Neugebauer, S. 1996 Administrative Records File Acquisition History for the 1995 Test. 1995 Census Test Results Memorandum No. 24. Bureau of the Census, U.S. Department of Commerce , Washington, D.C. Passel, J.S. 1991 Age-period-cohort analysis of census undercount rates for race-sex groups, 19401980: Implications for the method of demographic analysis. Pp. 326-332 in Proceedings of the Social Statistics Section. Alexandria, Va.: American Statistical Association. Politz, A.N., and W.R. Simmons 1949 An attempt to get ''not-at-homes" into the sample without callbacks. Journal of the American Statistical Association 44:9-31. 1950 An attempt to get "not-at-homes" into the sample without callbacks. Journal of the American Statistical Association 45:36-137. Research/Strategy/Management and Beldon & Russonello 1995 The People Speak. Unpublished contract report on focus group observations: April-June 1995. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Robinson, J.G. 1996 Integrated Coverage Measurement (ICM) Evaluation Project 15: Evaluation of Census Plus and Dual System Estimates with Independent Demographic Benchmarks. 1995 Census Test Results Memorandum No. 43. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

83

Robinson, J.G., B. Ahmed, P. Das Gupta, and K.A. Woodrow 1993 Estimation of population coverage in the 1990 United States census based on demographic analysis. Journal of the American Statistical Association 88:10611076. Schindler, E. 1993 Sampling for the Count, Sampling for Non Mail Returns. DSSD 2000 Census Memorandum Series, #M-2. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Steffey, D.L., and N.M. Bradburn, eds. 1994 Counting People in the Information Age. Panel to Evaluate Alternative Census Methods, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Thibaudeau, Y, and A. Navarro 1995 Optimizing sample allocation of the 2000 nonresponse follow-up. Pp. 736-739 in Proceedings of the Survey Research Methods. Alexandria, Va.: American Statistical Association. Treat, J.B. 1995a Response Rates Analysis: Respondent-Friendly Implementation Evaluation and Multiple Sample Forms Evaluation. 1995 Census Test Results Memorandum No. 8. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1995b Response Rates Analysis: Respondent-Friendly Implementation Evaluation. 1995 Census Test Results Memorandum No. 9. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1996a Analysis Comparing the NRFU Block Sample and the NRFU Unit Sample Nonresponse Follow-up Evaluation. 1995 Census Test Results Memorandum No. 31. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. 1996b Integrated Coverage Measurement (ICM) Evaluation Project 9: Effect on the Dual System Estimate and the Census Plus Estimate of the Adds and Deletes to the Census File. 1995 Census Test Results Memorandum No. 40. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. U.S. General Accounting Office 1992 Decennial Census: 1990 Results Show Need for Fundamental Reform. Washington, D.C.: U.S. Government Printing Office. Vacca, E.A., M. Mulry, and R.A. Killion 1996 The 1995 Census Test: A Compilation of Results and Decisions. 1995 Census Test Results Memorandum No. 46. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. West, K.K. 1996 Integrated Coverage Measurement (ICM) Evaluation Project 4: Response to Coverage Probes in the Integrated Coverage Measurement Person Interview. 1995 Census Test Results Memorandum No. 21. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. West, K.K., and R.R. Griffiths 1996 Integrated Coverage Measurement (ICM) Evaluation Project 1: Results from the Evaluation Interview. 1995 Census Test Results Memorandum No. 33. Bureau of the Census, U.S. Department of Commerce, Washington, D.C.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

REFERENCES

84

White, A.A., and K.F. Rust, eds. 1996 Sampling in the 2000 Census: Interim Report I. Panel to Evaluate Alternative Census Methodologies, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Whitford, D.C. 1996 The 1996 Integrated Coverage Measurement Test. Presentation to the 2000 Census Advisory Committee on December 7-8, 1995, updated May 16-17, 1996. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Wurdeman, K. 1996 1995 Census Test Results: Administrative Records Evaluation--Phase II. 1995 Census Test Results Memorandum No. 54. Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Zanutto, E. 1996 Modeling Administrative Records for Estimation and Imputation of Census Nonrespondents under Sampling for Nonresponse Follow-up: Summary Report (October 16). Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Zannutto, E., and A.M. Zaslavsky 1996 Estimating a Population Roster from an Incomplete Census Using Mailback Questionnaires, Administrative Records, and Sampled Nonresponse Followup. Proceedings of the Bureau of the Census Annual Research Conference 12:741-760. Zanutto, E., and A.M. Zaslavsky In press Estimating a population roster from an incomplete census using mailback questionnaires, administrative records, and sampled nonresponse follow-up. Proceedings of the Survey Research Methods Section of the American Statistical Association.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX 85

APPENDIX

Sampling in the 2000 Census Interim Report I

Andrew A. White and Keith F. Rust, Editors

Panel to Evaluate Alternative Census Methodologies Committee on National Statistics

Commission on Behavioral and Social Sciences and Education National Research Council

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

86

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Harold Liebowitz is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bru ce M. Alberts and Dr. Harold Liebowitz are chairman and vice chairman, respectively, of the National Research Council. This project is supported by funds provided by the Bureau of the Census, U.S. Department of Commerce, under contract number 50-YABC-5-66005 Additional copies available from: Committee on National Statistics National Research Council 2101 Constitution Avenue, N.W. Washington, D.C. 20418 Printed in the United States of America Copyright 1996 by the National Academy of Sciences. All rights reserved.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

CONTENTS 87

CONTENTS

INTRODUCTION BENEFITS OF SAMPLING SAMPLING FOR NONRESPONSE FOLLOW-UP INTEGRATED COVERAGE MEASUREMENT FINAL COMMENTS REFERENCES 89 90 92 96 97 98

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

CONTENTS 88

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

89

INTRODUCTION This first interim report of the Panel on Alternative Census Methodologies focuses on the use of statistical procedures, especially sampling, in the conduct of the 2000 census. The panel has examined the research memoranda (Bureau of the Census, 1995-1996) resulting from the 1995 census test in Paterson, New Jersey, Oakland, California, and northwest Louisiana. We have also heard presentations provided by staff of the Bureau of the Census on the test and their work. Additional analysis of the 1995 census test is anticipated, and further information will be available from the 1996 census test. However, since decisions concerning many important features of the 2000 census are now being made by the Bureau of the Census, the panel believes that it is timely to offer its assessment on certain aspects of the design for the 2000 census. The Census Bureau has proposed a new design for the year 2000 census that includes sampling to follow up households that do not mail back their questionnaires or provide a telephone response. Sampling will also be used as part of an integrated coverage measurementl procedure to obtain as complete a count as possible and to reduce differential undercoverage among areas and population groups. The 1990 census, in contrast, attempted to count every household by personal follow-up of those households that did not return a mail questionnaire. These procedures were both expensive and did not yield the desired goal: estimated by demographic analysis, 1.8 percent of the population was not counted in 1990 (up from 1.2 percent in 1980), including an estimated 5.7 percent of blacks. Hispanics, Asian and Pacific Islanders, American Indians and Alaska Natives, renters, and residents of poor inner-city areas were also undercounted by larger percentages than the nation as a whole (Edmonston and Schultze, 1995). Demographic analysis has estimated that the undercount rate was higher in 1990 than in 1980, after steadily decreasing since 1940. In 2000, the Census Bureau is likely to face an environment of constrained funding, decreasing survey response rates (Edmonston and Schultze, 1995), increasing cynicism toward government, and decreasing availability of a qualified temporary work force (Stuart, 1995). Consequently, it is likely that repeating 1990 methods with the same relative level of resources to conduct the 2000 census will yield results that are of worse quality than obtained in 1990 and that have bias and undercoverage problems of unknown size and direction at levels of geography not addressed by subsequent post-enumeration surveys. To reduce costs, increase response, increase accuracy across the various levels of geography, and reduce differential undercoverage of population groups and among areas, the Census Bureau is redesigning the census process. The Census Bureau has identified innovative ways to improve the response rate of households and to use sampling and statistical estimation

1 See the report of the Panel to Evaluate Alternative Census Methods (Steffey and Bradburn, 1994) for a discussion of coverage measurement.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

90

techniques as integral parts of the census enumeration methodology that are needed to deal cost-effectively with household nonresponse. These techniques are critical to the success of the year 2000 census. The panel is very encouraged by the considerable progress that the Census Bureau has made in work on improving response (by mail or telephone) to the questionnaire, including: redesigning the questionnaire, facilitating response to the mail questionnaire with reminders and a replacement questionnaire, providing multiple response options including use of the telephone and Be Counted forms,2 use of non-English questionnaires, use of partnerships with local officials, advertising and other outreach efforts, use of service-based and special targeted enumeration methods, and developing in cooperation with local governments (and the U.S. Postal Service) a full and accurate master-address list geographically referenced to the street locations in the TIGER (Topologically Integrated Geographic Encoding and Referencing) database. Many of these initiatives are new to decennial census procedures, and the significant improvements documented in the 1995 census test research memoranda on these ideas demonstrate serious efforts at revising the census process towards the goals of reducing costs, increasing response and accuracy, and reducing differential undercoverage for population subgroups. The panel intends to comment further on these initiatives in its subsequent reports; however, this report addresses only the use of sampling and statistical procedures in the 2000 census because of the Census Bureau's need to make decisions immediately on these topics. BENEFITS OF SAMPLING Even with the potential for great improvement in the 2000 census through use of the operational initiatives mentioned above, the use of sampling and other statistical procedures will be fundamental to achieving the goals of reduced costs, increased accuracy, and reduced differential undercoverage. The panel endorses the general recommendations of both the Panel to Evaluate Alternate Census Methods (Steffey and Bradburn, 1994) and the Panel on Census Requirements (Edmonston and Schultze, 1995). We support the use of sampling procedures in the follow-up of households that do not respond by mail (or telephone call) to the census (sampling for nonresponse follow-up). We also endorse the recommendations of those panels to use sampling procedures to collect relevant data that will be used in conjunction with the appropriate statistical procedures to derive a final enumeration that has acceptably low levels of differential undercoverage and other enumeration errors (sampling for integrated coverage measurement). The Census Bureau's use of sampling for these purposes is consistent with past approaches. We note that the Census Bureau has expertise second to none in the many aspects of sampling and in fact has a long history of applying sampling to certain parts of census operations (Goldfield, 1992). The research memoranda that the panel has seen from the 1995 census test give valuable insight into which components of the operations for nonresponse follow-up and integrated coverage

2

Be counted forms are unaddressed census questionnaires available in public places.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

91

measurement succeeded and which components require further research for the 2000 census. Although some design decisions remain to be made and tested, the panel argues below that nothing in the census test, nor any other development, suggests that a decennial census that reduces costs, reduces nonresponse bias, increases accuracy, and reduces differential undercoverage can be conducted without the use of some form of sampling for mail nonresponse follow-up and some form of sampling for integrated coverage measurement. The traditional census method (used in 1990) begins with the mailing of census forms to a comprehensive list of residential addresses. For households that do not respond to the mail questionnaire, census enumerators undertake an intensive follow-up effort to contact the households and elicit responses. This process is continued for an extended period of time in an attempt to physically enumerate every household and all the people in every household. Despite the gains in address list development, form design, prenotice and reminder mailings, and various outreach efforts, exclusive reliance on physical enumeration of all households cannot be successful in 2000. Based on the results of the 1990 census, it is highly unlikely that the Census Bureau can carry out this type of decennial census with acceptable accuracy within the current expected levels of funding. An effort to conduct a census in 2000 using 1990 methods--that is, attempting to the fullest extent to physically enumerate every household, with the funding levels that now seem probable--will likely result in a census of substantially lower quality than previous censuses. That lower quality will be readily apparent to all knowledgeable users of census data. The panel's view that sampling and statistical estimation are essential ingredients to the 2000 census does not derive only from a concern that there will not be sufficient resources to pay for using 1990 methods. Even if the resources were available to conduct such a census, we believe it is likely that the use of statistical procedures can allow more effective use of those resources. Our view that statistical procedures can improve quality at any census funding level is based on several limitations of the 1990 census and earlier censuses that are not always recognized in discussions of the relative merits of statistical procedures. We have considered four issues. First, there is an issue of timeliness. Any approach that attempts physically to enumerate every household through several field visits invariably takes considerable time to complete. The amount of time varies considerably by area, depending upon the mail return rate and the difficulty in contacting households. More important, it is inevitable that the quality of the information obtained from those households contacted late is of lesser quality than information from other households. Respondents simply do not remember or know who exactly was present on census day, or the characteristics of those people. A second, related issue is that there is a biased and uncontrolled sample of cases left over at the end of every field period about which little is known. Census forms for these households are inevitably completed using information from neighbors, landlords, and others. These "last-resort" cases constitute a significant proportion of census returns in some areas: in 1990 they accounted for nearly 4 percent of all households in the country. Although the information on these households is useful, it must be recognized that these data are collected by temporary employees who are generally inexperienced and under considerable pressure to complete their workloads in a timely fashion and that these data are often of much lower quality than those collected on the returned questionnaires and from field follow-up of household occupants. Therefore, last-resort data are not necessarily more reliable than

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

92

estimates based on statistical methods, although they are likely to be much more expensive. (Further analysis of these last-resort cases would be extremely valuable for understanding when they are and are not reliable and how to best make use of this information.) This discussion leads to a third issue, which is the availability of a work force that is capable of competently carrying out the census enumeration. Census takers are temporary employees, paid at a low hourly rate, who must be recruited locally in all areas of the country--most heavily in those areas with the lowest mail return rates. Evidence from the 1990 decennial census (Stuart, 1995) and from the 1995 census test strongly suggests that it will not be possible to locate enough temporary competent employees to complete a 100 percent mail nonresponse follow-up in the 2000 census. Even if it were possible, the cost would greatly exceed current expected levels of funding. Furthermore, it is likely that the areas with the lowest mail return rates and the highest concentration of groups that historically have been undercounted in the census are also those in which it will be most difficult to find sufficient numbers of enumerators. Fourth, there is ample evidence from recent censuses that the methods used in the 1990 census resulted in a substantial amount of both undercoverage and erroneous enumeration and that these errors are very differentially spread across demographic and geographic subgroups. From the 1990 experience, attempts to contact every household that does not return a form by mail do not appear capable of reducing these problems, and especially the differential undercount, to any appreciable extent. Although the initiatives to improve the master address list, develop outreach, make use of respondent-friendly instruments, use reminder cards, and other process improvements will help to reduce undercoverage and erroneous enumeration, we see no evidence from the 1995 census test to suggest that these enhancements alone will come acceptably close to eliminating the undercoverage and erroneous enumeration problems. The combination of the problems mentioned above indicates that it will also be necessary to include procedures for sampling for nonresponse follow-up (to address the first three issues) and for integrated coverage measurement (to address the fourth issue). In the next two sections we discuss these procedures in more detail. SAMPLING FOR NONRESPONSE FOLLOW-UP In addition to addressing the limited work force problem, sampling for nonresponse follow-up provides an effective means of addressing the related issues of timeliness, the use of last-resort returns, and the quality of field enumeration. By requiring that only a sample of those households that do not return a form by mail or telephone be followed up, more effort can be concentrated on the process of completing household enumeration in a given area, which can be carried out more quickly and more intensively (so that there remain fewer nonresponding cases that require last-resort procedures), and which can make use of better trained, more experienced personnel. Thus, the introduction of sampling will reduce other sources of error inherent in operations. These benefits of sampling are well recognized in survey research and are generally accepted when survey instruments are involved. The panel believes that these benefits are also likely to be substantial in the decennial census.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

93

It is not realistic to expect that costs, staffing, and time expended will all decrease in direct proportion to the sampling rate. Thus, a decision to sample 25 percent of households (not returning a form by mail) in an area does not mean that the enumeration can be done in one-quarter of the time, using one-quarter of the field staff, with the elimination of last-resort enumerations. However, the enumeration can very likely be done in less time than would a full enumeration, using fewer staff, and with a smaller fraction of last-resort enumerations. It is for this reason that the panel believes that sampling for nonresponse follow-up needs to be an integral part of the plans for the 2000 census. In discussing the comparative accuracy of census counts and the quality of census data with alternative methodologies, it is very important to consider the uses of census data. In particular, the effects of sampling on the accuracy of counts and other data will differ by the level of geography. In concluding that procedures for sampling for nonresponse follow-up will give better results than use of 100 percent follow-up, the panel is considering the use of data at broad and intermediate levels of geography, such as those identified in constitutional and legislative requirements for the use of census data. In sampling, a measurable sampling variance will be substituted for the bias resulting from an uncontrolled sample. Thus, for enumerating the populations of states, congressional districts, state legislative districts, and many governmental units, we believe that it is likely that reductions in errors and omissions in enumeration will outweigh the introduction of sampling error. However, for sufficiently small levels of geography, this assertion may not hold because sampling error (relative to the size of the population) increases as the population of a geographic area decreases. Other errors are more likely to have effects of similar relative magnitude at all geographic levels (although they will vary across geographic units, at any given level). Therefore, at the block level, for example, it seems unlikely that the average level of (relative) error will be less when sampling is used (although this conclusion would depend upon the extent of sampling). There is a tradeoff between the demands for accuracy for very small geographic areas and those of larger areas. In the panel's view, the needs for data for larger domains should take precedence. For example, while block unit data are important, they are almost always aggregated to form larger geographic areas. When such aggregation occurs, the relative effects of sampling error decrease, and the benefits of sampling for nonresponse follow-up will tend to outweigh the effects of sampling error. Sampling for nonresponse does not introduce bias when particular groups are sampled at higher or lower rates because appropriate methods will be used to create the final counts that account for the designed variations in sampling rate. In sum, it appears that the error introduced by sampling at the block and tract levels would not vitiate the value of census data for research and policy making. Another significant concern about census operations involves equity. It is important that the procedures used ensure results of uniform (and acceptable) quality for geographic areas or groups of similar size. The use of sampling for nonresponse follow-up has great potential to further this aim. In contrast, there is substantial evidence that the use of 100 percent follow-up of mail nonresponse in the 1990 census did not result in uniform and high quality to a satisfactory extent. There was considerable variation in the use of last-resort enumerations for different geographic regions, and it has been clearly demonstrated that census undercoverage varied greatly across geographic and demographic groups. Much

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

94

money and effort were concentrated at the end of the census operations but data quality decreased at the same time. The use of sampling for nonresponse follow-up has an equitable effect in that, on average, no group's estimated population is made systematically larger or smaller by sampling, regardless of whether certain groups have lower response rates and regardless of the rate of sampling among nonrespondents. A secondary issue is that estimates for some groups, particularly in small areas, may be less precise (although not systematically biased either upwards or downwards) because high nonresponse makes these estimates more dependent on sampling. It is possible to attain more nearly uniform accuracy for areas of comparable size by appropriate choices of sampling rates. Similarly, it is also possible to provide greater accuracy for small political units than for comparably sized areas that are not political divisions by appropriate sample allocation. Consideration of these tradeoffs within the budget allocated to sampling will be part of the sample design process. The use of sampling for nonresponse follow-up gives reasonable promise that accuracy can be improved and differences in the quality of enumeration can be diminished by concentrating the efforts of better enumerators on the sampled cases. While there should be gains in accuracy for all areas, those places where the methods used in 1990 were the least successful, and produced the most bias, stand to benefit most, in terms of improved accuracy of enumeration, through the adoption of sampling for nonresponse follow-up. Although the panel endorses the use of sampling for nonresponse follow-up as part of the operations for the 2000 census, the decisions concerning the design of the sampling plan should not be made without due consideration of a broad range of issues. Six issues that need to be considered are: (1) the levels of sampling and other error across geographic units at different levels of aggregation; (2) the costs of enumerating different types of households that do not respond by mail; (3) the estimation methodology; (4) how the estimation will be implemented in practice; (5) the timing of the data collection; and (6) the public perception of the use of sampling. The Bureau of the Census should adopt a systematic approach to developing a good sample design and associated estimation procedure. This cannot necessarily be achieved by postulating a small number of designs and determining which appears to optimize various criteria. Some further exploration of criteria is needed. We offer several examples of the need to proceed in this manner. First, consider two ways to truncate full-scale direct enumeration prior to initiating sampling for nonresponse follow-up: (1) after a given date (this procedure is called the direct application of sampling), and (2) after a given fraction of the population has been counted in an area. It has been suggested (Killion, 1995) that the former would be superior to the latter. Consideration only of sample size and sampling error, using an assumption that every household costs the same to enumerate, shows that this conclusion obtains. However, some households are more expensive to enumerate than others. Assuming a fixed number of completed household enumerations and comparing direct versus truncated sampling then yields the result that direct application of sampling will likely sample these higher cost cases at a substantially greater rate than a plan that introduces sampling after some threshold response rate is achieved. By continuing to enumerate physically to a threshold without sampling, a larger percentage of the less costly cases may be completed. Second, conversely, it has been suggested (Killion, 1995) that a plan that enumerates

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

95

90 percent and then samples 1 in 10 remaining households would be more acceptable than a plan that enumerates 70 percent and then samples 7 in 10 households (for example). Both of these procedures result in data collection from 91 percent of households. The view is that the former is seen as placing much less reliance on sampling than the latter. But the public perception of the reliance on sampling might be represented as well by the number of households that each sampled household is expected to represent as by the size of the population represented by sampling. In this case, it is not clear that having each sampled household represent nine others3 is preferable to having each sampled household represent itself and possibly one other. Moreover, we believe that the issue of truncation cut-off points should be resolved scientifically, using statistical criteria and examining the expected sampling variance at various levels of geography. Such a resolution could consider operational criteria such as the percentage of responses after a certain period of time. A variable cut-off strategy depending on local response rates could also be considered. Third, up to the present, the Bureau of the Census has proposed 100 percent followup until either 70 percent or 90 percent of a region has been physically enumerated. This choice begs the question as to whether some other percentage might produce superior results. Indeed, the best performance may come from a design with a variety of different truncation percentages to optimize enumeration in different areas. Finally, and perhaps most important, the advantages of a given sampling scheme will depend dramatically on the level of geography at which it is implemented. For example, a scheme that requires 90 percent completion at the block level before going to sampling will have greatly different characteristics than one that requires 90 percent completion at the county level. The former will be much more expensive than the latter, but the latter would lead to very great inequities in the level of precision available at various lower levels of geographic aggregation, e.g., at the tract level within each county. A county will often include areas that differ greatly with respect to census enumeration, e.g., areas with high and low mail back response, and areas with easy and difficult field follow-up. If the completeness criterion is set at the county level, then higher rates of completion will be obtained where it is easiest and other areas may still have much lower completion rates. It is important to win general acceptance of the concept of sampling for nonresponse follow-up. Although it may be necessary to propose some examples for public debate, the Census Bureau should not constrain itself to choose a specific and possibly oversimplified design at this time. Thus, the panel urges the Bureau to continue to apply a flexible approach to developing a sampling scheme for nonresponse follow-up on the basis of the results of the 1990 census and the 1995 test census in order to achieve a sensible balance among all of the important considerations. This might include different truncation points and different sampling rates in different places. This plan must be based on reasonable cost assumptions and must integrate the estimation and imputation procedures into its design. We recognize, however, that 2000 is fast approaching, and the Census Bureau should identify the most promising design(s) as soon as possible.

3

With the use of imputation, this is a literal statement.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

96

INTEGRATED COVERAGE MEASUREMENT As indicated above, even with the successful implementation of the various improvements in census methodology explored in the 1995 census test (other than sampling and statistical procedures), there will remain some people who are missed and some smaller degree of erroneous enumeration. These coverage errors will undoubtedly occur differentially across geographic areas and among different population subgroups, as they have in past censuses. Because this systematic error component will dominate other forms of error at broad levels of aggregation, it is essential to use integrated coverage measurement in the process of computing the final census count. Integrated coverage measurement involves two steps: first, sampling is used to identify the nature of the undercoverage problem; second, statistical procedures are used to incorporate the results of the sampling. Such integrated coverage measurement, as part of the census, is critical to reducing the differential undercount among population groups. Integrated coverage measurement in 2000 will differ from the coverage measurement component of previous censuses in several important ways. Two key instances are that it will have a larger sample size so that accurate estimates can be made at lower levels of aggregation, and it will be completed earlier than the postenumeration survey conducted in 1990. In addition, integrated coverage measurement will account for a larger fraction of the census count because it will replace several coverage improvement activities used in recent censuses, such as the recanvass operations and the parolee/probationer check. It will also partially replace vacancy/delete operations. In addition, it will be more closely connected operationally and temporally with the major census field activity and therefore will be a more integral and necessary part of completing an accurate census. The panel also sees an important interrelationship between the operations of the nonresponse follow-up plan and integrated coverage measurement. The quality of the integrated coverage measurement operation will almost certainly be affected by the length of delay between census day and the conduct of the integrated coverage measurement interview. With a longer time interval, it will be more difficult to get accurate information from a respondent about the household on census day. Also, a greater proportion of households will have moved, contributing substantially to the difficulty of obtaining accurate information. The use of sampling for nonresponse follow-up offers the potential to minimize any time interval between the date of the census and the conduct of the integrated coverage measurement interview. Minimizing the delay improves the quality of the census counts. Moreover, an integrated coverage measurement program, based necessarily on a relatively small sample, argues in favor of the use of sampling for nonresponse follow-up. Since a component of the final count is to be based on data from the integrated coverage measurement sample, a consideration of the total error of the census as a whole argues that it is not logical to expend tremendous resources to enumerate every last household physically prior to integrated coverage measurement. A portion of those resources would be better spent increasing the size of the integrated coverage measurement sample used in combination with nonresponse follow-up sampling. The Census Bureau's plans for selecting an integrated coverage measurement procedure involve comparing the performance of a dual systems estimation approach with the

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

97

performance of a new methodology called Census Plus. The panel expects to learn more about the performance of these alternative approaches to integrated coverage measurement in the 1995 census test in its continuing work and to comment on them in a subsequent report. At this juncture, the panel endorses the use of integrated coverage measurement without commenting on the two specific approaches being considered. The panel emphasizes that the success of any approach depends heavily on the structure of the integrated coverage measurement interview and the rules used to establish the list of persons present on census day. The Census Bureau has identified deficiencies in this regard in the 1995 census test, which were to be expected with the first trial of a novel approach. The panel is very much encouraged by the rapid developments that have taken place in this area in the months since the 1995 census test, and the panel is optimistic that a sound set of procedures can be in place for the 2000 census. FINAL COMMENTS A combination of sampling for nonresponse follow-up and for integrated coverage measurement is key to conducting a decennial census at an acceptable cost, with increased accuracy and overall quality, and reduced differential undercoverage. Sampling and statistical procedures build on the strengths of traditional census operations to collect the information quickly, reduce the dependence on last-resort information, and aid in the development of a competent enumeration staff. Building on considerable evidence from past Census Bureau tests that it is feasible to conduct follow-up sampling to reduce cost and nonresponse to field follow-up, the 1995 census test has provided sound evidence that mail nonrespondents can be followed up effectively on a sample basis and that sampling for nonresponse follow-up can be integrated successfully with coverage improvement sampling. The Census Bureau has made considerable progress in a wide range of techniques for conducting the census, including sampling, to improve the 2000 decennial census. The panel intends to comment further on these initiatives in its subsequent reports.

Copyright © 1997. National Academies Press. All rights reserved.

About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.

APPENDIX

98

REFERENCES Bureau of the Census 1995-Census Test Results Memorandum Series, Nos. 1-46. Bureau of the Census, 1996. U.S. Department of Commerce, Washington, D.C. Edmonston, B., and C. Schultze, eds. 1995 Modernizing the U.S. Census. Panel on Census Requirement in the Year 2000 and Beyond, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Goldfield, Edwin D. 1992 Innovations in the Decennial Census of Population and Housing: 1940-1990. Paper prepared for the Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, National Research Council, Washington, D.C. Killion, R.A. 1995 Agenda for October 31, 1995 SESC Meeting., Unpublished memorandum, Bureau of the Census, U.S. Department of Commerce, Washington, D.C. Steffey, D.L., and N.M. Bradburn, eds. 1994 Counting People in the Information Age. Panel to Evaluate Alternative Census Methods, Committee on National Statistics, National Research Council. Washington, D.C.: National Academy Press. Stuart, J. 1995 2000 Decennial Census Labor Force Issues: Escalating Enumeration Costs. Unpublished paper, Bureau of the Census, U.S. Department of Commerce, Washington, D.C.