Business data collection methodology: Current state and future outlook

2.1Qualitative research on the response burden

Compared to social surveys, business surveys have some unique characteristics that call for special attention because of their impact on accuracy of collected data [1, 3, 4, 5]. Among these characteristics is the business response process where: (i) the business management, not the surveyor, decides who should respond; (ii) respondents often need to carry out an internal data collection; (iii) survey questions typically require knowledge of work-related concepts; and (iv) data usually have to be retrieved from the business information system [6, 7]. The response process thus evolves at two levels: people go through their own cognitive processes of comprehension, retrieval, judgement, and communication [8, 9] while taking part in organizational processes [7, 10, 11, 12, 13, 14, 15]. Qualitative research (like questionnaire pretesting) sheds light on the response process and helps understand what and why errors happen in this process (see Section 2.2). Technology adds to the complexity of the response process in web business surveys but also opens up new possibilities for studying, supporting and extending this process, like the analysis of paradata (see Section 2.3).

As business survey response occupies people’s paid time and is often mandatory, burden imposed on businesses is considered one of the key constraints in surveys, next to internal costs, time, and quality considerations [16, 17, 18]. Sources of response burden are numerous [19] as are the activities that national statistical institutes (NSIs) have developed to deal with burden [20, 21].

Measuring burden is not easy and reported implementations are rare [19]. Dale and Haraldsen [22] proposed an operationalization of both actual and perceived response burden along with perceived causes of burden and motivational factors with nine questions. Statistics Finland implemented a reduced set of only four questions as part of a voluntary response burden questionnaire accessible through a click-through link on the last page of every business web survey in 2018: two questions collecting actual burden in time, one assessing perceived burdensomeness, and one open question for feedback [23]. However, due to unacceptably low response rates, the click-through link was removed and instead respondents were routed automatically to the response burden questionnaire after logging off the main survey. This procedure achieved an average response rate of 15 percent (range: 3–37 percent). More work is thus needed to address how best to activate respondents to participate in a response burden questionnaire and whether burden measures suffer from non-ignorable nonresponse. Detecting burden hotspots that may include not only large but also small establishments in certain sectors (e.g. [24]) also warrants further attention.

2.2Qualitative research on the response process

Questionnaire pretesting has been considered indispensable for designing or revising data collection instruments in official statistics [16, 25]. For social survey questionnaires, guidelines on testing methods have been discussed since the 1980s. For business surveys, the guidelines on implementation of such qualitative inquiry are still developing [26] with only [27, 28, 29] providing an overview of methods for questionnaire development and testing tailored to the business context.

Snijkers [30] reported their experience in using pre-field qualitative research in response to a mandate for combining two existing business surveys: Statistics Netherlands’ Quarterly and Annual Survey of Finances of Enterprises and the Dutch Central Bank’s Monthly Balance of Payments Survey. Before drafting the questionnaire, the researchers sought to study the response processes of businesses by launching a qualitative feasibility study including on-site visits to large non-financial businesses. The results led to the conclusion that combining both surveys would yield no new benefits to the businesses, even though the proposal seemed logical to the two sponsors. Specifically, the researchers concluded that data quality improvements and response burden reduction may only be achieved with a combined questionnaire tailored to the business context.22 Businesses’ internal structures, organizing the response process, and giving businesses enough time to get prepared should also be taken into consideration. Based on this experience, the researchers argued that starting questionnaire pre-testing at the stage when a draft questionnaire has been developed is too late. More preferable is to study the business context earlier in order to obtain a fuller picture of how information is distributed between people, between sources, between departments, and over time, and use these insights when drafting a questionnaire.

Irastorza et al. [31] applied pre-field qualitative research in the context of extending the survey universe to include micro businesses of 5–9 employees in the European Survey of Enterprises on New and Emerging Risks (ESENER), a survey which focuses on how workplaces manage occupational safety and health risks (OSH) in practice. In-depth interviews with these micro businesses revealed that most OSH-related responsibilities were outsourced to external service providers who handled the relevant arrangements. Thus, the investigators identified a potential mismatch between paper compliance and actual workplace practice. The study concluded that comprehension of survey questions may vary considerably with business characteristics, in particular size, sector, and the way OSH obligations are met (in-house vs. external service provider), and that regular qualitative inquiries are needed to reconcile survey answers with the realities of workplace practices.

Brand et al. [32] consider the use of post-field qualitative interviews to investigate a peculiar response pattern observed in the Business Tendency Survey (BTS), a monthly panel survey conducted by the Central Bank of Turkey to track business conditions and sentiment. One of the key questions is, what is your impression of the general economic conditions in your industry, compared to the previous month? More optimistic, remain unchanged, or more pessimistic. In the period 2007–2018, between 60–70 percent of respondents replied “remain unchanged” with the exception of some months in 2008 until mid-2009 [33]. A concern was raised regarding question interpretation as the literature suggests that a neutral response can be a hidden “don’t know” [34], a face saving “don’t know” [35], a result of hesitancy or lack of information [36], or an outcome of satisficing [37]. In-depth interviews with respondents that choose the “remain unchanged” answer for a long time revealed that most of these respondents interpret “remain unchanged” as the same increase or decrease in optimism as before. Only if the monthly reported increase or decrease in optimism rose or fell, these respondents would choose the increase or decrease option. The study thus revealed that once a positive/negative expectation was reported, all subsequent positive/negative expectations of the same magnitude were reported as “remain unchanged”. Therefore, the study quashed initial concerns about satisficing and raised concerns about unintended interpretation that should be further investigated because of its consequences for the interpretation and relevance of derived indicators (calculated as a difference between positive and negative answers).

2.3Response process in web surveys

In web surveys, a useful quantitative complement to qualitative research for studying the response process is the collection of paradata, that is automatically generated process data [38] or any data about the process by which the data was collected. Once the registration system is set up, paradata is typically created in real time and inexpensively, and has a wide range of uses from managing fieldwork to evaluating survey errors [39, 40]. Paradata in business surveys has received increasing attention [2]. In particular, audit trails (a time-event schedule which records date and time of each mouse click and keystroke in the questionnaire) are being used for identifying different response profiles and highlighting problems in comprehension, which can be addressed using other means (e.g. cognitive interviewing, usability testing). NSIs, such as Statistics Netherlands [41] and, more recently, the UK Office of National Statistics [42] have reported their experiences. The raw logs contain massive amounts of noisy data in terabyte-size form, so a key issue is to extract relevant information in an efficient manner. Statistics Netherlands, for instance, reported problems with processing huge amounts of paradata which the UK Office of National Statistics largely avoided by analyzing micro-level data from a subsample and by relying on aggregate data from Google Analytics for the full sample.

Statistics Canada has developed an automated procedure to clean large amounts of data by parsing out the most important information and creating a report in some hours [43]. Their paradata is generated by Microsoft Internet Information Services logs, which records HTTP transactions, showing actions taken in the web questionnaire (e.g. time per page, type of browser and device used, help button usage, edits triggered, path through questionnaire etc.). The logs contain neither the respondent’s answers nor the data on web pages accessed outside of the questionnaire. Analysis of these data makes it possible to identify critical pages where the respondent spent a lot of time, saved the data, or abandoned the questionnaire. It also reveals scanty use of help pages (e.g. used by only one percent of respondents in the Monthly Survey of Manufacturing) and modest corrections of data upon triggered edits (e.g. done by as few as eight percent of respondents in the Monthly Survey of Manufacturing; only irregular reporters changed the data more than half of the time). Paradata can be further used to improve data collection strategies. Statistics Canada is in the process of developing models to predict the time of a response using web logs and call history data. Units predicted to respond soon might, for example, be placed lower on the prioritization list for follow-up calls to allocate resources more efficiently.

Another priority research area among business surveys is cutting down on the amount of manual editing and follow-up calls. In business surveys, editing collected data may take up to 40 percent of the budget [44, 45]. Asking respondents to correct or validate data when they are entered in the questionnaire may reduce the need for verification on the survey organization’s side [46], but it also presents challenges. For example, the Australian Bureau of Statistics noted issues with embedded validation (also referred to as in-form edits or error messages) in relation to the response process [47]. Embedded validation should be neither a replacement for good questionnaire design nor an impediment to the majority of respondents with no need for assistance to complete the questionnaire. The respondent should always have a way out and never get stuck on a question; so hard edits should be used with extreme care and mainly limited to routing questions. The MoSCoW prioritization technique (Must have, Should have, Could have, and Won’t have [48]); may help decision making about which validation rules to embed. However, to support response and validation, a list of all survey questions or sections and a summary or reconciliation page should be available [42, 49]. Other recommendations for embedded validation include providing a clear diagnosis of the problem and the required action, or at least provide an opportunity to explain an unusual combination [50]. As proposed by Lorenc et al. [46], validation messages should be constructed and evaluated in a questionnaire design, evaluation, and testing (QDET) framework, but Price [47] notes that cognitive and usability testing is challenging because validation messages are rarely triggered, thus testing requires careful purposive sampling of businesses with problematic response behavior.

A related area of research is the validation of external data within the survey. For example, Statistics Norway has implemented an implicit validation check of government subsidy amounts so that the amount is provided in brackets as part of the question text. The respondent either re-enters the same number or provides a different one [49]. However, it remains unclear whether such an implicit dependent interviewing approach actually improves the data quality, or results in a confirmation bias [50].

3.1Integrating alternative (non-survey) data sources

In Europe, for example, the general recommendation is to use “administrative and other data sources whenever possible to avoid duplicating requests for data” [16]. Interestingly, countries are still at very different stages regarding the use of administrative data. Usage of these data varies from supporting survey data collection and replacing some survey data to estimating errors and using the administrative data as the main source that could be improved by survey data.

Some NSIs are in the early stages of exploiting administrative data by ensuring legal bases and verifying the source quality [52]. Recently, the Turkish Statistical System (TurkStat) has overcome legal milestones to acquire on-demand and scheduled administrative data transfers of tax records and other information from the Revenue Administration and Social Security Institution for the production of short-term indicators, annual business statistics, and national accounts since March 2018 [53]. The Hellenic Statistical Authority (ELSTAT) has recently put in place a new production system for the compilation of Structural Business Statistics based on a combination of multiple data sources: tax data, social insurance data, commercial data, and a small-scale supplemental sample survey for the estimation of variables not available in the administrative sources [54].

However, even when a legal basis for the use of administrative data has been formed, its content, format, and scope might not be satisfactory to be used for official statistics. The Statistical Office of the Republic of Serbia (SORS) found out that some important variables were missing in the Central Register of Compulsory Social Insurance for the calculation of registered employment and had to develop new methodologies to augment this administrative data with the Statistical Business Register data [55]. The Statistical Business Register itself relies on survey and administrative data for updating [56]. On the other hand, administrative data may help estimate errors, among them particularly problematic coverage errors [57, 58].

Statistics Canada plans to eliminate (almost) all of its 40 agricultural surveys and a census that collects data directly from farm operators every five years [59]. Two types of data sources are considered essential to reach this goal: a) about 300 separate administrative datasets, e.g. tax data, data from supply-management sectors (including dairy, chicken, eggs and turkey), data from insurance agencies on planted and insured crops, land valuations, honey bee permits, winery establishment grants, etc.; and b) satellite imagery with remote-sensing and geospatial information. Some data will come from other harmonized business surveys and model-based predictions. A key planned innovation is to evaluate the quality of a single item in the alternative data source (micro approach) and use it as a replacement for a respondent’s answer if a sufficient quality threshold is reached, thus, smartly personalizing the questionnaire at the farm-unit level whenever reliable data is available for that unit. This idea will be tested in the 2021 agricultural census which will run a direct data collection (featuring also an interactive web mapping application for geo-coding of reported data) in parallel to predicting values with alternative data sources.

3.2Automation and technical standards

A typical characteristic of business surveys is that businesses may receive several surveys and often receive the same survey several times [5]. Poor survey coordination and administration may cause irritation among businesses and are often consequences of the traditional non-tailoring and stove-pipe (or silo) statistical production. An increasing number of NSIs are moving towards a centralized and integrated coordination system.

One such infrastructure model, proposed by the Centre for the Development of Vocational Training (Cedefop), includes several components, including a relational database that is linked to sample sources (e.g. official registers, panels, etc.), the survey software (supporting questionnaire design, translation, fieldwork, etc.), statistical software (for data processing, analysis, output creation, etc.), data storage, and other documentation (e.g. metadata) [60]. Such models support data quality management from questionnaire development through fieldwork to data dissemination (e.g. [61]).

A prominent example of an NSI changing to a centralized coordination and integration system is Statistics Portugal. Portugal has moved from paper to electronic questionnaires, embracing a multi-source and mixed-mode approach, and implementing more automated data collection [51, 62]. Over 98 percent of all questionnaires were completed online in 2017. This effort was the result of an extensive centralization of data collection and the creation of an integrated infrastructure to support the statistical production (a new central data collection department and an Integrated Survey Management System, denoted as SIGINQ). One of its components is WebInq, an internet data collection service which offers a flexible way of linking various respondents, companies, and surveys. It also allows respondents to find help on electronic questionnaires and the products and services prepared with the collected data. Another important milestone was the creation of the Simplified Business Information System (IES) in 2006 [63]. This system facilitates several dispersed but identical legal obligations of businesses, e.g. annual accounts, tax statements, statistical information, etc. Statistics Portugal achieved complete coverage of the business universe (from 50,000 to 400,000); reduction of information availability from 12 to 6.5 months; automatic data collection by electronic means; a significant increase in detail; and the complete elimination of one of the costliest surveys. Plans to further simplify the IES include pre-filling questionnaires with previously-submitted and already-validated tax data. As a complement to web questionnaires, Statistics Portugal’s Automated Data Transmission (TAD) service, which also involves the upload of XML files, as well as web services, already reached 24 percent of collected questionnaires in 2017.

By integration, standardization and harmonization of data collection systems in Portugal, uniform processes across the various production units could be created, eliminating differences between common processes. By economies of scale, productivity increased, the risk of failures decreased, and resources were freed up to invest in innovative techniques and develop new capabilities. On the other hand, a harmonized system resulted in loss of flexibility. Critical success factors include strong top management sponsorship, the involvement of a multidisciplinary team, extensive design requirements, and internal change management. To the best of our knowledge, Portugal is the only country that has implemented such an advanced and integrated system. Another country also having implemented a coordinated system is Belgium.

Statistics Belgium adopted XBRL (eXtensible Business Reporting Language) as a standard for business data collection [64]. XBRL is a freely available global standard for exchanging business (especially financial) information, such as financial statements. In 2008 Statistics Belgium started to develop an XBRL-based web questionnaire as an e-government tool and as the standard for data collection amongst businesses and finalized it in 2017. Since then all surveys adopting the XBRL format have been monitored in one single system (StatData), directly connected with the Business Register. All the surveys follow the same standardized process in terms of loading the sample, loading data to pre-filled forms, creating user-ids and web forms, and exporting data. The XBRL adoption makes it possible to re-use existing concepts and create new specific concepts for each survey. Apart from the cost reduction, the standardized steps allow for more flexible human resources, as input, output, and data-processing of different surveys have similar characteristics. For instance, a survey ‘organizer’ can make changes and add validations directly to the form, a task that used to be carried out by a computer scientist. Further, since all surveys can be organized in a similar way, the daily data transfers can be similarly processed and stocked in a library. Since all of these libraries contain similar tables with identical variables and similar structure, the process of accessing data of a variety of statistics is simplified.

3.3Use of mobile devices

Nowadays, offering respondents the option to complete surveys online is standard practice in household and establishment surveys alike [6, 65]. However, with the proliferation of mobile devices (e.g. smartphones, tablets), survey designers have to ensure their web surveys are compatible, and that the use of a mobile device doesn’t harm response rates or data quality. Previous research has found that unit and item nonresponse rates tend to be higher on smartphones compared to PCs and tablets, and that measurement error tends to be higher due to visibility issues related to small screens [66]. However, other research finds that differences in measurement error tend to be small and respondents using smartphones tend to provide similar responses to those responding via a tablet or PC [67].

Business surveys have not faced the same urgency as household surveys to optimize their web surveys for mobile devices, generally because the response process tends to a) rely on records, b) involve multiple individuals in completing the survey request, and c) be lengthy and complicated to complete. Businesses are often assumed to use desktop and laptop machines, though the empirical evidence supporting this assumption is lacking. Morrison et al. [68] shed light on this issue by reporting device usage in five establishment surveys across three U.S. government agencies. Using paradata, the researchers find that a very small percentage of establishments who respond to a web survey do so through a smartphone. The highest rates of smartphone usage among web survey respondents were observed for the National Agricultural Statistics Service (NASS)’s 2017 Census of Agriculture (COA; 6 percent) and the 2018 June Crops Agricultural Production Survey (2 percent), respectively. This was followed by the Bureau of Labor Statistics (BLS)’ Annual Refiling Survey (1.4 percent) and Job Openings and Labor Turnover Survey (0.5 percent), and the National Center for Science and Engineering Statistics (NCSES)’ Higher Education Research & Development Survey (0.2 percent). Interestingly, the BLS and NCSES surveys have higher percentages of web responses, but lower percentages of mobile responses, compared to the NASS surveys. The finding suggests that the prevalence of mobile response depends on the population; for example, NCSES respondents, located at colleges and universities, are contacted at their offices and likely to be at their desks when they receive the survey request. Additional findings from the NASS COA survey show a lower rate of breakoffs for phone and tablet (both 0.2 percent) compared to desktop machines (1.6 percent), lower average session durations for phone and tablet devices compared to desktops, and higher rates of device switching from mobile devices to desktops than the reverse sequence among respondents with two or more sessions.

Mobile devices are also used by survey organizations for collecting responses and other administrative information in the field. Cadena [69] reports a novel use of mobile devices for updating the Mexican Statistical Business Register and georeferencing establishments. One of the key derivatives of the Business Register is the National Statistical Directory of Economic Units (DENUE), a highly demanded data product which consists of a complete listing of all businesses in the country, uploaded in a geographic information system (GIS). The DENUE contains detailed data on every business unit concerning identification, precise location, contact, establishment size, and activity they carry out. With these characteristics, users are able to filter the subuniverse of interest and augment it to additional layers of geographical information using digital cartography.

Since 2004, the Mexican Economic Census, conducted every five years, has experimented with the use of mobile devices to locate and collect information from businesses. Interviewers were equipped with a Personal Digital Assistant (PDA) for 10 percent of geographical areas in 2004 and for all in 2009. In 2014, PDAs were replaced with a subnotebook (changeable to a tablet), which included the most recent directory containing over five million economic units, questionnaires, digital cartography, and satellite images, among other tools. These devices allowed for updating in real-time the status and location of every business unit in the digital cartography and DENUE. The use of mobile devices led to an estimated cost savings of about 20 percent compared to previous data collections on paper because of multiple efficiencies and savings in manual labor costs [69].

3.4Transformation: Content simplification and harmonization

Setting up an integrated system requires standardization and harmonization [51]. Harmonizing units and content is at the core of a “respondent-centric approach” at the U.S. Census Bureau [70]. The unit problem has attracted more research attention only recently [5, 71]. The U.S. Census Bureau first concluded that for the 52 largest, high impact businesses, one reporting unit does not meet the needs of all businesses researched. In the next step, a “complexity score” was created for each multi-unit business based on the number of establishments, industries (unique 8-digit NAICS), states, and tax reporting entities. The research concluded that the top one-percent with the highest complexity score represent 20 percent of total burden and requires an account manager. A kind-of-activity unit will be implemented as the harmonized unit by late-2020 for all other multi-unit businesses because 60 percent of these businesses have a single kind-of-activity unit.

Another important aspect is content harmonization, namely harmonizing definitions, survey questions, and instructions. The U.S. Census Bureau has begun with concepts that are common across surveys and most critical for economy-wide statistics (inventory, payroll, sales, control data, expenses) but encountered obstacles, including the failure to fully understand user needs, resistance from subject-matter experts, and lack of funding and resources for testing harmonized content [70].

Transformations that aim at coordination and integration within NSIs are complex [72, 73, 74]. As a case example of what transformation entails, the UK Office of National Statistics is proposing to move away from publishing short-term statistical outputs separate for the retail, motor trade, and wholesale industries, and merge these industries together into a single Distributive Trade output [75]. Qualitative feasibility research conducted with businesses across the three sectors studied how amenable businesses were to harmonized reporting, for instance, whether retail can provide turnover data exclusive of VAT like wholesale; or whether businesses can provide “total retail/wholesale/motor trade turnover” instead of “total turnover”, which could result in missing or misclassified turnover data should businesses have activities outside of their industry classification. Transformation should ensure consistent data at the point of collection, thus reducing the need for later adjustments.

4.1Developments in communicating with businesses

Communicating with businesses about the survey request is a necessary process for delivering questionnaires to sampled units and obtaining timely, complete, and accurate responses. In follow-up to Cox and Chinnappa [3] Snijkers and Jones [6] identified three stages of business survey communication: a) pre-field stage (e.g. establishing contact with selected businesses, informing businesses about the survey and the intended respondent); b) field stage (e.g. sending advance letters, administering reminders, implementing nonresponse follow-up procedures, providing businesses with additional resources such as FAQs, websites, and help desks); and c) post-field stage (e.g. feedback). According to Snijkers and Jones [6] an effective communication strategy does not only provide the necessary instructions about the survey procedure, it also provides a tailored and persuasive means of inducing participation.

Any communication strategy has to primarily convince businesses that the benefits of taking part in the survey outweigh the time, resources, and costs required to complete the task. Although this challenge is somewhat different for mandatory versus voluntary business surveys, the same question applies: what is an effective communication strategy?

Recently, research by several statistical agencies shed some light on modestly researched effectiveness of response enhancement measures and improved communication strategies. For example, in response to the majority of their agricultural surveys becoming mandatory, Statistics Netherlands conducted a pilot study introducing new targeted communication products to improve the image of their agency in the eyes of sampled agricultural businesses [76]. These products included instructional videos, factsheets, and pre-due date reminder cards, and were implemented in the pilot in the same way for the newly mandatory surveys and a reference survey that remained voluntary. For the mandatory surveys the response rate improved significantly (from 50 to 75 percent), while for the voluntary reference agricultural survey the response rate remained at the initial low level (of about 50 percent). Even though this was not a rigorously designed experiment, Houben et al. [76] concluded that the communication products failed to increase the response rate for the voluntary survey, and the legal obligation must have had an effect. A second pilot study investigated the effectiveness of including additional information about the approaching enforcement procedure in a reminder letter for the mandatory 2017 Research and Development Survey. This contributed to a 6-percentage point higher response rate compared to previous years. Another pilot study, implemented experimentally in the non-mandatory Survey on Arts and Culture Education, showed that sending an attractively designed pre-due date reminder card tailored to the target group, was less effective than the standard pre-due date letter: 7.7 percent response rate increase compared to 10.7 percent; though each reminder was effective in increasing the response rate and sending additional reminders was more effective than sending fewer reminders. A general conclusion from these pilot studies is that direct communication measures (like advance, pre-due date, and reminder letters) work best to get response. Additional analyses at Statistics Netherlands showed that these direct measures are indeed effective response enhancement measures [77]. These measures may be seen as a nudge for businesses to complete the questionnaire [78]. The more indirect materials (like factsheets and cards) may be nice to have, but their effect on response rates seems low or negligible, though they could be important for maintaining a good relationship with businesses.

Many NSIs are centralizing and standardizing their data collection [6, 21, 74], while at the same time tailoring their communication to targeted groups of businesses [79, 80]. Statistics Estonia developed an automated notification system and training to facilitate cooperation in all mandatory business surveys to avoid the need to impose penalties for nonresponse [81]. The automated system provides online help resources and notifies establishments about obligations in the following calendar year and approaching deadlines. Statistics Estonia also invites previous nonrespondents and new sample units to attend face-to-face training with survey staff but results so far are disappointing: attendance is very low, and businesses do not follow-up with these training procedures [81].

A move towards improved facilitation for businesses was also implemented by the Italian National Institute of Statistics (ISTAT), which centralized and enhanced their inbound and outbound contact center services in 2016–2017 [82]. The enhanced inbound service provides assistance to responding units in the access and navigation of the data submission portal through a toll-free telephone number and dedicated email address. Telephone is the preferred channel among businesses, representing 75 percent of all inbound contacts. Three levels of assistance are provided with increasing levels of specialization and complexity. The outbound service initiates telephone contacts with the most relevant non-responding units who are registered in the business statistical portal, but who have not yet responded some time before, or a few days after, the punctual deadline. The service provides several functions, including reminding non-responding units to comply before the approaching deadline, offering support for the questionnaire/interpretation of questions, data entry assistance, and insertion of register changes in the appropriate section of the statistical portal. Although costly, the first results seem very promising as the response rates increased considerably and the number of days to get response decreased substantially.

The Polish Reporting Portal is another example of a centralized system that facilitates business response [83]. The system automatically sends three types of announcements (advance pre-notification to all units about the upcoming survey, a pre-due date reminder, and a final reminder to non-responding units), but also offers other functions, including addressing break-offs by outbound messaging to respondents who started but didn’t finish the task, and offering questionnaire support through a centralized telephone service or direct contact with specialized staff responsible for the survey. Every business has to designate one employee as a person in charge of all statistical reporting. The system allows this person to delegate parts of their mandate to other people (e.g. filling in specific parts of a questionnaire). Statistics Poland started developing this portal in 2007 and, as a result of legal changes, electronic submission of data became obligatory in 2009 for all except the smallest businesses. The number of user accounts increased from about 34,000 in 2008 to 871,000 in July 2018; the number of collected questionnaires increased from 320,000 to about 3 million.

4.2Experiments in communication and contact strategies

One of the challenges in assessing the effectiveness of a new or enhanced communication strategy is that they are rarely implemented in a controlled setting. Production usually trumps experimentation and, thus, isolating the effect of a single or multiple design changes can be difficult if the implementation is altered for production reasons. However, recent efforts suggest that the frequency of controlled experiments in establishment surveys is increasing, implementing the idea of embedding research in recurrent business surveys propagated by Willimack et al. [27].

One active area addresses the impact of invitation/reminder modes on the response rate, nonresponse bias, and costs. The U.S. Bureau of Labor Statistics (BLS) conducted a series of experiments to evaluate the email as an invitation mode compared to more expensive paper based invitations by postal mail [84]. In one experiment email and postal mail invitations yielded equivalent overall response rates, although the email group responded at a slower rate than the postal mail group. The email group elicited a higher yield of web responses (about 30-percentage points higher) and led to a 21-percent reduction in costs overall compared to the postal mail.

A second experiment examined the effect of different mode sequences for non-response follow-up in a mixed-mode web/paper survey. Businesses with email addresses on the sampling frame were randomized into three groups, each receiving an email invitation followed by three additional email follow-ups. The email follow-ups were augmented with postal mail for each of the three follow-ups in the first group, for the second and third follow-ups in the second group, and only for the third follow-up in the third group. The first group (fewest email-only/most combined email-paper contacts) yielded the highest response rate overall and was more likely to respond online compared to the other groups. Thus, the results suggest that email should be used as a supplement to postal mail, and not as a replacement.

Sakshaug et al.[85] also found email-only contacts to be ineffective, this time in a web-only survey of businesses conducted at the German Institute for Employment Research (IAB). In their experiment, the invitation and reminder modes were fully crossed with email and postal contacts for businesses with both contact information on the sampling frame. Email addresses were collected in a forerunner survey. The costliest paper invitation-paper reminder sequence yielded the highest response rate and lowest nonresponse bias overall. The next-best sequence was email-paper followed by paper-email and, lastly, email-email, which yielded the lowest response rate and largest nonresponse bias overall. The study also found that administering supplementary paper contacts (invitation and reminder) to the units with invalid email addresses was effective for increasing response rates and reducing nonresponse bias. For businesses without an available email address, the authors tested a strategy of requesting email addresses through a prenotification letter for the purpose of emailing the survey invitation. However, the return rate was very low (8 percent) and only about 40 percent of those businesses who provided an email address actually participated in the web survey. Thus, compared to a reference paper-only contact without prenotification letter, the email address request was ineffective from a response rate, nonresponse bias, and cost perspective.

Providing incentives to survey units to activate the social exchange process and increase the perceived trust of the survey organization [86, 87], is another design feature which is often embedded within a communication strategy to induce cooperation, particularly in household surveys. Although incentives do not need to be monetary in nature to be effective, they should be viewed as having some value and interest to the business. A case example is Statistics Portugal, which delivers regular Personalized Feedback Reports to all responding establishments [51, 88]. These reports contain a mix of individual and aggregate statistical data to benchmark individual companies against the sector or economy based on the collected data. The report achieves a high level of value recognition from businesses. Whether such incentives produce higher participation rates is an open question. Statistics Netherlands experimented with providing an unconditional incentive in the form of a small folder with the main results of the previous year’s survey enclosed in the invitation letter [76], but the results indicated that the folder did not improve response rates.

Willimack [89] reports the results of several experiments conducted at the U.S. Census Bureau to aid the development of an effective communication plan for the mandatory, web-only 2017 Economic Census. The experiments, conducted in several annual and sub-annual surveys, examined variations in the type, timing, and sequence of contacts (advance notice, certified vs non-certified mail, earlier due date reminder, accelerated nonresponse follow-up letter; combination of earlier due date reminder and accelerated follow-up), optimal targeting of escalation techniques under adaptive design scenarios (targeted allocation of industries with low response rates), envelope appearance and labeling (ink color, envelope size, pressure vs. non-pressure sealed), and alternative motivation strategies. The separate experiments generally found support for the use of pre-due date reminders, red ink on the envelope to emphasize due dates and past due notices, pressure-sealed envelopes, accelerated nonresponse follow-ups, certified mail for later follow-up mailings, targeted selection of units for certified follow-up mailings. Little (or no) support was found for advance notices, half-page (vs. standard) size envelopes, flyers inserted in mailings with motivational messages, and subsampling without targeting. An open question, which the authors acknowledge, is whether the effectiveness of these individual (and separately tested) design features is likely to translate if strung together in a holistic communication strategy.

Instead of testing individual design features across separate surveys, the UK Office of National Statistics (ONS) took an enhanced holistic approach by applying several behavioral “nudges” to help businesses understand the purpose and expectation of the requested survey participation [90]: reciprocity (a prenotification letter and FAQ), messenger effect (a helpful and informal voice in the prenotification letter and an official and monitoring voice in the invitation letter), hassle factor (a simplified message and an illustrative 3-point checklist diagram), head start (pre-checking the first box of the checklist diagram as “already done!”), “make it timely” (highlighting what information is required and when), and “getting something back” (an infographic based on previous survey results relevant to the business and a big thank you note).

The intervention was implemented in the Monthly Business Survey for a random sample of businesses in the construction sector with fewer than 20 employees for newly selected sample units with the “business-as-usual” control group. The results were promising as this communication plan yielded a higher rate of responses before and after the punctual deadline compared to the control group. As with any holistic approach, however, it is unclear whether the improved response rate is driven by a small set of features or all of them.

The U.S. Bureau of Labor Statistics (BLS) tested whether an existing data collection structure of the Annual Refiling Survey, a web survey that is very fast at collecting data on every site’s main business activity(ies) and location from a large sample, could be leveraged to conduct a second “piggyback” survey, especially to reach small target populations [91]. After completing the first survey, respondents were shown a transition page and asked to complete a few additional survey questions that took about two minutes to complete. These questions asked whether the respondent could (or not) report data on several topics (e.g. the current number of job openings, total sales, the top three revenue producing products or services) and about their relationship to the sampled company and department. In one test, a proportional random sample of single-site companies was drawn from the existing survey, mirrored the standard procedure of two email invitations whenever an email was available and two postal mails to all units, and yielded an overall 43 percent response rate with only a slight difference between emailed and mailed invitations. An additional sample was drawn from the population not included in the initial survey, which yielded a 19 percent response rate, to evaluate the possibility of getting a sample representative of the whole US economy. The results of the piggyback survey are thus promising but more research is needed to understand the types of questions that could be asked and the impact on subsequent response.