Expert Elicitation and Its Role in RBI Daniel Engeljohn, Ph.D. Office of Policy, Program, and Employee Development, FSIS, USDA It’s use: For ranking the relative public health risk of FSIS inspected products Goal of Risk Based Inspection „ To allocate FSIS resources in a manner that best protects public health „ Estimations of the contribution of a particular food to a subsequent human illness is difficult „ There are several ways to attribute food to illness; each has its own strengths and caveats Methods for Attribution „ Three different methods for illness attribution. „ Expert elicitation „ Predictive models „ Epidemiological data analysis „ By comparing the results of multiple methods FSIS can improve the final rankings. Please Note:: Many of the following ideas were summarized by the food attribution working group and published in Emerging Infectious Diseases, Michael B. Batz, Michael P. Doyle, J. Glenn Morris, Jr., John Painter, Ruby Singh, Robert V. Tauxe, Michael R. Taylor, and Danilo M.A. Lo Fo Wong, for the Food Attribution Working Group Expert Elicitations FSIS recently conducted two independent expert elicitations to define the relative risks posed to public health by processed meat and poultry products „ Two elicitations: „ One with 22 experts and the other with 12 experts from industry, academia and public health sector „ Both generally defined 24 meat and poultry product categories „ Both ranked these foods to relative risk to public health „ Strengths „ Can be performed even when there is little available data „ Can help to resolve discrepancies between other methods „ Caveats: „ Judgment based „ May be less objective than data driven decisions Predictive Models „ Estimate the public health risk of a food based on a variety of data inputs. By estimating illnesses attributed to each product a ranking of FSIS foods can be determined. „ FSIS has developed predictive models to estimate the number of illnesses attributed to meat, poultry, egg products (including raw and ready-to-eat products). „ Strengths: Reliance on human dose-response curves or surveillance data Modeling post production mitigations (i.e. consumer handling and „ Objective „ Based on observable data „ Can help identify data needs „ Caveats: „ can create uncertainty in predictions of human illness „ cooking) is complex Predictive Models: Molecular Subtype Data „ Molecular markers for microbial source tracking „ PFGE, serotypes, subtypes, phage types, and genotype assays „ Denmark Salmonella process „ A working model for how salmonella serotypes are used to attribute salmonellosis to particular foods „ Caveats: „ Some cases of salmonellosis do not have a serotype or subtype strongly associated with a single food or species of source animal Predictive Models: Estimated Human Salmonellosis Cases from Meat, Poultry, and Eggs, 1998-2003* „ Human case data, by serotype „ CDC Public Health Laboratory Information System, 1998 – 2003 „ Food prevalence data, by serotype „ FSIS in-plant samples of beef, ground beef, chicken, turkey, pork, and processed egg products, 1998 – 2003 „ Pennsylvania Pilot Project samples of shell eggs, ca. 1995 „ Food consumption data „ USDA Economic Research Service, 1998 – 2003 0 1000 2000 3000 4000 5000 6000 7000 8000 1998 1999 2000 2001 2002 2003 Intact Beef Ground Beef Chicken Turkey Pork Eggs Egg Product * Approximately 28,000 to 30,000 culture- confirmed salmonellosis cases per year, 1998-2003 Attribution of the Predicted Median Cases of Listeriosis for US Population on per Serving and per Annum Basis ƒ Factors in volume production, product risk, plant sanitation practices, and historical culture results from product, food contact surfaces, and the production environment ƒ Model is updated on a monthly basis using current culture results and ranks all eligible RTE establishments according to Lm risk for the purpose of sample allocation ƒ Model can be used to predict the total number of illnesses expected annually for each level of sanitation 0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 Prevalence or Cases Cases x 1000 % Prevalence Cases x 1000 1.262 1.275 1.116 1.430 0.840 0.942 0.730 % Prevalence 1.450 1.316 1.027 0.760 0.545 0.641 0.612 2000 2001 2002 2003 2004 2005 2006 The Lm model predicts which RTE establishments are likely to pro duce contaminated product based on prevalence in FSIS testing Predictive Models: Epidemiological Data Analysis „ Uses actual human illness events clinically diagnosed and attributed to a food vehicle by epidemiological evidence „ Foodborne Outbreak Reporting System (eFORS) „ CDC FoodNet (active surveillance of foodborne illness) „ Strengths: „ Objective „ Based on observed human illnesses „ Caveats: „ Foodborne illness grossly under diagnosed „ Uncertainty about the coefficients used to compensate „ Interviews, surveys, and case control studies used to attribute clinical cases to foods rely on human reporting and can be “anecdotal” „ Data sets tend to be small. (see first caveat). - Step Input Salmonella Campylobacter Data Source 1 Incidence 14.4/100,00 12.6/100,000 FoodNet Annual Report for 20036 2 Population estimate 2003 290,788,976 290,788,976 US Census Bureau16 3 Underreporting multiplier 38 38 Mead et al.7 4 Foodborne fraction 0.95 0.80 Mead et al.7 5 Poultry attribution fraction 0.3351 0.6936 Food Safety Research Consortium17;18 6 Young poultry fraction 0.838 0.838 ERS19 7 Total illnesses 1,591,197 1,392,297 Step = (1)(2)(3) 8 Total foodborne illnesses 1,511,637 1,113,838 Step = (4)(7) 9 Total foodborne illnesses from poultry 498,840 772,558 Step = (5)(8) 10. Total foodborne illnesses from young chickens 424,389 647,251 Step = (6)(9) 11. Costs per illness $1800 $610 ERS19 12. Total Cost of illnesses from product and pathogen $759 million $390 million Step = (10)(11) Estimating illnesses due to young chickens using epidemiological data from the United States 2003 Epidemiological Data Analysis: Summary: Current Status „ In the immediate future, FSIS intends to base estimates of food item inherent risk on the results of the first expert elicitation „ The risk rankings in RBI are currently based on the results of two expert elicitations with close agreement „ FSIS will continue to combine expert rankings with other information (e.g., volume) to arrive at an overall establishment’s inherent risk Summary: FSIS Next Steps „ FSIS will continue to assess expert elicitation results with current knowledge from epidemiological studies and predictive modeling via risk assessments regarding attribution of foodborne disease risk „ FSIS is further assessing whether/how to collapse/expand the 24 food categories from the expert elicitations into categories more closely informed by other work (e.g., the Center for Science in the Public Interest outbreak groupings) Summary: FSIS Next Steps (continued) „ FSIS will assess the use of epidemiological data to incorporate severity of illness „ FSIS will evaluate methods to incorporate cost of illness into expert risk rankings „ FSIS will conduct sensitivity analyses to ensure appropriate weights are given to components of RBI, including the expert risk rankings „ FSIS will update the algorithm, its components, and the expert risk rankings, as appropriate (e.g., to incorporate emerging pathogens, as warranted) „ FSIS will maintain ongoing communication with all stakeholders, state and local partners in order to maintain a shared understanding of attribution