Elastic Language in Persuasion and Comforting: A Cross-Cultural Perspective [1st ed. 2019] 978-3-030-28459-6, 978-3-030-28460-2

Table of contents :
Front Matter ....Pages i-xvi
Introduction (Grace Zhang, Vahid Parvaresh)....Pages 1-9
Theoretical Foundations (Grace Zhang, Vahid Parvaresh)....Pages 11-60
Methodology (Grace Zhang, Vahid Parvaresh)....Pages 61-82
Elastic Language in the Chinese Data (Grace Zhang, Vahid Parvaresh)....Pages 83-161
Elastic Language in the English Data (Grace Zhang, Vahid Parvaresh)....Pages 163-229
Comparison Between the Chinese and English Findings (Grace Zhang, Vahid Parvaresh)....Pages 231-257
Elastic Language and Impact Factors (Grace Zhang, Vahid Parvaresh)....Pages 259-271
Conclusions (Grace Zhang, Vahid Parvaresh)....Pages 273-279
Back Matter ....Pages 281-297

Citation preview

Nuclear Engineering and Design 355 (2019) 110312

Contents lists available at ScienceDirect

Nuclear Engineering and Design journal homepage: www.elsevier.com/locate/nucengdes

The role of verification & validation process in best estimate plus uncertainty methodology development

T

J. Zhang Tractebel (ENGIE), Boulevard Simon Bolivar 34-36, 1000 Brussels, Belgium

A R T I C LE I N FO

A B S T R A C T

Keywords: Best estimate plus uncertainty (BEPU) Industrial practices Regulatory requirements Technical standards Uncertainty quantification (UQ) Verification and validation (V&V)

In the process of a best estimate plus uncertainty (BEPU) methodology implementation for nuclear power plant safety analyses, the verification and validation (V&V) process including the uncertainty quantification (UQ) of the used computer codes and plant models plays an essential role. From the technical point of view, a BEPU methodology must be based on fully verified and validated codes and models, with quantified key model uncertainties. From the regulatory point of view, the adequacy of the simulation codes and plant models for the intended BEPU application must be assessed through the VVUQ process. These high level requirements of VVUQ result in high-cost for BEPU methodology development, preventing wider applications. Pragmatic, graded applications and practices of VVUQ in the BEPU methodology development are needed to allow to take full benefit of BEPU applications.

1. Introduction Since 1990’s, best estimate plus uncertainty (BEPU) methodology for nuclear power plant design basis accident analysis (in particular, loss-of-coolant accidents or LOCAs) has being developed and applied to meet the increasing technical and regulatory requirements for licensing new plant design or major plant modifications, power uprate, new core and fuel design, as well as for support to plant operation (Wilson, 2013). According to International Atomic Energy Agency (IAEA) general safety requirement (GSR) Part 4 Requirement 18 (IAEA, 2016); “Any calculational method and computer codes used in the safety analysis shall undergo verification and validation”. IAEA specific safety guide SSG-2 (IAEA, 2019) recommends that “the methods used in the computer codes for the calculation should be adequate for the purpose. The requirements for the validation and verification depend on the type of application and purpose of the analysis.” The BEPU approach “allows the use of a best estimate computer code together with more realistic, that means best estimate and partially most unfavourable, initial and boundary conditions. However, in order to ensure the conservatism required in analysis of design basis accidents the uncertainties need to be identified, quantified and statistically combined. Availability of systems is usually assumed in a conservative way.” The BEPU approach “contains a certain level of conservatism and is at present accepted for some design basis accident and for conservative analyses of anticipated operational occurrences.”

Finally, IAEA provides a guidance on uncertainty evaluation for best estimate safety analysis for nuclear power plants (IAEA, 2008). The advantages of the BEPU methodology are:

• It allows to realistically and accurately simulate phenomena that govern the accidents and transients of interest; • It is efficient to determine the plant limiting conditions in the integrated multi-physics analysis environment; • It allows to better assess the actual margins and to assure that nocliff-edge effects may exist; • It is transparent and apt for quality assurance. The BEPU might be thus the preferred methodology for licensing safety analysis and simulating plant operation and accident conditions. However, the development and application of BEPU methodology requires a higher-level requirement on the verification and validation, and uncertainty quantification (VVUQ) of the used calculational method and simulation model. This may result in high-cost for BEPU implementation, and hence prevent the industry to take full benefit from the BEPU applications. Pragmatic and graded applications and practices of VVUQ are needed. This paper will focus on the overview of the relationship between the verification, validation and uncertainty quantification (VVUQ) and the BEPU methodology development processes. After a short review of the applicable technical standards and procedures (§2), the regulatory requirements and recommendations are reviewed (§3), followed by

E-mail address: [email protected]. https://doi.org/10.1016/j.nucengdes.2019.110312 Received 17 December 2018; Received in revised form 28 June 2019; Accepted 21 August 2019 0029-5493/ © 2019 Elsevier B.V. All rights reserved.

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

Nomenclature AHP AIAA ANS ASME BIC BE BEPU CCVM CET CFD CSNI ECCS EM EMDAP FSAR GSA GSR IAEA IB IET IP ITF IUQ LOCA

LWR MCDM NEA NPP OECD

Light-Water Reactor Multi-Criteria Decision Making Nuclear Energy Agency Nuclear Power Plant Organization for Economic Co-operation and Development PCT Peak Cladding Temperature PDF Probability Density Function PIRT Phenomena Identification and Ranking Table PMI Predictive Maturity Indexes PREMIUM Post-BEMUSE Reflood Models Input Uncertainty Methods PWR Pressurized Water Reactor Q-PIRT Quantitative Phenomena Identification and Ranking Table RCS Reactor Coolant System SA Sensitivity Analysis SAPIUM Systematic APproach for model Input Uncertainty quantification Methodology SET Separate Effects Test SM Simulation Model SRQ System Responses Quantity SSG Specific Safety Guide UA Uncertainty Analysis USNRC United States Nuclear Regulatory Commission UQ Uncertainty Quantification VV Validation and Verification

Analytical Hierarchical Process American Institute of Aeronautics and Astronautics American Nuclear Society or American National Standards American Society of Mechanical Engineering Boundary and Initial Conditions Best-Estimate Best-Estimate Plus Uncertainty CSNI Code Validation Matrix Combined Effect Tests Computational Fluid Dynamics Committee on the Safety of Nuclear Installations Emergency Core Cooling System Evaluation Model Evaluation Model Development and Assessment Process Final Safety Analysis Report Global Sensitivity Analysis Generic Safety Requirements International Atomic Energy Agency Intermediate Break Integral Effects Test Inverse Propagation Integral Test Facility Input Uncertainty Quantification Loss Of Coolant Accident

and application. It is clear that verification deals with the relationship between the physical model and the simulation model and that validation aims to quantify the accuracy of the simulation model based on comparisons of physical experiments with calculation outcomes from the simulation model. The simulation model can only be used to predict the behaviour of the intended application after the V&V and the UQ. Therefore, V&V can be considered as the foundation of the BEPU methodology, as it is essential to answer the critical question: How confident should be we with the prediction results of the developed simulation model for the application? Parallel and close to this V&V process, the uncertainty in each of the activities should be quantified in the uncertainty analysis or quantification (UA or UQ) process. Uncertainty quantification is a process of implementation of a set of tools and formalisms for quantifying sources and subsequent propagation of uncertainty from all sources that affect the reliability of the output. The development of technical standards and procedures for code

industrial applications and good practices (§4), conclusions and perspectives (§5). 2. Technical standards and procedures The whole process of development and application of a BEPU methodology for the intended application consists of the following main activities and processes, as summarized in Fig. 1:

• Modelling: • • • • •

Development of physical models required for an intended application (e.g., analysis of a transient or accident in a pressurized water reactor using system and sub-channel thermal hydraulic codes), Simulation Model: Implementation of the physical models in the computer code and development of the simulation model (note that in this paper, the Simulation Model is referred to the computer code, nodalization, model option and algorithms to approximate the solution of physical equations), Scaling and Experimentation: Establishment of the experimental database for simulation of the physical behaviour for the intended application, Verification and Validation (V&V): Assessment of the correctness of the computer code and quantification of accuracy of the simulation model, Prediction: Use of the verified and validated simulation model to predict (or extrapolate) the expected “best estimate” (BE) response of the system for the application, and assessment of adequacy or predictive capability, Uncertainty analysis: Quantification of uncertainties of all the above activities in the whole development process (e.g., due to approximation, model deficiency, scaling distortion, measurement uncertainty, etc.) and application process (initial and boundary conditions, etc.).

The above modelling, verification and validation process is consistent with the concepts proposed by LANL (Thacker et al., 2004), except that the concept of reality is replaced by physical experiments

Fig. 1. The verification and validation process in relation with the BEPU methodology. 2

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

The solution verification focuses on estimating the numerical solution error, assuring the input and output data for the problem of interest, which allows quantification of errors introduced during application of the code to a particular simulation (e.g., performing a grid or time convergence study by successively refining the mesh or time step until a sufficient level of accuracy is obtained). The code verification focuses on the identification and removal of errors in the source code and numerical algorithms, and improving software using software quality assurance practices. Validation can also be further divided in three levels, as shown in Fig. 3 and proposed by Oberkampf and Barone (Oberkampf and Trucano, 2007):

verification, validation and uncertainty quantification (VVUQ) are under constantly evolution since the publication of the first ANS standard in 1987 (ANS, 1987). Of particular interest to BEPU methodology using thermal hydraulic codes are the AIAA (AIAA, 1998) and ASME (ASME, 2009) guides for Computational Fluid Dynamics (CFD) and heat transfer, and the ongoing development of the ASME standard for nuclear thermal-fluids software (Harvego et al., 2011). Some slight differences in the definition of the computer model V&V process in various standards, and the best-fit one for our use are those in the AIAA guide (AIAA, 1998):

• Verification: the process of determining that a (computer or simu•

lation) model implementation accurately represents the developer’s conceptual description of the model (i.e., mathematical model) and the solution to the model. Validation: the process of determining the degree to which a (computer or simulation) model is an accurate representation of the real world from the perspective of the intended uses of the model.

1. Quantification of the accuracy of the simulation model results through comparing the system response quantities (SRQs) of interest with experimentally measured SRQs. The simulation model accuracy is quantitatively estimated at the conditions where experimental data is available, and the validation is addressed by defining and computing validation metric(s) associated to SRQs; 2. Use of the simulation model, in the sense of interpolation or extrapolation of the simulation model, to make predictions for conditions corresponding to the intended use of the simulation model; 3. Determination of whether the estimated accuracy of the simulation model results, for the conditions of the intended use, satisfies the accuracy requirements specified for the SRQs of interest.

Briefly speaking, verification is the assessment of the accuracy of the solution to a simulation model by comparison with known solutions (i.e., how well are mathematical formulas represented in computational model?). Validation is the assessment of the accuracy of a simulation model through comparison with experimental data (i.e., how well do code calculation outputs represent reality?). In other words, verification is primarily a mathematics issue; validation is primarily a physics issue (Oberkampf and Trucano, 2002). The simulation model V&V is related to, but fundamentally different from software V&V. Code developers of computer programs perform software V&V to ensure code correctness, reliability, and robustness. The code users, however, seek to apply credible predictive models based on fundamental physics of the problem that are solved in all applications. The adequacy of the predictive models need to be assessed according to simulation model V&V guidelines and procedures. The expected outcome of the simulation model V&V process is thus the quantified level of agreement between experimental data and simulation model calculation, and most importantly the accuracy of the simulation model. Verification can be further divided in two types of activities as shown in Fig. 2, as proposed by Oberkampf et al. (2004): Numerical algorithm (or solution) verification and software quality engineering practices (or code verification).

The last two levels of validation concern the simulation model predictive capability assessment. It should use the assessed simulation model accuracy (first level) as input, and incorporate the following additional assessment (Oberkampf and Trucano, 2007): a) additional uncertainty estimation resulting from interpolation or extrapolation of the simulation model beyond the existing experimental database to future applications of interest; and b) comparison of the accuracy requirements needed by a particular application relative to the estimated accuracy of the simulation model for that specific extrapolation to the applications of interest. If the calculated validation metric result meets the accuracy requirements, the simulation model is considered as adequate to the intended application (i.e., used for BEPU methodology development). If the accuracy requirements are not met, one may need to either update (or re-

Fig. 2. The two types of model verification (Oberkampf et al., 2004). 3

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

Fig. 3. The three level of validation process (Oberkampf and Trucano, 2007).

to real plant conditions that are not totally available in experiments is a difficult task. Some specific methods, such as the predictive capability maturity model (Oberkampf and Trucano, 2007; Oberkampf et al., 2007; Rider et al., 2015) may be helpful, but not yet mature and practical enough to be applied to nuclear power plant safety analyses using system thermal hydraulic codes. Note also that for certain nuclear thermal hydraulic codes, the code developers usually perform extensive developmental code assessment (V&V). However, the code users (or model developers) should also perform independent V&V for the selected codes for the intended applications, and a graded approach of VVUQ may be applied in this case (see Section 4).

calibrate) the simulation model or improve or add experimental measurements (Oberkampf and Barone, 2006). This means a new round of VVUQ. A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing was proposed by Oberkampf and Roy (Roy and Oberkampf, 2011). The framework includes the following key steps for uncertainty quantification: 1) identification of all sources of uncertainty associated to model inputs, numerical approximation and model form, 2) the characterization of uncertainties including quantification of model input uncertainties, 3) the estimation of the uncertainty due to numerical approximation (e.g., by elimination or estimation of code and solution verification errors), 4) propagation of quantified input uncertainties through the simulation model to obtain uncertainties in the SRQs, 5) the estimation of the model form uncertainty, and 6) the determination of the total uncertainty in the SRQs at the application conditions of interest (e.g., extrapolation to application conditions of interest).

3. Regulatory requirements and recommendations In 1988, the USNRC revised the regulatory requirements for LOCA safety analysis, which allows use of a realistic evaluation model with uncertainty quantification (i.e., BEPU analysis methods). Regulatory Guide 1.157 (USNRC, 1988) describes acceptable models, correlations, data, model evaluation procedures, and methods for meeting the specific requirements for a realistic calculation of ECCS performance during a LOCA. In order to demonstrate the implementation of such BEPU methodology, the USNRC technical program group has developed a Code Scaling, Applicability and Uncertainty evaluation (CSAU) approach (USNRC technical program group, 1989). The CSAU is a structured, traceable and practical approach to quantify uncertainty. It addresses in a unified and systematic manner questions related to:

In VVUQ procedure, the validation process can be performed by way of comparison of simulated results with available experimental measurements. It is therefore performed at the conditions where experimental data are available and the validation is addressed by defining and computing validation metrics associated to SRQs. The final objective is to extrapolate the prediction results and total uncertainty to the application conditions of interest for where no experimental data is available (step (6)). The extrapolated uncertainty is included in the prediction of the computational model at the conditions of interest. It should be noted that the above VVUQ concepts and procedures are derived from development of software applicable to design of missiles and their equipment where impact of empiric and semi-empiric correlations are not pronounced so far as in nuclear power plant system thermal hydraulics modelling. Therefore, some of the elements and steps should be adapted for applications to the latter case (see §4). In particular, the extrapolation of the simulation model and its uncertainty

• the scaling applicability of best-estimate code, • the applicability of the code and plant model to scenarios of interest to NPP safety studies, and • the evaluation of uncertainties in calculating the SRQs of interest,

when the code is used to perform a calculation for a specified scenario and NPP design.

It was recommended by the group of experts as an acceptable approach to develop a BEPU methodology for best estimate LOCA analysis 4

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

that complies with the USNRC regulatory guide RG-1.157 (USNRC technical program group, 1989). As shown in Fig. 4, the CSAU methodology consists of 14 steps, organized into 3 major elements:

•

• Element 1 – Requirements and code capabilities: scenario modelling •

requirements are identified in a Phenomena Identification and Ranking Table (PIRT) and compared against code capabilities to determine the codes applicability to the particular scenario and to identify potential limitations. Element 2 – Assessment and Ranging of Parameters: code capabilities to calculate processes important to the scenario are assessed against experimental data to determine code accuracy and scale-up

capability and to specify ranges of parameters variations needed for sensitivity studies. Element 3 – Sensitivity and Uncertainty Analysis: the effects of individual contributors to total uncertainty are obtained and the propagation of uncertainty through the transient is properly determined.

The VVUQ process is detailed in Element 2 & 3, which is the key part of the CSAU methodology. The CSAU approach described above was recently endorsed as an acceptable structured Evaluation Model Development and Assessment Process (EMDAP) (USNRC, 2005). It describes a process that the USNRC considers acceptable for use in development and assessment of

Fig. 4. Code scaling, applicability and uncertainty (CSAU) evaluation methodology (USNRC technical program group, 1989). 5

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

evaluation models (i.e. collection of codes and procedures) that may be used to analyse transient and accident behaviour within the design basis of a NPP. The EMDAP essentially follows the same main principles of the CSAU methodology described above, however with more emphasis to the evaluation model development, verification and validation, uncertainty quantification, and application process. As shown in Fig. 5, the EMDAP consists of 4 elements and 19 steps:

• •

• Element •

1 – Establish Requirements for Evaluation Model Capability: In this first element, the exact application envelope for the evaluation methodology is determined. Furthermore, the importance of constituent phenomena, processes, and key parameters within this envelope are agreed upon. Element 2 – Develop Assessment Base: In this second element, the

purpose is to provide the basis for development and assessment of the evaluation methodology. This includes acquiring appropriate experimental data relevant to the scenario being considered and ensuring the suitability of experimental scaling. Element 3 – Develop Evaluation Model: In this third element, the evaluation model is developed and organized to meet the requirements defined in Element 1. Element 4 – Assess Evaluation Model Adequacy: In this fourth and last element, the adequacy and capability of the evaluation model are assessed and documented. The final step of this last element, it is to be decided whether the code is adequate or not.

The VVUQ process is detailed in Element 4, which is the key part of the EMDAP approach.

Fig. 5. Evaluation model development and assessment process (EMDAP) (USNRC, 2005). 6

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

selected validation matrices (SETs or combined effect tests); 4. To assess the scalability (biases or distortion) of the simulation model and uncertainty analysis by comparing with the IETs; 5. To assess the adequacy of the simulation model and uncertainty analysis for real plant applications.

Last but not least, USNRC Software quality assurance guides are detailed in two NUREG reports (USNRC, 1993; USNRC, 2000). An example of application of these guidelines to the USNRC thermal hydraulic code TRACE can be found in (Bajorek et al., 2015). In France, the safety authority ASN recently published recommendations for the qualification of scientific computing tools used to verify compliance with the safety criteria associated with the first fuel barrier (ASN, 2017). In the U.K., the safety authority ONR published recommendations for the validation of computer codes and calculation methods (ONR, 2016). In Germany, the use of best-estimate codes is allowed, combined with conservative initial and boundary conditions, and efforts are being made to include uncertainty evaluation in the regulation with a revision in the German Nuclear Regulation. The Reactor Safety Commission is also recommending a BEPU LOCA licensing analysis (Glaeser, 2008).

For practical application purpose, the VVUQ were often performed on a selected or improved code (such as the generic purpose codes RELAP, TRACE, CATHARE or ATHLET) or a specifically developed code (such as WCOBRA-TRAC, MARS), for a well-defined scenario (e.g., large-break LOCA) of a particular reactor design (e.g., PWR 2-, 3- or 4loops) and safety injection system (e.g., Cold leg injection or combined injection, upper plenum injection), and for a particular fuel design (14 × 14, 15 × 15 or 17 × 17, 8–14 ft.). 4.2. Good practices

4. Industrial applications and good practices The good practices for VVUQ as proposed and used by the industry (Prošek and Mavko, 2007; Martin, 2016) or as developed by the research organizations (Rider et al., 2010; Pourgol-Mohamad et al., 2011; Unal et al., 2011; Stoots et al., 2012; Petruzzi and D’Auria, 2016; Radaideh et al., 2019b) are summarized in the following sub-sections.

4.1. Industrial applications The early USNRC approved BEPU large-break LOCA evaluation methodologies (EMs), such as the Westinghouse’s BELOCA (Young et al., 1998) and ASTRUM (Frepoli, 2008) and the Framatome’s RLBLOCA (Martin and O’Dell, 2005), followed the CSAU approach (USNRC technical program group, 1989). Similar methodologies have been developed and approved in other countries, such as ESM-3D in France (Sauvage et al., 2005), KREM in South Korea (Ban et al., 2004), statistical methods in Germany (Kozmenkov and Rohde, 2013, 2014; Seeberger et al., 2014), and the BEPU LOCA method in Spain (Queral et al., 2015). The recently USNRC approved BEPU LOCA evaluation methodologies, such as the Westinghouse’s FSLOCA (Frepoli and Ohkawa, 2011) and the Framatome’s RLBLOCA Rev. 3 followed the EMDAP approach (USNRC, 2005). Most recently, the BEPU analysis capability is established and demonstrated using the RELAP5-3D code, in response to the Nuclear Regulatory Commission’s proposed new 10 CFR 50.46(c) rulemaking on emergency core cooling system/LOCA performance analysis (Zhang et al., 2016a,b; 2017). The proposed new 10 CFR 50.46(c) rulemaking imposes more restrictive cladding embrittlement criteria and new analysis methods (refined core modelling, BEPU) are also required to provide a complete characterization of the reactor core margins under LOCA conditions. Since 2000, many BEPU applications have also been made on the non-LOCA transient analyses, although only a few has been approved for licensing purpose (Kawamura and Hara, 2000; Abdelghany and Martin, 2010; Avramova, 2010; Zhang et al., 2013; da Cruz et al., 2014; Pecchia, 2015; Martin, 2016; Brown et al., 2016; Nguyen, 2017; Burns and Brown, 2017; Walters et al., 2018; Radaideh et al., 2019a). Indeed, the application of VVUQ and BEPU approaches have been extended beyond system and subchannel thermal-hydraulics to containment thermal hydraulics, nuclear data, reactor core neutronics, fuel performance, multi-physics, spent fuel criticality safety analysis, etc. In the development of BEPU methodologies, the major activities for the VVUQ process (namely the Element 2 & 3 of CSAU or Element 4 of EMDAP) are:

4.2.1. Assessment of capability and limitations of the simulation model The first step of VVUQ is to assure that the developed or selected code is capable of representing correctly the process and phenomena that have been identified as highly ranked during the PIRT process (Element 1 of both CSAU and EMDAP). This assessment can be performed by a thorough review of the user manuals of the developed or selected code (in particular the basic field equations, code structure and solution method, models and correlations), by comparing with the capability requirements as identified in the former steps. For example, if multidimensional effects and multi-fields are ranked important for transients like large-break LOCAs, the developed or selected code should include these models in an acceptable way. It is essential also to verify that the developed or selected code has been used in the past for similar applications, and identify its limitations. 4.2.2. Assessment of applicability of the simulation model The applicability of the simulation model for the intended application is established by evaluation of the simulation model calculation results against relevant experimental data, as selected in the so-called validation matrices for the intended application (in Element 2 of both CASU and EMDAP). The OECD/NEA CSNI has published extensive validation matrices of separate effect (OECD/NEA, 1993; OECD/NEA, 1993) and integral effect (Aksan et al., 1987) tests (IETs) that provide information on existing experimental data. More recently, a total of 116 thermal hydraulic phenomena have been identified for water cooled reactors including new reactors (Aksan et al., 2018), which cover virtually all LOCA and non-LOCA thermal-hydraulic transients. EPRI (EPRI, 2014) compiled an assessment database and demonstrated a process for ranking the data and mapping it to the thermal-hydraulic phenomena of interest for transients or accidents of interest. For developing and qualifying a simulation model (or input deck) for the test facility or plant, the users of the developed or selected code should follow the detailed guidelines for nodalization and model options provided by the developers. Participation in code user groups and training programs to share experiences with other users is fundamental to reduce the so-called user effect. One key element of the VVUQ process is to qualify the developed nodalization and model options for the intended application (Bucalossi and Petruzzi, 2010; Petruzzi and D’Auria, 2016). In this process, one should check both the geometrical fidelity of the nodalization and the capability of the code nodalization to reproduce the expected transient scenario. Acceptability criteria have to be defined and satisfied during

1. To assess the capability and limitations of the code field equations, closure laws and numerical solutions to simulate the physical phenomena or processes in the systems or components of interest; 2. To assess the applicability of the simulation model (code and input model, including nodalization and model options) to simulate the physical phenomena or processes in the systems or components of interest; 3. To quantify the uncertainty of the simulation model for the important physical phenomena or processes by comparing with the 7

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

uncertainty quantification (IUQ) methods by inverse propagation of the information based on the discrepancy between code simulation and experimental results. The OECD/NEA PREMIUM benchmark has been a valuable exercise on methods of uncertainty quantification of physical model input parameters, and their application to the physical models involved in the PWR LOCA reflooding prediction. A review and comparison of the available IUQ methods used in PREMIUM is given in (Reventós et al., 2016). Different methods and thermohydraulic codes were used within the benchmark (Mendizábal Sanz et al., 2017). Results were more dependent on the quantification methods (Maximum Likelihood inference, Bayesian inference or Coverage method), rather than on the codes employed (RELAP, CATHARE, ATHLET, etc.). Furthermore, the results of quantification showed a strong dependency on topics such as:

the nodalization-qualification. If needed, sensitivity studies on nodalization (i.e., time step size, convergence and solution method) should be performed to improve the quality of code prediction results by demonstrating that a convergent solution has been obtained and providing information on uncertainty resulting from the choice of nodalization and resolution technique. To identify and reduce the effects of inadequate nodalization schemes, the incorrect use of physical models outside their scope, the code options of different physical models, and other sources of avoidable errors in calculations of code, it is essential to check if the results are consistent with the physical models. Post-processing techniques have been developed to verify this consistency, such as the SCCRED tool (Petruzzi and D’Auria, 2016), making it possible to distinguish between the uncertainties in the calculation and the modelling errors due to user effects. For the fully verified and validated generic purpose codes, although the validation matrices have been partly used by the code developers to evaluate and improve system codes, the users should also perform the so-called independent validation, based on one or more integral effect tests that are similar to the intended application, in order to ensure that the modelling techniques used for the test facility model adequately predict the experimental data. A further assessment against separate effects tests that deal with phenomena identified as important for the intended application is also suggested. Finally, cross-code verification (or benchmarking) should be made when the experimental data is not available, which is valuable and useful to explain the difference in the predicted behaviour on the same problem.

• The set of selected SRQs used in the quantification • The set of selected input parameters to be quantified • Selected tests for quantification • The simulation models, which, in general depend on the thermal hydraulic codes being used.

Based on the experience feedback from the previous OECD/NEA PREMIUM benchmark, a systematic approach devoted to model input uncertainty evaluation (i.e. quantification and validation) has been proposed to improve the reliability of the analysis and the confidence on the extrapolation of its results to the NPP case (Baccou et al., 2017). The proposed systematic approach is shown in Fig. 6. The SAPIUM approach consists in the following 5 key elements:

• Element 1 includes the definition of the objectives of the evaluation

4.2.3. Quantification of the model input uncertainty by comparison with SETs and CETs Another key issue of BEPU methodology is the input uncertainty quantification. For any simulation model, there are two types of input parameters: those are not subjective to calibration (boundary and initial conditions, independent variables, etc.), and those are subject to calibration (physical model input parameters, calibration parameters, unknown constants, etc.). The uncertainty bounds for the former category can be obtained from experiments or plant specifications and expert opinion, while the model input uncertainty can be inferred by the input

(e.g., quantify and validate the input uncertainties of the reflooding heat transfer models for application to plant analysis), the selection of a NPP (e.g., a 3-loop PWR) and a scenario (e.g., a cold-leg break LOCA), as well as the SRQs of interest (e.g., peak cladding temperature or PCT) associated with the identified important physical phenomena (e.g., quench front propagation, interphase friction, dispersed film boiling, etc.). The latter is obtained by applying a Phenomena Identification and Ranking Table (PIRT) process,

Fig. 6. The systematic approach to model input uncertainty quantification (Baccou et al., 2017). 8

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

•

•

initially and primarily based on expert judgement (Wilson and Boyack, 1998), and later quantitatively confirmed by the global sensitivity analysis (GSA) method (Luo et al., 2010; Martin, 2011; Yurko and Jacopo, 2012). Element 2 is related to the construction of the experimental database for model input uncertainty quantification and validation that will control the capability of the method to extrapolate its results to real situations. It should be based on available SETs, CETs and IETs but can also require extra experiments if necessary. It includes the assessment of adequacy of an experiment and of completeness of an experimental database. At the end of this step, a ranking between experiments within the database could be performed using the Multi-Criteria Decision Making (MCDM) outranking approaches (as illustrated in (Baccou et al., 2018) or analytic hierarchy process (AHP) (Saaty, 1982). Element 3 is related to the simulation model. It consists in assessing the applicability of the code for modelling the identified important phenomena as well as for modelling the considered SETs/CETs/ IETs. Moreover, this element requires to follow nodalization strategy and model option selection that should be consistent between the

•

experimental facility and similar components in the nuclear power plant. A special attention should be also devoted to the construction of error metrics (to evaluate the accuracy code/experiment) and the definition of a scale of accuracy. Finally, important uncertain model input parameters have to be identified by using the global sensitivity analysis (GSA) methods in this element. Element 4 consists in inferring from the experimental knowledge, the information related to model input uncertainties. The experimental knowledge is here associated with a subset of the database constructed in Element 2 (the remaining subset will be used for model input uncertainty validation). It then requires selecting a set of differences between code calculation and experimental value. Finally, the inference can be performed. Besides the choice of the model input uncertainty quantification methods (FFT, Monte Carlo, Bayesian, …), an appropriate uncertainty modelling for each uncertain input (interval, pdf, possibility,…) should be done by taking into account the real state of knowledge (nature of uncertainty and available information) and by reducing as much as possible extra assumptions. Key questions of this element are also related to the strategy to follow in presence of several experiments (quantification

Fig. 7. General framework for assessing adequacy of a thermal-hydraulic analysis and associated uncertainties in plant safety analysis (Nourbakhsh and Banerjee, 2013). 9

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

4.2.4. Assessment of the scalability (biases or distortion) of the simulation model and uncertainty analysis by comparing with the IETs IET experimental data is valuable source of information for the justification of new evaluation model (or methodology) based on the BEPU approach. IETs should be used to provide an appraisal of the scalability of the simulation model, in specific the interaction among sub-models, compensating errors, predictability of important phenomena and the automated simulation model features. The biases or distortion, if any, shall be quantified in order to improve the code models and correlations, and as well for correcting the compensation errors. A good example has been provided to highlight the use of IET experimental data to justify the validity of the uncertainties of the physical models of best estimate system code, and more generally of the consistency of a BEPU methodology based on the approach of propagating uncertainty in code input (Geiser et al., 2017). In order to demonstrate the scalability of the simulation model and the uncertainties taken into account in the calculation of the BEPU (evaluation model) methodology at the NPP, it is recommended to study the representative IET of the transients of interest on ITF (Integral Test Facility) at different scales. The BEPU methodology for Intermediate Break (IB) LOCA based on the propagation of the input uncertainty approach with the CATHARE best-estimate system code was applied to two IETs representing a selected IB LOCA transient from the two OECD/NEA ROSA tests. The results indicate that the uncertainties of the physical models considered in the BEPU methodology provide uncertainty bands of the peak cladding temperature (PCT) that bound the experimental data in both tests. In particular, the maximum (95/95 percentile) of PCT for each test is greater than the measured PCT. The consistency and conservatism of the methodology for IB LOCA are therefore demonstrated.

to safety analysis for real plant. This requires an assessment of the adequacy of the simulation model and uncertainty analysis for this application. A general framework for assessing the adequacy of a thermal hydraulic analysis and uncertainties associated with its results in the context of a safety decision is given in Fig. 7 (Nourbakhsh and Banerjee, 2013). In accordance with the CSAU or EMDAP process, the first step is to specify the nature of the safety analysis, and the details of the subsequent evaluation may vary depending on the specific SRQ (or figure of merit) required for the safety analysis. This is followed by the VVUQ process (identification and evaluation of the key phenomena, selection of the code, assessment of the applicability of the code, scaling analysis and code validation). The uncertainty analysis must ultimately be performed during the plant safety analysis. This should begin with identifying sources of uncertainty that may affect the SRQ. All relevant uncertainties, whether they can be appropriately addressed by code calculations or not, should be taken into account, including the uncertainties of completeness of experimental database. However, not all identified uncertainties can or need to be quantified and many uncertainties may not affect the results of the safety analysis. The important uncertainty contributors that must be considered can be confirmed by using the global sensitivity analysis method. The most widely used uncertainty analysis method is the forward input uncertainty propagation by statistical method. The procedure for statistical analysis of the uncertainty chosen to propagate the uncertainties must be designed to generate a given sample of code executions, and appropriate statistical methods must be used to develop probabilistic statements demonstrating compliance with the acceptance criteria. If a detailed statistical uncertainty analysis could not be performed for some models or input parameters, the assessment must show that a sufficient conservative bias has been retained in the methodology to justify its applicability. The alternative assessment of the impact of such uncertainties can greatly reduce the effort required for the formal propagation of uncertainties. An example of such methods is to use the worst case assumptions or a plausible bounding estimate to address uncertainty. This approach is particularly reasonable if the worst-case assumptions do not affect the outcome of the safety analysis. It should be noted that evaluating the impact of uncertainties one after the other using the alternative approaches described above does not allow for synergistic effects to be taken into account when the impact of an individual uncertainty depends on the values assumed by the remaining uncertain parameters. Therefore, these uncertainties should be assessed together using the GSA method. A review of the GSA methods can be found in (Iooss, 2018). The most commonly used method is the variance-based methods, such as Sobol indexes (Sobol, 1993), ANOVA (Adetula and Bokov, 2012), Shapley effect (Radaideh et al., 2019c), etc. The effects of uncertainties that are appropriately addressed in the simulation model are reflected in the results of the uncertainty analysis, including the probability distribution for the SRQ. For certain uncertainties that are not properly accounted for by the propagation of uncertainties and where their impact cannot be evaluated by alternative methods, compensatory measures may be proposed to offset the impact of such uncertainties. For example, these offsets may include elimination of such uncertainties through design and operational changes, as well as procedures. Finally, the decision on the adequacy of the BEPU methodology is made on the comparison of the calculated SRQ (including uncertainties) against the acceptance criteria.

4.2.5. Assessment of the adequacy of the simulation model and uncertainty analysis for real plant applications As shown in Section 2, the final objective is to apply the simulation model best estimate prediction and uncertainty analysis (BEPU) method

4.2.6. Application of graded approach of VVUQ requirements In some cases, the user selects a fully developed, verified and validated generic purpose system thermal hydraulic code (such as the USNRC’s RELAP5 (ISL, 2001) or TRACE code (USNRC, 2008) for the

•

per experiment or a unique quantification for all experiments considered together?) as well as in case of several quantifications (how to combine input uncertainties, keeping in mind that several options exist?). Element 5 is based on the propagation of the model input uncertainties obtained in Element 4, together with other uncertain input parameters, through the computer code. It can be included in an iterative process with Element 4. It exploits the remaining subset of the experimental database identified in Element 2 and not used in Element 4. The propagation first implies the selection of an uncertainty model for each uncertain input (interval, pdf, possibility, …) that can be different from the uncertainty modelling associated with Element 4. Moreover, the input sampling procedure should be specified as well as the quantities of interest derived for the output sample that will be used for validation (e.g. percentiles in the probabilistic framework). Finally, a key point of this step is the definition and computation of validation metrics. It requires to reach a consensus on the definition of “validated uncertainty bands” (i.e. which important properties an uncertainty band has to satisfy to be accepted) and to introduce relevant criteria that mathematically translate this definition.

It should be noted that Elements 1–3 are common to any BEPU methodology based on CSAU (USNRC technical program group, 1989) or EMDAP (USNRC, 2005), focusing on the application of a qualified (i.e., fully verified and validated, with model input uncertainties quantified and validated) code for accident analysis. The good practices from those industrial development and applications and new researches will also be taken in the framework of the new OECD/NEA project SAPIUM (Baccou et al., 2017).

10

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

intended application (e.g., development of a BEPU methodology for non-LOCA analysis). A graded approach may be applied to the VVUQ requirements (USNRC, 2005). This graded approach allows to simplify the uncertainty quantification and analysis based on the VVUQ performed by the code developers and the target applications. Such graded BEPU methodology has been recently proposed by several authors (Kawamura and Hara, 2000; Abdelghany and Martin, 2010; Avramova, 2010; Zhang et al., 2013; Kozmenkov and Rohde, 2013, 2014; da Cruz et al., 2014; Pecchia, 2015; Queral et al., 2015; Brown et al., 2016; Martin, 2016; Zhang et al., 2016a,b; 2017). It is expected that this graded approach will motivate the development and application of the BEPU methodology to a wider transient scenarios.

validation, and uncertainty quantification in scientific computing. Comput. Methods Appl. Mech. Eng. 200 (25–28), 2131–2144. ASME, 2009. Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer, ASME V&V20-2009. USNRC, 1988. Best-Estimate Calculations of Emergency Core Cooling System Performance, RG-1.157, USNRC. USNRC technical program group, 1989. Quantifying Reactor Safety Margins: Application of Code Scaling, Applicability and Uncertainty Evaluation Methodology to a LargeBreak Loss of Coolant Accident, NUREG/CR-5249, USNRC. USNRC, 2005. Transient, and accident analysis methods, RG-1.203, US Nuclear Regulatory Commission. USNRC, 1993. Software Quality Assurance Program and Guidelines, NUREG/BR-0167, ADAMS ML012750471, USNRC. USNRC, 2000. Software Quality Assurance Procedures for NRC Thermal-Hydraulic Codes, ADAMS ML010170081, NUREG-1737, USNRC. Bajorek, S.M., et al., 2015. Development, validation and assessment of the TRACE thermal hydraulics systems code. In: Proc. 16th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-16). ANS, Chicago, IL. ONR, 2016. Validation of Computer Codes and Calculation Methods, Nuclear Safety Technical Assessment Guide NS-TAST-GD-042 Revision 3, ONR. Glaeser, H., 2008. GRS Method for Uncertainty and Sensitivity Evaluation of Code Results and Applications, Science and Technology of Nuclear Installations, Volume 2008, Article ID. Young, M.Y., Bajorek, S.M., Nissley, M.E., Hochreiter, L.E., 1998. Application of code scaling applicability and uncertainty methodology to the large break loss of coolant accident. Nucl. Eng. Des. 186 (1–2), 39–52. Frepoli, C., 2008. An overview of westinghouse realistic large break LOCA evaluation model. Sci. Technol. Nucl. Installations. Oberkampf, W.L., Pilch, M., Trucano, T.G., 2007. In: Predictive Capability Maturity Model for Computational Modeling and Simulation, SAND2007-5948. Sandia National Lab. Frepoli, C., Ohkawa, K., 2011. The development of a Realistic LOCA Evaluation Model Applicable to the Full range of Break sizes: Westinghouse Full Spectrum™ LOCA (FSLOCA™) Methodology. In: The 14th International Topical Meeting on Nuclear Reactor Thermal-hydraulics (NURETH-14), Toronto, Ontario (Canada). CNS. Martin, R.P., O’Dell, L.D., 2005. AREVA’s realistic large break LOCA analysis methodology. Nucl. Eng. Des. 235 (16), 1713–1725. Sauvage, J.-Y., Keldenich, M., 2005. ESM-3D: a fully extended statistical method based on CATHARE 3D for loss of coolant accident transients analysis. In: Proceedings of the 13th International Conference on Nuclear Engineering (ICONE-13), Beijing, China. Ban, C.-H., Lee, S.-Y., Sung, C.-K., 2004. Development and Application of KEPRI Realistic Evaluation Methodology (KREM) for LB-LOCA. In: Proc. International meeting on updates in best estimate methods in nuclear installation safety analysis (BE-2004), Washington, DC, United States, 14-18. Seeberger, G.-J., Pauli, E.-M., Trewin, R., Zeisler, L.-P., 2014. Statistical large-break LOCA analysis for PWRs with combined ECC injection. Kerntechnik 79 (2), 103–110. Kozmenkov, Y., Rohde, U., 2013. Application of statistical uncertainty and sensitivity evaluations to a PWR LBLOCA analysis calculated with the code ATHLET. Part 1: Uncertainty analysis. Kerntechnik 78 (4), 354–361. Kozmenkov, Y., Rohde, U., 2014. Application of statistical uncertainty and sensitivity evaluations to a PWR LBLOCA analysis calculated with the code ATHLET. Part 2: Sensitivity analysis. Kerntechnik 79 (2), 97–102. Queral, C., Montero-Mayorga, J., Gonzalez-Cadelo, J., Jimenez, 2015. AP1000® largebreak LOCA BEPU analysis with TRACE code. Ann. Nucl. Energy 85, 576–589. Zhang, H., Szilard, R., Bayless, P., Zou, L., Zhao, H., 2016a. Early-Demonstration Of BEPU Analysis with the RELAP5-3D Code In Response To Proposed 10 CFR 50.46(c) Rulemaking. In: Proc. of ICAPP 2016. ASME, San Francisco, CA. Zhang, H., et al., 2016b. Comparisons of Wilks’ and Monte Carlo Methods in Response to the 10CFR50.46(c) Proposed Rulemaking. In: NUTHOS-11: The 11th International Topical Meeting on Nuclear Reactor Thermal Hydraulics, Operation and Safety, Gyeongju, Korea, October 9-13, 2016, KNS. Zhang, H., Szilard, R., Zou, L., Zhao, H., 2017. Sensitivity analysis of LBLOCA in response to 10 CFR50.46c new rulemaking. In: Proc. 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-17), Xi’an, China. ANS. Kawamura, S., Hara, T., 2000. Best Estimate Methods for Licensing Analysis. In: Proc. Int. Mtg. BE-2000, Washington, DC, November 12-16, American Nuclear Society (ANS). Abdelghany, J.M., Martin, R.P., 2010. Uncertainty analysis for containment response of U.S. EPR TM reactor to large break loss-of-coolant accidents. In: Proceedings of the 2010 International Congress on Advances in Nuclear Power Plants – ICAPP ’10. ASME. ASN, 2017. Qualification des outils de calcul scientifique utilisés dans la démonstration de sûreté nucléaire – 1re barrière, Guide de l ’ASN N° 28, ASN. Avramova, M., 2010. Verification, validation and uncertainty quantification in multiphysics modeling for nuclear reactor design and safety analysis. Prog. Nucl. Energy 52 (7), 601–614. Zhang, J., Segurado, J., Schneidesch, C., 2013. Towards an industrial application of statistical uncertainty analysis methods to multi-physical modelling and safety analyses. In: Proc. OECD/CSNI Workshop on Best Estimate Methods and Uncertainty Evaluations, Barcelona, Spain, 16-18 November, 2011, NEA/CSNI/R(2013)8, OECD. da Cruz, D.F., Rochman, D., Koning, A.J., 2014. Propagation of nuclear data uncertainty for a control rod ejection accident using the total Monte-Carlo method. In: Proc. PHYSOR 2014 – The Role of Reactor Physics toward a Sustainable Future, Kyoto, Japan, September 28 – October 3. Pecchia, M., 2015. Criticality safety evaluation of a Swiss wet storage pool using a global uncertainty analysis methodology. Ann. Nucl. Energy 83, 226–235. Martin, R.P., 2016. Best-estimate-plus-uncertainty-informed deterministic safety

5. Conclusions and perspectives This paper has reviewed the role of simulation model VVUQ process in the BEPU methodology development. It covers the applicable technical standards and procedures, regulatory requirements and recommendations, industrial applications and practices. From the technical point of view, a BEPU methodology must be based on fully verified and validated codes and models, with important model input uncertainties quantified. From the regulatory point of view, the adequacy of the simulation model (computer codes and plant models) for the intended BEPU application must be assessed through the VVUQ process. The BEPU applications will bring safety benefits to the design and operation of nuclear power plants. However, there are still obstacles and open questions in the industrial applications and practices of model VVUQ to BEPU methodology development. From the economic point of view, the costs associated with performing a high-quality VVUQ could be very high. Therefore, the long-term benefits of using the BEPU methodology must be weighed by the users against the costs associated with the simulation model development and VVUQ. It is essential to adapt a pragmatic and graded approach to the application of VVUQ requirements for codes and models that are fully verified and validated by the developers. This, in turn, motivates the continued development and cost-effective application of the BEPU methodology. References Wilson, G.E., 2013. Historical insights in the development of best estimate plus uncertainty safety analysis. Ann. Nucl. Energy 52, 2–9. IAEA, 2016. Safety Assessment for Facilities and Activities, IAEA Safety Standards Series No. GSR Part 4 (Rev. 1), International Atomic Energy Agency, Vienna. IAEA, 2019. Deterministic Safety Analysis for Nuclear Power Plants. IAEA Specific Safety Guide No. SSG-2. IAEA, Vienna. IAEA, 2008. Best Estimate Safety Analysis for Nuclear Power Plants: Uncertainty Evaluation, IAEA Safety Report Series No. 52, International Atomic Energy Agency, Vienna. Thacker, B.H., Doebling, S.W., Hemez, F.M., Anderson, M.C., Pepin, J.E., Rodriguez, E.A., Concepts of Model Verification and Validation, LA-14167-MS, LANL, October 2004. ANS, 1987. Guidelines for Verification and Validation of Scientific and Engineering Computer Programs for the Nuclear Industry, ANSI/ANS-10.4-1987. AIAA, 1998. Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, American Institute of Aeronautics and Astronautics, AIAA G-077-1998. Harvego, E.A., Schultz, R.R., Crane, R.L., 2011. Development of a consensus standard for verification and validation of nuclear thermal-fluids software. Nucl. Eng. Des. 241, 4691–4696. Oberkampf, W.L., Trucano, T.G., 2002. Verification and validation in computational fluid dynamics. Prog. Aerosp. Sci. 38, 209–272. Oberkampf, W.L., Trucano, T.G., Hirsch, C., 2004. Verification, validation, and predictive capability in computational engineering and physics. Appl. Mech. Rev. 57 (5), 345–384. Oberkampf, W.L., Trucano, T.G., 2007. Verification and Validation Benchmarks. SAND2007-0853. Sandia National Laboratories. Oberkampf, W.L., Barone, M.F., 2006. Measures of agreement between computation and experiment: validation metrics. J. Comp. Phys. 217, 5–36. Rider, W.J., Witkowski, W.R., Mousseau, V., 2015. In: UQ’s Role in Modeling and Simulation Planning, Credibility and Assessment Through the Predictive Capability Maturity Model, SAND2015-20747. Sandia National Laboratories. Roy, C.J., Oberkampf, W.L., 2011. A comprehensive framework for verification,

11

Nuclear Engineering and Design 355 (2019) 110312

J. Zhang

Vol. 2, “Facility and Experimental Characteristics, NEA/CSNI/R(93)14/Part.2, OECD/NEA. Aksan, N., D’Auria, F., Glaeser, H., 2018. Thermal-hydraulic phenomena for water cooled nuclear reactors. Nucl. Eng. Des. 330, 166–186. Aksan, N., et al. 1987. Code Validation Matrix of Thermal-Hydraulic Codes for LWR LOCA and Transients, CSNI Report No. 132, OECD/NEA. EPRI, 2014. Data Sources for Capability Assessments of Next Generation Safety Analysis Codes, Technical Report 3002003110, EPRI. Reventós, F., de Alfonso, E., Mendizábal Sanz, R., 2016. PREMIUM: A Benchmark on the Quantification of the Uncertainty of the Physical Models in System Thermal-hydraulic Codes – Methodologies and Data Review, NEA/CSNI/R(2016)9, OECD/NEA. Mendizábal Sanz, R., et al., 2017. Post-BEMUSE Reflood Model Input Uncertainty Methods (PREMIUM) Benchmark – Final Report, NEA/CSNI/R(2016)18, OECD/NEA. Baccou, J., Zhang, J., Nouy, E., 2017. Towards a systematic approach to input uncertainty quantification methodology. In: Proceedings of the 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-17), Xi’an, China. Wilson, G.E., Boyack, B.E., 1998. The role of the PIRT process in experiments, code development and code applications associated with reactor safety analysis. Nucl. Eng. Des. 186 (1–2), 23–37. Luo, H., Wu, Q., Mousseau, V., 2010. Quantified PIRT for reactor system safety analysis code validation using characteristic time ratios. In: Proc. NUTHOS-8, Shanghai, China. Martin, R.P., 2011. Quantifying phenomenological importance in best-estimate plus uncertainty analyses. Nucl. Technol. 175. Yurko, J.P., Jacopo, B., 2012. Quantitative Phenomena Identification and Ranking Table (QPIRT) for Bayesian uncertainty quantification. In: 2012 International Congress on Advances in National Power Plants (ICAPP ’12). American Nuclear Society, Chicago, IL. Baccou, J., Chojnacki, E., Georgenthum, V., Marais, S., 2018. On the methodological treatment of input uncertainty quantification: illustration in the RIA framework. In: BEPU-2018 Conference, Lucca, Italy. Saaty, T., 1982. Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World, vol. Analytic hierarchy process series, RWS Publications. Geiser, H., Vacher, J.-L., Rubiolo, P.R., 2017. The use of integral effects tests for the justification of new evaluation models based on the BEPU approach. In: Proceedings of the 17th International Topical Meeting on Nuclear Reactor Thermal Hydraulics (NURETH-17), Xi’an, China. Iooss, B., 2018. Sensitivity analysis of model outputs: principle, methods and issues for BEPU methodology, BEPU-2018 conference, Lucca, Italy. Sobol, I.M., 1993. Sensitivity estimates for nonlinear mathematical models. Math. Modell. Computat. Exp. 1, 407–414. Adetula, B.A., Bokov, P.M., 2012. Computational method for global sensitivity analysis of reactor neutronic parameters. Sci. Technol. Nucl. Installations. ISL, 2001. RELAP5/MOD3.3 code manual, volume III: Developmental assessment problems, NUREG/CR-5535/Rev 1-Vol III, USNRC. USNRC, 2008. TRACE V5.0 Developmental Assessment Manual, ADAMS ML120060208, ML120060187, ML120060191, ML120060172, USNRC.

evaluation. Trans. Am. Nucl. Soc. 115, 1136–1139 Las Vegas, NV, November 6–10. Brown, C.S., Zhang, H., Kucukboyaci, V., Sung, Y., 2016. Best estimate plus uncertainty analysis of departure from nucleate boiling limiting case with CASL core simulator VERA-CS in response to PWR main steam line break event. Nucl. Eng. Des. 309, 8-–22. Nguyen, T.N., 2017. Surrogate-based multi-experiment calibration of the BISON fission gas behavior model. Nucl. Eng. Des. 320, 409–417. Burns, J.R., Brown, N.R., 2017. Neutron cross section sensitivity and uncertainty analysis of candidate accident tolerant fuel concepts. Ann. Nucl. Energy 110, 1249–1255. Walters, W., Bobolea, R., Maupin, K., Deveney, R., Parker, J., Segard, K., Barner, R., 2018. ARTEMIS/RELAP5 Integrated transient analysis application to non-LOCA transients, In: Proc. TOPFUEL 2018, Prague, Czech Republic, 30 September – 04 October. Radaideh, M.I., et al., 2019a. Advanced BWR criticality safety part I: model development, model benchmarking, and depletion with uncertainty analysis. Prog. Nucl. Energy 113, 230–246. Radaideh, M.I., Wieselquist, W.A., Kozlowski, T., 2019b. A new framework for samplingbased uncertainty quantification of the six-group reactor kinetic parameters. Ann. Nucl. Energy 127, 1–11. Radaideh, M.I., Surani, S., O'Grady, D., Kozlowski, T., 2019c. Shapley effect application for variance-based sensitivity analysis of the few-group cross-sections. Ann. Nucl. Energy 129, 264–279. Prošek, A., Mavko, B., 2007. The state-of-the-art theory and applications of best-estimate plus uncertainty methods. Nucl. Technol. 158. Bucalossi, A., Petruzzi, A., 2010. Role of best estimate plus uncertainty methods in major nuclear power plant modifications. J. Nucl. Sci. Technol. 47 (8), 671–683. Nourbakhsh, H., Banerjee, S., 2013. Examining the adequacy of thermal-hydraulic analyses for safety decisions. In: Proc. the 15th International Topical Meeting on Nuclear Reactor Thermal-Hydraulics (NURETH-15), Pisa, Italy, May 12-17. Martin, R.P., 2016. Content, completeness, and consistency of analytical models for regulatory consideration. Nucl. Technol. 193, 96–112. Rider, W.J., Kamm, J.R., Weirs, V.G., 2010. Verification, Validation and Uncertainty Quantification Workflow in CASL, SAND2010-234P, Sandia National Laboratories. Pourgol-Mohamad, M., Mosleh, A., Modarres, M., 2011. Structured treatment of model uncertainty in complex thermal-hydraulics codes: technical challenges, prospective and characterization. Nucl. Eng. Des. 241, 285–295. Unal, C., Williams, B., Hemez, F., Atamturktur, S.H., McClure, P., 2011. Improved best estimate plus uncertainty methodology, including advanced validation concepts, to license evolving nuclear reactors. Nucl. Eng. Des. 241, 1813–1833. Stoots, C., Larson, T., Schultz, R., Gougar, H., McCarthy, K., Petti, D., Swiler, L., Corradini, M., 2012. Verification and Validation Strategy for LWRS Tools, INL/EXT-12-27066 Revision 0, Idaho National Laboratory. Petruzzi, A., D’Auria, F., 2016. Standardized consolidated calculated and reference experimental database (SCCRED): a supporting tool for V&V and uncertainty evaluation of best-estimate system codes for licensing applications. Nucl. Sci. Eng. 182, 13–53. OECD/NEA, 1993. Separate Effects Test Matrix for Thermal-Hydraulic Code Validation, Vol. 1, Phenomena Characterisation and Selection of Facilities and Tests, NEA/CSNI/ R(93)14/Part.1, OECD/NEA. OECD/NEA, 1993. Separate Effects Test Matrix for Thermal-Hydraulic Code Validation,”

12