Parallel Computational Fluid Dynamics 2003 Advanced Numerical Methods Software and Applications [1 ed.] 0444516123

Citation preview

PARALLEL COMPUTATIONAL FLUID DYNAMICS ADVANCED NUMERICAL METHODS SOFTWARE AND APPLICATIONS

Proceedings of the Parallel CFD 2003 Conference Moscow, Russia (May 13-15, 2003)

Edited by B. CHETVERUSHKIN Institute of Mathematical Modelling Russian Academy of Sciences, Moscow, Russia

A.

ECER

J. PERIAUX

IUPUI, Indianapolis Indiana, U.S.A.

Dassault-Aviation Saint-Cloud, France

Assistant Editor N.

SATOFUKA

P.

Kyoto University of Technology Kyoto, Japan

FOX

IUPUI, Indianapolis Indiana, U.S.A.

2004

ELSEVIER Amsterdam - Boston - Heidelberg - London - New York - Oxford - Paris San Diego - San Francisco - Singapore - Sydney - Tokyo

PARALLEL COMPUTATIONAL FLUID DYNAMICS ADVANCED NUMERICAL METHODS SOFTWARE AND APPLICATIONS

This page is intentionally left blank

ELSEVIER B.V. Sara Burgerhartstraat 25

ELSEVIER Inc. 525 B Street, Suite 1900

ELSEVIER Ltd ELSEVIER Ltd The Boulevard, Langford Lane 84 Theobalds Road

P.O. Box 211,1000 A E Amsterdam San Diego, CA 92101- 4495 Kidlington, Oxford OX5 1GB

London WCIX 8RR

The Netherlands

UK

USA

UK

© 2004 Elsevier B.V. All rights reserved. This work is protected under copyright by Elsevier B.V., and the following terms and conditions apply to its use: Photocopying Single photocopies of single chapters may be made for personal use as allowed by national copyright laws. Permission of the Publisher and payment of a fee is required for all other photocopying, including multiple or systematic copying, copying for advertising or promotional purposes, resale, and all forms of document delivery. Special rates are available for educational institutions that wish to make photocopies for non-profit educational classroom use. Permissions may be sought directly from Elsevier's Rights Department in Oxford, UK: phone (+44) 1865 843830, fax (+44) 1865 853333, e-mail: permissions @elsevier.com. Requests may also be completed on-live via the Elsevier homepage (http://www.elsevier.com/locate permissions). In the USA, users may clear permissions and make payments through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; phone: (+1) (978) 7508400, fax: (+1) (978) 7504744, and in the UK through the Copyright Licensing Agency Rapid Clearance Service (CLARCS), 90 Tottenham Court Road, London W1P 0LP, UK; phone: (+44) 20 7631 5555; fax: (+44) 20 7631 5500. Other countries may have a local reprographic rights agency for payments. Derivative Works Tables of contents may be reproduced for internal circulation, but permission of the Publisher is required for external resale or distribution of such material. Permission of the Publisher is required for all other derivative works, including compilations and translations. Electronic Storage or Usage Permission of the Publisher is required to store or use electronically any material contained in this work, including any chapter or part of a chapter. Except as outlined above, no part of this work may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the Publisher. Address permissions requests to: Elsevier's Rights Department, at the fax and e-mail addresses noted above. Notice No responsibility is assumed by the Publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made.

First edition 2004 Library of Congress Cataloging in Publication Data A catalog record is available from the Library of Congress. British Library Cataloguing in Publication Data A catalogue record is available from the British Library.

ISBN: ISSN:

0-444-51612-3 1570-9426

@ The paper used in this publication meets the requirements of ANSI/NISO Z39. 48-1992 (Permanence of Paper). Printed in The Netherlands.

PREFACE

Parallel CFD 2003, the fifteenth in the high-level international series of meetings featuring different aspects of parallel computing in computational fluid dynamics and other modern scientific domains has been held since May 13th , till May 15th , 2003 in Moscow, Russia. The themes of the 2003 meeting included the traditional emphases of this conference parallel algorithms, CFD applications, and experiences with contemporary architectures. More than 100 presentations were included into the conference program in the following sessions: Numerical Methods and Parallel Algorithms, Turbulence and Acoustics Problems, Combustion Problems, Metacomputing and GRID Technologies, Environment and Ecology Problems, Plasma and Kinetics Problems, Parallel Software, Heat and Mass Transfer Problems, Aerodynamic and Hydrodynamic Flows The proceedings of ParCFD 2003 represent more then 60% of conference articles. All papers' included into this volume have been refereed. We hope that this book will become an interesting reading to scientists and engineers working in different modern scientific areas connected with parallel computing.

Boris Chetverushkin, Chairman, Parallel CFD 2003

V

This page is intentionally left blank

ACKNOWLEDGEMENTS

Parallel CFD 2003 was organized by the Institute for Mathematical Modelling of Russian Academy of Sciences in collaboration with Joint Supercomputer Centre of Russia. The organisers are grateful for strong financial support received from: Russian Academy of Sciences Russian Foundation for Basic Researches (RFBR) Ministry of Industry, Science and Technology of Russian Federation Ministry of Atomic Energy of Russian Federation European Community on Computational Methods in Applied Sciences (ECCOMAS) Russian Society for Scientific and Engineering Computations (ONIV) HEWLETT-PACKARD, Russia IBM, Russia INTEL, Russia Assistance of our sponsors allowed to organize scientific as well as social program of the conference and to provide special support for postgraduate students, young researches and scientists working under Russian State Programs and RFBR grants. Many people worked a lot to organize and execute the conference. We are especially grateful to Pat Fox for her kind assistance and to all members of international scientific committee. We also want to thank the key members of the local organizing committee Natalia Romanyukha, Serge Polyakov, Alexandr Maslov and to share our success with our colleagues from the Institute for Mathematical Modelling assisted in organization of Parallel CFD2003.

Boris Chetverushkin, Chairman, Parallel CFD 2003

vii

This page is intentionally left blank

TABLE OF CONTENTS Preface

v

Acknowledgements

vii

1. Invited Papers OlegM. Belotserkovskii Mathematical Modeling Using Supercomputers with Parallel Architecture Andrew T. Hsu, Tianliang Yang, Isaac Lopez, and Akin Ecer A Review of Lattice Boltzmann Model for Compressible Flows

1

Osamu Inoue DNS of the Generation and Propagation of Sound

29

SugaA. Sugavanam CFD on the BlueGene/L Supercomputer

37

Edwin A.H. Vollebregt and Marc R. T. Roest Software Frameworks for Integrated Modeling

19

41

2. Numerical Methods And Parallel Algorithms Vladimir V. Aristov, Anna A. Frolova, and Serguei A. Zabelok Parallel Algorithms of Direct Solving the Boltzmann Equation in Aerodynamics Problems

49

Dan-Gabriel Calugaru and Damien Tromeur-Dervout Aitken Acceleration of Dirichlet - Neumann Algorithm for Flow in Porous Media with Discontinuous Permeability

57

Tatiana G. Elizarova and Olga Yu. Milyukova Parallel Algorithm for Numerical Simulation of 3D Incompressible Flows

65

Mikhail V. lakobovski, Sergey N. Boldyrev, Sergey A. Sukov Big Unstructured Mesh Processing on Multiprocessor Computer Systems Igor Ye. Kaporin and Igor N. Konshin Parallel Conjugate Gradient Preconditioning via Incomplete Cholesky of Overlapping Submatrices Vladimir P. Memnonov Coupling Scheme for Continuum and Parallel DSMC Parts to a Numerical Solution of a Two-Dimensional Subsonic Problem

ix

73 81

89

X

Olga Yu. Milyukova Parallel Iterative Methods with Factorized Preconditioning Matrices for Solving Discrete Elliptic Equations

97

Martin Maihofer, Albert Ruprecht A Local Grid Refinement Algorithm on Modern High-Performance Computers

105

Eugene V. Shilnikov Viscous Gas Flow Simulation on Nested Grids Using Multiprocessor Computer Systems

113

Andrei V. Smirnov Domain coupling with the DOVE scheme

119

Samuil M. Bakhrakh, Sergey V. Velichko, Valentin F. Spiridonov Crash-free Technology for LEGAK Calculation of Solid Flows on Distributed Memory Multiprocessors

129

A.I. Tolstykh, M.V. Lipavskii, E.N. Chigirev On Arbitrary-order Schemes for Parallel CFD Calculations

137

Alexander G. Churbanov and Petr N. Vabishchevich Domain Decomposition Techniques to Solve the Navier-Stokes Equations

145

K. Lipnikov and1'. Vassilevski Parallel Adaptive Solution of Stokes and Oseen Problems on Unstructured 3D Meshes

153

Kirill Yu. Bogachev and Georgy M. Kobelkov Numerical Solution of a Tidal Wave Problem

163

3. Turbulence And Acoustics Problems S.M. Bakhrakh, N. A. Volodina, P. N. Nizovtsev, V.F. Spiridonov, E.V.Shuvalova Numerical Simulation of Initial Perturbation Growth with Oblique Impact of Metal Plates

173

Vladimir G. Bobkov, Tatyana K. Kozubskaya High Accuracy Simulation of Acoustic Noise from Sources in Pipes

181

Valentin A. Gushchin, Alexei V. Kostomarov, Paul V. Matyushin, Tatyana I. Rozhdestvenskaya Parallel Computing of 3D Separated Homogeneous and Stratified Fluid Flows around the bluff bodies

189

xi

Vladimir F. Tishkin, Nikolai V. Zmitrenko, Marina Ye. Ladonkina, Nadezhda G. Proncheva, SvetlanaM. Garina A Study of a Gravity-Gradient Mixing Properties by the Means of a Direct Numerical Modeling

197

Li Jiang and Chaoqun Liu Parallel DNS for Flow Separation and Transition around Airfoil

205

YingXu, Tianliang Yang, James M. McDonough and Kaveh A. Tagavi Parellelization of Phase-field Model for Phase Transformation Problem

213

Evgueni M. Smirnov, Alexey G. Abramov, Nikolay G. Ivanov, Pavel E. Smirnov, Sergey A. Yakubov DNS and RANS/LES-Computations of Complex Geometry Flows Using a Parallel Multiblock Finite-Volume Code Igor I. Wertgeim and Victor G. Zakharov Numerical Methods and Parallel Algorithms for Study of Burgers' Turbulence V.A. Zhmaylo, O.G. Sin'kova, VN. Sofronov, V.P. Statsenko, Yu.V. Yanilkin, A.R. Guzhova, A.S. Pavlunin Numerical Study of Gravitational Turbulent Mixing in Alternating-Sign Acceleration

219

227

235

4. Combustions Problems Fariborz Taghipour Development of a CFD-Based Model for Photo-Reactor Simulation

243

M. Zolver, J. Bohbot, D. Klahr, A. Torres An Unstructured Parallel Solver for Multi-Phase and Reactive Flows in Internal Combustion Engines

251

5. Metacomputing And Grid Technologies Stanley Chien, Yudong Wang, Akin Ecer, and Hasan U. Akay Grid Scheduler with Dynamic Load Balancing for Parallel CFD

259

Marc Garbey Aitken-Schwarz Method for CFD Solvers

267

Soon-Heum Ko, Chongam Kim, Oh-Hyun Rho, and Sangsan Lee A Grid-based Flow Analysis and Investigation of Load Balance in Heterogeneous Computing Environment

277

Damien Tromeur-Dervout Krylov Method Algorithms Adapted to the Grid Computing

285

xii

Erdal Yilmaz, Akin Ecer, Hasan U. Akay, Resat U. Payli, Stanley Chien, and Yudong Wang Parallel Computing in Grid Environment

293

6. Environment and Ecology Problems Valery N. Gloukhov Parallel Computations in Problems of Climate Modeling Sergey A. Strelkov, Tamara A. Sushkevich, Ekaterina V. Vladimirova, Ekaterina I. Ignatijeva, Alexey K. Kulikov, Svetlana V. Maksakova, Vladimir V. Kozoderov, Boris A. Fomin, Evgeney A. Zhitnitskii, IrinaN. Melnikova, Aleksandr N. Volkovich New Automatic Code Radiation Earth with Parallel Computing Gennady V. Kupovych, Alexander I. Sukhinov, Alexander G. Marchenko, Alexander G. Klovo Hydrothermodynamic Model for Sea Surface Layer and its Realization on the Distributed Computing Cluster

301

309

315

Tamara A. Sushkevich Mathematical Modeling of the Multidimensional Radiation Transfer Problems with Parallel Computing

319

Yukio Tanaka and Motohiko Tsugawa Development of a High Resolution Parallel Ocean Circulation Model on the Earth Simulator

327

Hirofumi Tomita, Koji Goto, andMasaki Satoh A Comparison Study of Computational Performance between a Spectral Transform Model and a Gridpoint Model

333

Gabriel Winter Althaus, Bias Galvdn Gonsalez, Begona Gonzalez Landin, David Greiner Sanchez, Onesimo Mansogo Ntanga, Silvia Alonso Lorenzo Solving Economic and Environmental Optimal Control in Dumping of Sewage with a Flexible and Parallel Evolutionary Computation

341

7. Plasma And Kinetics Problems E.N. Donskoy, A.N. Zalyalov Bremsstrahlung Account in Photon Transport

349

ValeriyA. Galkin Background of Computations for Mathematical Models, Based on Conservation Laws Systems and Applications to Fluid Dynamics

357

D. V. Khotyanovsky, D.D. Knight, A.N. Kudryavtsev, M.S. Ivanov Parallel Simulation of Laser-Induced Shock Wave Reflection Transition

365

xiii

D. P. Kostomarov, IN. Inovenkov, E.Y. Echkina, A.V. Leonenko, O.A. Pavlova and S. V. Bulanov Application of Parallel Programming Method for 3D MHD Computer Simulations of Magnetic Reconnection in Plasma Valeri I. Saveliev Study of Wave Equations for a Moving Charge in the Media with Discontinued Dielectric Properties

373

381

8. Parallel Software Hasan U. Akay, Erdal Oktay, Xiaoyin He andResat U. Payli A Code Coupling Application for Solid-Fluid Interactions and Parallel Computing

393

Audrey A. Aksenov, Sergey A. Kharchenko, Vladimir N. Konshin, Victor I. Pokhilko FlowVision Software: Numerical Simulation of Industrial CFD Applications on Parallel Computer Systems

401

Dominique Eyheramendy Object-Oriented Parallel CFD with JAVA

409

Victor Ivannikov, Serguei Gaissaryan, Arutyun Avetisyan, Vartan Padaryan Development of Scalable Parallel Programs in ParJava Environment

All

S.P. Kopyssov, I. V.Krasnoperov, A.K.Novikov, V.N.Rychkov Object-oriented Software for Domain Decomposition Methods with Local Adaptive Refinement

425

Peter S. Krinov, Mikhail V. Iakobovski, Sergey V. Muravyov Large Data Volume Visualization on Distributed Multiprocessor Systems

433

Alexey V. Moriakov, Vladimir P. Vasyukhno, Mikhail E. Netecha, Grigori A. Khacheresov Programs LUCKY & LUCKY_C - 3D Parallel Transport Codes for The MultiGroup Transport Equation Solution for XYZ Geometry By PmS n Method

439

S.Timchenko, V.Bimatov, P.Kuznecov, Yu.Sidorenko Multiobjective Asynchrone Parallel Genetic Algorithm for Reentry Trajectory Optimization

447

A.N. Bykov, A.S. Zhdanov Combined Model OpenMP & MPI for Parallel Computations in Gas Dynamics Program

455

xiv

9. Heat And Mass Transfer Problems §. Celasun, Y. Ozturk An Example to the Numerical Processing of Nonisothermal Flow of non-Newtonian Fluids

463

Anil Deane, Paul Fischer A Spectral-Element Model of Mantle Convection

469

J. M. McDonough, T. Yang, M. Sheetz Parallelization of a Modern CFD Incompressible Turbulent Flow Code

413

M.Soria, F.X.Trias, C.D.Perez-Segarra, A.Oliva Direct Numerical Simulation of Turbulent Natural Convection Flows Using PC Clusters

481

10. Aerodynamic And Hydrodynamic Flows Boris N.Chetverushkin, Natalia G. Churbanova, Marina A. Trapeznikova and Natalia Yu. Romanyukha Solution of Computational Fluid Dynamics Problems on Parallel Computers with Distributed Memory

489

X. J. Gu, R. W. Barber and D. R. Emerson Simulating Microfluidic Devices on Parallel Computers

497

S. Peigin, B.Epstein, S.Gali Multilevel Parallelization Strategy for Optimization of Aerodynamic Shapes

505

Nobuyuki Satofuka, Koji Morinishi and Keigo Kamitsuji Parallel Computation of Micro Fluid Dynamic Problems

513

Y. D. Shevelev, F.A.Maximov, V.A.Mihalin, N.G.Syzranova Numerical Modeling of External 3-D CFD Problems on the Parallel Computers and Aerodynamic Shape Optimization

521

Dmitry M. Davidenko, Iskender Gokalp, Emmanuel Dufour, Daniel Gaffle Numerical Simulations of Supersonic Combustion of Methane-Hydrogen Fuel in an Experimental Combustion Chamber

529

V. E. Fortov and I. V. Lomonosov Equations of State of Matter at Extreme Conditions

537

Richard B. Pelz (1957-2002) This volume of the Proceedings of Parallel Computational Fluid Dynamics is dedicated to the life and memory of Richard Bruce Pelz. Richard was an important influence on the development of parallel CFD and on this conference, which he loved, led once (at Rutgers University, in 1992), and served for many years through his participation on its standing scientific committee. Rich earned his B.S. in Aerospace and Ocean Engineering at Virginia Tech in 1979, spending the first of what would be three summers at Grumman Aerospace Corporation following his junior year. He commenced doctoral studies at the Courant Richard Pelz in front of a volcano in Institute at NYU under Professor Antony Hokkaido, Japan. He spent the last summer Jameson, and followed his advisor to of his life in Kyoto visiting at the Research Princeton in 1980. There he earned his Institute for Mathematical Sciences (RIMS). M.A. and Ph.D. degrees, finishing up in 1983. He went to MIT as a post-doc with Professor Steven Orszag, and was again fated to follow his advisor back to Princeton in 1984. Finishing up post-doctoral appointments in Princeton's Applied and Computational Mathematics program, he began as an Assistant Professor at Rutgers, in the Mechanical and Aerospace Engineering Department in 1986. He worked his way up the ranks to Professor, teaching a wide variety of engineering, applied mathematics, and computational methods courses, while maintaining an aggressive, innovative, and interdisciplinary research program. He pioneered CFD applications on several parallel machines during the late 1980s and early 1990s, which was a wild period of development for computer architecture and parallel programming models. Along the way, he won Honorable Mention in the Gordon Bell Prize competition in 1988 for a speed-up of 800 on a 1024-processor N-CUBE on a problem that he addressed with spectral elements. In addition to the Parallel CFD series, Rich co-organized international programs at the Newton Institute, Cambridge, UK, and was active in the Society for Industrial and Applied Mathematics and the American Physical Society. He was three times a Visiting Professor at the Institute for Theoretical Physics, UC Santa Barbara, and also visited the Geophysical Fluid Dynamics Laboratory in Princeton. He loved the power of simulation, but was a balanced scientist: one of his passions was the marriage of the computational with the analytical. Rich was known by his colleagues as a meticulous investigator and a generous friend. He was always willing to share his knowledge and help out younger scientists, who approached him

XV

xvi

easily. He was cheerful and balanced as a reviewer. He was excited on his last day, as on most days, about his chances at cracking an unsolved problem in finite-time vortex singularities. Rich pursued an example of finite-time singularities in Euler flows, in which twelve vortex tubes (which he called the "dodecapole") collapsed towards a vorticity null. He pursued this problem for several years without funding and received new funding from the National Science Foundation just before he died. Colleagues at Rutgers and the University of New Hampshire are today carrying out the research agenda he formulated. Rich left a beloved wife, Margaret Schleissner, and a daughter Emily. Friends across the globe learned with shock of his unexpected death during surgery. He left us in his prime. Memories of his wit and wordplay, of his deep knowledge of music and culture, of his dedication to students and colleagues, and of the tenacity of his scientific approach are preserved in our minds at a high point. A special issue of Fluid Dynamics Research, of which he was an editor, is dedicated to him. A special meeting was convened by Professor Koji Ohkitani in his honor in January 2003 in Japan, where he was a respected scientific visitor. The international community of Parallel Computational Fluid Dynamics will miss him dearly and joins other colleagues in acknowledging its own debt to Richard with this dedication.

Parallel Computational Fluid Dynamics - Advanced Numerical Methods, Software and Applications B. Chetverushkin, A. Ecer, J. Periaux, N. Satofuka and P. Fox (Editors) © 2004 Elsevier B.V. All rights reserved.

1

Mathematical modeling using supercomputers with parallel architecture O.M.Belotserkovskii Academician RAS Institute for Computer Aided Design RAS 19/18, 2 nd Brestskaya St., Moscow 123056, Russia In the paper the review of many works executed by our scientific school during last years is presented. The general principles of construction of numerical algorithms for highperformance computers are described. Several techniques are highlighted, which are based on the method of splitting with respect to physical processes and widely used in computing nonlinear multidimensional processes in fluid dynamics, in studies of turbulence and hydrodynamic instabilities, and in medicine and other natural sciences. We present new Russian developments here. 1. INTRODUCTION Numerical simulation plays a particularly important role when the physics of the phenomenon under study is not quite clear and the intrinsic mechanisms of interactions are not fully understood ("ill-posted problems"). A numerical experiment (where the statement of a problem, the solution method, and the implementation of an algorithm are treated as an integrated complex) essentially serves to refine the starting physical model. By computing various modifications of the model, facts and results are accumulated. Eventually, this leads to a possibility for selecting the most feasible variants. At present mathematical modeling becomes more and more actual, that is caused by new opportunities given by significant achievements in development of high performance computers with parallel architecture. As a rule, it is necessary to consider the complex nonlinear multidimensional unsteady equations with complicated internal structure and rheology. Literally for last one — two years there was a powerful jump in development of the supercomputer engineering as itself (Japan, USA). So the performance of new Japanese supercomputer NEC Earth Simulator (NEC ES) comes to 40 trillion mathematical operations per second. American specialists develop the supercomputers with yet more performance (hundred trillions per second). To this and other questions also the 6th International Conference "High Performance Computing in Asia Pacific Region" (HPC-6), Bangalore, India, December 2002, was devoted. It is extremely important, in our opinion, for users of such computing machinery ("End Users") to comprehend the priorities of problems and methods (algorithms, solvers), allowing rationally using so high performance tools for research. Now by us at Russian Academy of Science the project on development of effective parallel algorithms ("supersolver") for solution of high complexity problems, is started.

2

The purpose of the project is to develop effective ("general") parallel algorithms to solve wide circle of the problems of high complexity, described by equation systems in partial derivatives. Basis of the development is application of some modern algorithms applicable for calculations by parallel computers, such as method of splitting, "large particles" method, method of flows, statistical approaches, Riemann's schemes, method of finite elements etc. The idea consists in creation of a certain technology of parallel computing as a set of original codes of the software documentary prototypes, parallel - dependent part of which will not be changed, and the applied part is created by the user modification of the closest program prototype. Realization of the project will allow to reduce sharply the terms of development of new parallel applications and to use effectively available multiprocessing computing resources. In the paper some stages of the development of such supersolver are briefly formulated. As the application the solution of high complexity problems in such most actual directions of science and engineering as aerodynamics and fluid dynamics (on the basis of "large particles" method the package of applied problems "Gas Dynamics Tool" is developed) are considered, problems of seismology (construction of three-dimensional seismograms is especially important) and dynamic fractures are investigated. Application of computer technologies in medicine (new and very perspective field of applications) seems to be very important. Practically, these directions (and the problems, connected with the developments in nanoelectronics also) were noted by the majority of contributors at the conference HPC-6. We present new Russian developments here. 2. TURBULENCE AND HYDRODYNAMIC INSTABILITIES The approaches, which are mainly discussed, further on, are those, which use for a description of the free-developed shear turbulent flow for extended temporal intervals the complete (and closed) set of the dynamical equations for true values of velocities and pressure, as well as the statistical methods. The combined application of both these approaches (based on the use of hydrodynamic equations and on the statistical Monte-Carlo methodic) permits to understand in more detail the structures of turbulence, and to determine the rational ways of the construction of corresponding mathematical models. The cycle of works in this field was started from Karman Lecture given by O.M.Belotserkovskii in 1976, March 15-19*. The problem of construction of turbulence theory has a long history beginning from classical works of Reynolds and Richardson. In the middle of twenties years of last age Keller and Fridman have proposed the idea of stochastic description of turbulent processes that had invaluable principal significance. A number of important achievements were obtained (for instance, Kolmogorov-Obukhov spectrum etc., see [1-9]). Nevertheless, physical principles of turbulence development remain unclear (for instance, what is the source of energy of Belotserkovskii O.M. Computational experiment: direct numerical simulation of complex gas-dynamics flows on the basis of Euler, Navier-Stokes, and Boltzmann models.Karman Lecture. -Von Karman Institute for Fluid Dynamics.- 15-19 March 1976, Brussel.- in Numerical Methods in Fluid Dynamics".- Ed. by H.J.Wirz, J.J.Smolderen.- Hemisphere. Washington-London.-1978.-pp.339-387.

3

chaotic motion). Currently, experimental data point clearly on existence of the large-scale "coherent" structures (especially, for fully developed turbulence), where the main part of energy transferred is disposed [10]. Let's note that, for high Reynolds number, the energetic part of spectrum is far from dissipative one in terms of wave numbers (see fig. 1). The last decades are marked by a new approach (O.M.Belotserkovskii, 1985) in study of turbulence, namely — by direct numerical simulation (DNS) of processes of hydrodynamic flow on the basis of solution of hydrodynamic and kinetic equations [11]. Preliminary results of such approach demonstrate an important difference from the traditional statistical approaches. For instance, O.M.Belotserkovskii in his works [11-14] showed significant role of large-scale structures in the turbulence development. Direct numerical simulation allows to understand the influence of non-linear interactions on the development of turbulence and on the structure of flow. In the future, because of abovementioned, it would be expedient to carry out the numerical investigations of unsteady hydrodynamic phenomena on the basis of DNS by means of creation of theories of instability development and transition to turbulent stage. The fundamental principles for constructing of mathematical models for fully developed free turbulence and hydrodynamic instabilities are considered in the present paper. Such a "rational" modeling is applied for a variety of unsteady multidimensional problems [11-14]. The main ideology of "rational" approach for direct numerical simulation of the characteristics of fully developed shear turbulence by the high Re — investigation of large ordered structures (LOS) and small-scale stochastic turbulence (ST) is based on two hypotheses: "independence of LOS and ST" and "weak influence of molecular viscosity (or more generally, the mechanism of dissipation) by the study of LOS" [14-17]. Eventually, Kolmogorov has received the form of a spectrum in an inertial interval nothing assuming in general about a kind of dissipative members [1]. Really, for the wide class of phenomena, by the high Reynolds numbers within the lowfrequency and inertial intervals of turbulent motion, the effect of molecular viscosity and of the small elements of flow in the largest part of perturbation domain are not practically essential neither for the general characteristics of macroscopic structures of the flow developed, nor for the flow pattern as a whole. This makes it possible not to take into consideration the effects of molecular viscosity when studying the dynamics of large vortices, and to implement the study of those on the basis of models of the ideal compressible gas (discrete Euler equations) using the methods of "rational" averaging ("upwind" oriented schemes), but without special subgrid approximation and application of semi-empirical models of turbulence. Mentioned type of the oriented numerical schemes introduces automatically some scheme viscosity, which plays a role of implicit subgrid model [11-14]. Fig.l demonstrates an experimental data [14] for spectra of energy density Ei of longitudinal velocity component fluctuations for various turbulent flows (k^ - Kolmogorov wave number, E - local energy dissipation rate). For low-frequency and inertial parts of a spectrum (where LOS, which accumulate up to 80 % of flow energy, are formed) turbulent flows strongly differ from each other depending on a kind of flow and external conditions. And it demands direct calculation of LOS. But in dissipative interval practically universal behaviour of a spectrum is observed. It once again confirms very weak dependence of molecular viscosity on properties of LOS. Using dissipative-stable and divergent-conservative oriented schemes, for example, second or third order of accuracy) we choose the "optimal" scale (size) of resolution h for direct numerical simulation of the discrete Euler equation. The insignificant contribution of

4

energy from dissipative (high-frequency) part of a spectrum is suppressed by scheme viscosity or it is taken into account approximately. Curves in fig.l show the influence of scheme viscosity on the numerical solution for different orders of accuracy of the scheme. We can see that the schemes with second-third order of accuracy allow getting solutions for LOS without substantial influence of scheme dissipation. By using the abovementioned approach (method of "large particles" [12, 18]) the package of applied programs GasDynamicsTool was developed for numerical solving of wide range of gas dynamic problems (A.Zibarov [19]). At the same time, the properties of flows within the boundary layers and within the thin layers of mixing, at the viscous interval of turbulence, as well as those of flows by the moderate Reynolds numbers and in the domain of laminar-turbulent transition, are primarily determined by the molecular diffusion, and for these flows it is necessary to consider incompressible Navier-Stokes models (V.Gushshin, S.O.Belotserkovskii and others [12, 14, 20—24]). The pulsation motions in turbulence are of chaotic type and have an unstable, irregular character, thus constituting a stochastic process. In view of that, one can speak here only on the obtaining of the mean characteristics of the motion of that type (like the moments of various orders) by way of a statistical processing of the results using, for example, kinetic approaches: direct Monte Carlo approaches (similar to the Crook equation, V.Yanitskii [1214, 25-27]) and free turbulence based on the analogy of turbulent mixing with the Brownian motion of liquid particles (S.Ivanov and V.Yanitskii [27]). Numerical simulation of the rarefied hypersonic flow around bodies is of especial interest. Analysis of macroscopic fluctuations in such a flow around a cylinder was performed on the basis of Monte Carlo approach (S.Stefanov [28]). The numerical studies of various kinds of hydrodynamic instabilities (Rayleigh-Taylor, Richtmyer-Meshkov, Kelvin-Helmholtz) are of an unquestionable interest, especially by the three-dimensional simulations extended to the large temporal intervals including turbulent stage. Multimode perturbations and spectral characteristics in 2D and 3D are studied numerically by various conditions. For example, a multi-mode problem, evolution of 2D perturbation into 3D structure, and growth of the thickness of a three-dimensional turbulent mixing zone are analyzed for Rayleigh-Taylor instability development (A.Oparin [13, 14, 29— 31 and others]). Parallel software complex for simulation of spatial atmospheric flows induced by largescale conflagration or explosion was developed. Both natural catastrophes (for instance, largescale forest fire) and big industrial accidents can be considered as a source of such soiled admixtures. By development of convective columns over the seat of a fire a large amount of dust is involving in these columns by rising fluxes of air. Then, soiled admixtures (dust, soot, etc) ascent on big heights over the Earth surface, spread and gradually falling out, result in the soiling of regions of the order of 100 km and more in size. By industrial accident the products of chemical trade or radioactivity can be spread by similar way. A lot of numerical work was made in 2D geometry (both plane and axisymmetric) by different scientists. The experience shows that only numerical modeling in 3D geometry can answer on different relevant questions. 3D modeling requires powerful computational resources and using of parallel computation technology ([14, 32-34, 36]). The development of the pointed approach is contained in mentioned above papers [12-14, 18, 27, 31, 43] and new monograph [34], where the problems of astrophysical turbulence, convection and instability also are included.

5

We note again, that developed methods for investigations of coherent ordered structures are based on consideration of "discrete" Euler equations with dissipative — stable and divergent —conservative finite-different schemes, ("upwind" approximation [12, 35] possessing of scheme viscosity), and do not use special (explicit) subgrid approximations and semi-empirical turbulence models. Stochastic flows in the core of the wake (jet, mixing layer) are investigated at kinetic level by Monte-Carlo methods. Recently the methods, which are rather close to approach developing by us from 1976, began to appear (without references to our works, unfortunately) in abroad also (for example [37-40 and others]). 3. SUPERSOLVER Finally, software for the solution of nonlinear high complex problems (supersolver) is considered [43]. Using the experience of numerical simulation of various 3D problems, Universal Parallel Technology for numerical investigation of 3D problem (with mathematical model based on the set of partial differential equations of hyperbolic type) is under developing for multiprocessor supercomputer. The usage of such technology will reduce significantly the time for probing of new 3D applications. This approach is based on selecting of independent software blocks (there only the parallelization is carried out), implementing various constituent parts of the combined numerical method. Software package for parallel computations of the problems within common approach is presented. The source code is a framework for the development of highscalable parallel software, mathematical model of which is based on the equations of the mentioned type. Goals and advantages of using the parallel framework package are: acceleration of development of new parallel applications, software reuse, effective use of existing parallel computers, possibility to switch rapidly to a new method, object-oriented and componentbased programming, possibility to use IDE to create programs for standard models, etc. Applications of the technology and the methods used are demonstrated on a few multidimensional problems in the different areas (for example, simulation of the development of forest-fire, fire storm, tornado etc.). Fig.2 demonstrates three types of "inheritance tree", on the basis of which supersolver was created: problem oriented, Riemann approximation and hybrid schemes (some details in [43]). 4. APPLICATIONS 4.1. Gas dynamics (CFD) The first calculations of LOS by FLUX-method [12] were made (A.Babakov) many years ago (1978). Turbulent wake (vortex sheet) downstream of a cylinder are shown in fig.3 (a, b) for Mcc=0.54 (0.64) and M^O.9. A lot of calculations for different approximation grids with the perturbations of the different kinds have clearly shown, that stable unsteady solution, which defines large structures and has good agreement with experimental data [4], was obtained. Recent calculations (temperature fields) behind the cylinder are presented in fig.4. The flows near the landing modules in Mars's atmosphere at sub-, trans- and supersonic flight conditions for both zero and nonzero angles of attack are examined with the nonviscous perfect compressible gas model. For numerical simulation of flows and aerodynamics

6

properties of axisymmetrical shaped vehicles at zero angle of attack a tailor-made FLUX program package is employed. This package includes a graphics module that works both in run-time mode and as a graphics postprocessor. For the conical shaped landing module the flow separation is governed by the presence of the edge. For the other form of landing module the separation at the similar regimes is governed by the position of the shock wave closing a local supersonic region, not by the viscosity effects. Simulation of the steady and unsteady 3-dimensional flows is carried out on multiprocessor computational systems using a tailor-made program package FLUX-MCS (FLUX for Multiprocessor Computer Systems). For sub- and transonic regimes the flow in a near wake is unsteady (fig.5 and fig.6). On fig.6 the temperature fields are shown in two perpendicular sections. For numerical simulation of gasdynamical processes GasDynamicsTool [19, 41] the multi-purpose package, based on Large Particle Method, was developed. Package exploits numerical simulation of two- and three-dimensional unsteady processes in multi-component systems of non-viscous/viscous compressible gases. The processes of thermoconductivity, diffusion and chemical reactions might be taken into consideration. A universal method of resolving Euler and Navier-Stokes equations allows applying the package to wide range of gasdynamics phenomena in science and industrial applications. The package fully featured by visualization and preprocessor tools that allow analyzing complex 3D flow with different boundary geometries. The code has several types of solvers with numerous submodels describing different physical and chemical processes. Fig.7 (a, b, c) illustrates the possibilities of this approach. 4.2. Hydrodynamic instabilities For the specified class of the phenomena calculation of 3D problems represents special interest. As example, at fig.8a the turbulent stage of the development of 3D Rayleigh-Taylor instability (t=12) under random initial perturbations is presented. Results were obtained by A.Oparin. Fig. 8c illustrates the accent of upper boundary of bubbles in dependence on combination A t gt2, that characterizes turbulent mixing zone (TMZ) Z=aA t gt2. We can see that in 3D case value of a coefficient is significantly differ from 2D calculation, that represents great practical interest. A number of problems, modeling the large-scale high-energy convective flows in the atmosphere, are solved numerically with the help of the software package based on the described parallel framework. The framework is also utilized (by L.Kraginskij and A.Oparin [43]) in the other application fields. One of the studied problems (with the help of Supersolver) is to investigate numerically dynamics of the pollution, distribution and toxic dust precipitation in the Earth atmosphere for large-scale fire. Nature catastrophes and large industrial accidents are sources of such kind of pollution. The results of the numerical experiment let us explore the phenomenon better and to get a dynamical picture of precipitation of the dust on soil. The obtained results could be used as the basis for predicting high-energy accident subsequences and reducing damage and life loses. Another series of numerical modeling is connected with the phenomena of firestorm, tornado and similar flows with large whirls. We try to build the spatial model of the tornado and show some progress in understanding its dynamics on different stages: from the very beginning to the developed flow with stable structures. Figures 9, 10 and 11 illustrate the results of these calculations.

7

4.3. Seismic data processing Solving of direct seismic problem is a matter of great importance for exploring of medium with complex structure as well as for exploring untraditional deposits of hydro-carbonates: oil, gas and condensed gas. It was recently stated that oil deposits could have not organic nature. So they could be found not only in upper layers under the surface but in crystalline base in porous collectors also. A numerical model is proposed to solve direct seismic 2D and 3D problems (wave propagation) in complex heterogeneous medium. The wave-field initiated by different sources of initial disturbance in elastic heterogeneous medium is treated. The effect of acoustic-elastic boundary is demonstrated in fig. 12 (M.Antonenko). 4.4. Safety of housing and industrial constructions under intensive dynamic loadings One of the most actual problems - safety of housing and industrial constructions under intensive dynamic loadings (industrial accidents, seismological activity, acts of terrorism, falling of planes and fragments of spacecraft and etc.) is considered. The numerical simulation (by I.Petrov) of such problems permits to formulate a quantitative estimation of stability and safety of constructions i.e. to predict the placement and the size of the probable destruction area in dependence of the intensity and character of the impact, a place of the loading, geometrical characteristics of a construction, mechanical properties of materials of which it consists. The behaviour of a lattice concrete construction under a lateral impulse of loading, velocity fields, pressure fields, deformations, areas of destruction are simulated. For mathematical modeling of the behaviour of considered constructions material the dynamic system of the equations in partial derivatives of hyperbolic type of the rigid deformable body mechanics is used. In fig. 13 we can see the character of destruction of lattice building obtained by numerical simulation. It is interesting to note that the propagation of the disturbances has arrow-headed character, not spherical (fig. 13b). 4.5. Nonlinear contact shell dynamics The problems of deformable rigid body mechanics and simulation of nonlinear contact shell dynamics in incompressible fluid for 2D and 3D cases also were investigated (V.Yakushev). In fig. 14 we can see finite element model of liquefied gas-carrier. The calculation of its deformations has great practical importance. In fig. 15 numerical modeling results of collision of the tanker with quay are presented. On character of deformations we can spot the scenario of the collision. 4.6. Computer models in medicine And finally, construction of computer models in medicine is the new perspective direction allowing here again to use rich saved up experience. As example the result of mathematical modeling of circulatory and respiratory systems of human organism and others are presented (A.Kholodov [42], I.Petrov [44], V.Kljuzhev, A.Talalaev and others). In fig. 16 we can see the example of calculated average velocity field in arterial system (A.Kholodov). For modeling of circulatory and respiratory systems of human organism the different systems of equations are used in dependence of anatomic constitution of corresponding human organ: Navier-Stokes equations, equations of filtration type, etc.

8

Special numerical methods (majorant schemes on unstructured grids) were developed for solving such problems. Complicated multi-connected domain of integration is considered [42]. In fig. 17 the model of cranial trauma simulation is presented (I.Petrov). The results of calculation have shown that the propagation of the disturbances after impact is realized on the surface of the skull more faster, than through brain tissue. This fact is very important in clinical practice for the choice of the treatment scenario. In fig. 17b numerical results are presented. We can see clearly the tissue injure. The unique medical and biologic stand was developed in Burdenko Main Clinical Hospital specially for modelling and studying of a collective brain of people (V.Kljuzhev, A.Talalaev, A.Lisitsky, 1997-2002). The results of brain condition modification are analyzed on the basis of methods not realized acoustic suggestion with a position of consideration of a human brain as unique "decoder" received by him meaningly and unconsciously information. The main purpose of present work is the development of the medical and biologic stand created for studying of influence of consciousness and mentality of people on a physical reality by modelling virtual "a collective brain" of the person. On fig. 18 the general view of the stand is shown. Stand unites 8 gauges (polygraphs) and allows to register electric processes of human brain synchronously. On fig. 19 electroencephalograms (EEG) of 4 healthy persons are presented (all persons have received identical task). By special mathematical processing of EEG generalized "virtual collective brain" is created (fig.20a). On fig. 20b EEG of 4 patients are presented (at the left above - injured brain). All this allows modeling the major processes in the human brain! ("...3D computer map of the human brain is composed. 20 000 neurosurgers and neuropathologists worked during several years in 6 countries. The data for this map were obtained as a result of brain scanning of 7 thousands of people. The sense of this work is extremely important - it was clear what regions of brain manage by it's different functions. For example, where is speech born, where is the appetite stimulated, where is aggressive thoughts placed and where is the passionate love pulled out?"... Newspaper "Izvestiya", August 13, 2003, Russia.) 5. CONCLUSION In the nearest future the development of rational models rather than high-speed computers will determine the effectiveness of the numerical experiment in various branches of science and technology. HARD + BRAIN = const!

REFERENCES 1. Kolmogorov A.N. Equations of turbulent motion of incompressible fluid (in Russian)// Izv. Acad. Nauk SSSR. Ser. Fiz. - 1942. - vol.6 - pp.56-58. 2. Batchelor G.K. Theory of homogeneous turbulence - Cambridge University Press.- 1955. 3. Townsend A. A. The structure of turbulent shear flow. — Cambridge: Emmanuel College. 1956. 4. Van Dyke M. Album of fluid motion. - Palo Alto (Calif): Parabolic Press.-1982.

9

5. Hinze J.O. Turbulence.- New York: McGraw-Hill-1975 6. Monin A.S., Yaglom A.M. Statistical fluid mechanics: Mechanics of turbulence.Cambridge Massachusetts: MIT Press.-1975 7. Orszag S.A. Handbook of turbulence. Fundamentals and applications. Ed. by Frost W. and Moulden T.H. - New York - London: Plenum Press. - 1977. - vol. 1 - pp.311 -347. 8. Hydrodynamic instabilities and the transition to turbulence. Ed. by Swinney H.L. and Gollub J.P. Topics in Applied Physics.- Berlin - Heidelberg - New York: SpringerVerlag.-1981-vol.45 9. Strange attractors (Russian translation).- Moscow: Mir.- 1981 10. Cantwell B.J. Organized motions in turbulent flows// Ann. Rev. Fluid Mech.-1981 vol.13-pp.457-515 11. Belotserkovskii O.M. Direct numerical modeling of free induced turbulence (in Russian) // Zh. Vychisl. Mat. i Mat-Fiz. - 1985. - vol.25.-M>12. - pp.1857-1883. [transl. in J. Comput. Maths. Math. Phys. - 1985. - vol.25. -X26. - pp.166-183.] 12. Belotserkovskii O.M. Numerical simulation in the mechanics of continuous media, (in Russian) - Moscow: Fizmatlit 2-nd ed. - 1994.- See also: Modern solution methods for nonlinear multidimensional problems. Mathematics, Mechanics, Turbulence. - USA: Lewinston-Queenston-Lamper. The Edwin Mellen Press. - 2000. 13. Belotserkovskii O.M. and Oparin A.M. Numerical experiment on the turbulence: from order to chaos, (in Russian) - Moscow: Nauka 2-nd ed. - 2000. USA: 1-st ed. translated Bogell House Inc. Publishers// Int. Journal of Fluid Mech. Research. - 1996 - v.23. - N°56.-pp.321-488. 14. Belotserkovskii O.M. Turbulence and instabilities. — Moscow: 1-st ed.// Moscow Inst. of Physics and Technology (MIPT). - 1999.- USA: 2-nd ed., Lewinston-QueenstonLamper. The Edwin Mellen Press. - 2000. 15. Landau L.D. and Lifshitz E.M. Fluid mechanics. - London: Pergamon. - 1956. 16. Herring J.R., Orszag S.A., Kraichnan R.H., Fox D.G. Decay of two-dimensional homogeneous turbulence// J. Fluid Mech. - 1974. - vol.66. - pp.417—444. 17. Kuznetsov V.R., Sabel'nikov V.A. Turbulence and combustion (In Russian). - Moscow: Nauka. - 1986. 18. Belotserkovskii O.M., Davydov Yu.M. Method of large particles in gas dynamics: Numerical experiment (In Russian). - Moscow: Nauka. - 1982. 19. Zibarov A.V. Package of applied programs GAS DYNAMICS TOOL and its application in problems of numerical simulation of gasdynamic processes (In Russian). - Doctoral thesis. - Moscow. - 2000. 20. Belotserkovskii O.M., Gushchin V.A., Kon'shin V.N. A method of splitting for analyzing stratified free-surface flows (In Russian)// Zh. Vychisl. Mat. i Mat. Fiz. - 1987. — vol.27. - Jfe4. - pp.594-609. 21. Belotserkovskii S.O., Mirabel A.P., Chusov M.A. On the construction of supercritical modes for a plane periodic flow (In Russian)// Izv. Akad. Nauk SSSR. Ser. Fiz. Atmosf. i Okeana. - 1978. - vol.14. -Jfcl. - pp.11-20. 22. Belotserkovskii S.O. Simulation of viscous incompressible flows based on Navier-Stokes equations (In Russian). -Doctoral thesis. - Moscow. - 1979. 23. Gushchin V.A., Matyushin P.V. Numerical simulation of separated flow past a sphere// Computational Mathematics and Mathematical Physics. - 1997. - vol.37. - N29. pp. 1086-1100.

10

24. Gushchin V. A. Direct Numerical Simulation of Transitional Separated Fluid Flow Around a Circular Cylinder// Proc. of the colloquium "Transitional Boundary Layers in Aeronautics". — Amsterdam. — 6-8 December 1995. — Royal Netherlands Academy of Arts and Sciences. - The Netherlands. - pp.113-121. 25. Yanitskii V.E. A statistical method for solving some problems of the kinetic theory of gases and turbulence (In Russian). - Doctoral thesis. - Moscow. - 1984. 26. Belotserkovskii O.M., Yanitskii V.E. The statistical method of particles-in-cells for solving problems of rarefied gas dynamics 1, 2 (In Russian)// Zh. Vychisl. Mat. i Mat. Fiz. - 1975. -vol.15. -K5. -pp.1195-1208. -N«6. -pp.1553-1557. 27. Belotserkovskii O.M., Ivanov S.A.,Yanitskii V.E. Direct statistical modeling of some problems in turbulence theory// Computational Mathematics and Mathematical Physics. 1998. - vol.38. - JSb3. - pp.474-487. 28. Stefanov S.K., I.D.Boyd, C.-P. Cai. Monte Carlo analysis of macroscopic fluctuations in a rarefied hypersonic flow around a cylinder// Phys. Fluids. - 2000. - vol.12. - N»5. pp.1226-1239. 29. Inogamov N.A., Oparin A.M. Three-dimensional array structures associated with Richtmyer-Meshkov and Rayleigh-Taylor instability// J. of Experim. and Theor. Physics. - 1999. -vol.89. -X°3.-pp.481-499. 30. Inogamov N.A., Oparin A.M., Dem'yanov A.Yu., Dembitskii L.N., Khokhlov V.A. On stochastic mixing caused by Rayleigh-Taylor Instability// J. of Experim. and Theor. Physics. - 2001. - vol.92. - J\T«4. - pp.715-743. 31. Belotserkovskii O.M., Oparin A.M. A Numerical Study of Three-Dimensional RayleighTaylor Instability Development// Computational Mathematics and Mathematical Physics. - 2000. - vol.40. -Ka7. - p p . 1054-1059. 32. Muzafarov I.F., Utyuzhnikov S.V. Numerical simulation of convective columns over large conflagration into a atmosphere// High Temperature.- 1995.- vol.33. - N°4. pp.588-595. 33. Konyukhov A.V., Meshcheryakov M.V., Utyuzhnikov S.V. Numerical simulation of the processes of propagation of impurity from large-scale source in the atmosphere// High Temperature. - 1999. - vol.37. -N26. - pp.873-879. 34. Belotserkovskii O.M., Oparin A.M., Chechetkin V.M. Turbulence: New Approaches, -(in Russian) - Moscow: Nauka. — 2003. 35. Tolstykh A.I. High accuracy non-centered compact difference schemes for fluid dynamics applications. - Series of Advances in Mathematics for Appl. Sciences - 1994, vol.21 36. Konyukhov A.V., Meshcheryakov M.V., Utyuzhnikov S.V., Chudov L.A. Numerical modeling of turbulent large-scale thermic. // Izvestiya RAS, Series Fluid Mechanics.1997, N3, p.93 37. D.Drikakis, P.Smolarkiewicz. On spurious vertical structures. // J.Comp.Physics.- 2001vol.172.-pp 309-325 38. L.Margolin, W.Rider. A rationale for implicit turbulence modeling. ECCOMAS Comp. Fluid Dynamics Conf., 2001, UK, pp 1-20 39. J.P.Boris, F.F.Grinstein, E.S.Oran, and R.L.Kolbe. New insights into large-eddy simulations. Fluid Dyn. Res., Vol.10, Iss.4-6, pp. 199-228 (1992) 40. C.C.Fereby and F.F.Grinstein. Large Eddy Simulation of High-Reynolds-Number Free and Wall-Bounded Flows. J. Comput. Phys., Vol.181, No.l, pp.68-97 (2002)

11

41. Belotserkovskii O.M., Zibarov A.V. Use GasDynamics Tool 5.0 for Three-Dimensional Shock Wave Processes Simulation. Abstracts of Russian-Indian International Workshop on High Performance Computing in Science and Engineering, Moscow, June 2003 42. Computer models and Progress in Medicine, (in Russian) - Moscow, (under edit. O.M.Belotserkovskii, A.S.Kliolodov) Nauka, 2001 43. Belotserkovskii O.M., Antonenko M.N., Konyukhov A.V., Kraginskij L.M., Oparin A.M., Fortova S.V. Universal Technology of Parallel Computations for the Problems Described by Systems of the Equations of Hyperbolic Type. A step to Supersolver. CFD Journal (Japan) - vol. 11, N 4, Jan 2003, pp 456-466. 44. Petrov I.B. About numerical modeling of biomechanical processes in medicine practice, (in Russian) // Journal of RAS: Information Technology and Computer Systems, vol. 1-2, 2003, pp. 102-112

12

Fig.l. Spectra of energy density E\{k) of longitudinal velocity component fluctuations for various flows. = £"i(Ev3)"1/4, k is a wave number. Choice of the "optimal" scale of resolution h*.

Fig.2. Supersolver: inheritance tree.

Fig. 3a. Turbulent wake downstream of a cylinder. Comparison experimental results (Mo, = 0.54, Sh=0.18, Cx =0.9) with calculation (Moo = 0.64, Sh= 0.178 C x =0.89)

Fig. 3b. Comparison of calculated and experimental results for transonic flow past a circular cylinder. Constant-vorticity lines, M» = 0.9:

Fig. 4. Unsteady regime (cylinder). Temperature field.

„ . Tig.

c

J.

Density field M =2.5. Attack angle =12" Simulation data: CX=1.35 Experimental data; Cx= 1.39

13

Fig. 6. Unsteady regimes. M = 1.0. Temperature field is shown.

Fig. 7a. 3D animations allow achieve more comprehensive recognition of phenomenon or process under consideration.

Fig. 7c. Impacted penetration in ceramic target 3-D cloud image, voxel graphics.

Fig. 7b. Hypersonic flow over rocket stabilizer.

Fig. 8a. Growth of the thickness of 3D turbulent mixing zone (TMZ).

14

Fig. 8b. The graph of ascent of the upper boundary of bubbles for 3D computation and for corresponding 2D one.

Fig. 9. Large-scale fire (if =10 km, Q = 0.05 MWt/sq.m., /off = 60 min)

Fig. 8c. 3D Turbulent Mixing Zone. Simulated self-similar behaviour of «light-to-heavy» penetration depth in 3D in comparison with 2D.

Fig. 10. Large-scale fire storm (R = 10 km, Q = 0.05 MWt/sq.m. toft = 60 min)

15

Fig. 11. Evolution of vertical mesa-scale vortex in tornado-like structure

Fig. 12. Result of numerical simulation. Interaction of plane wave with oil collector: dynamic of wave field. Velocity of vertical displacement shown.

Fig. 13a. Framed construction destruction.

Fig. 13b. Lattice. Safety of lodgings and industrial buildings and constructions. Squared stress deviator. Impact at 45° .

16

Fig. 14. Finite-element model of liquefied gas-carrier.

Fig. 16. The example of calculated average velocity field in arterial system

Fig. 17b. Cranial trauma model: equivalent stress.

Fig. 15. Numerical modeling of collision of the tanker with quay wall

Fig. 17a. Cranial trauma model

Fig. 18. A line of Russian electroencephalographs with an opportunity of realization biofeedback, used at creation of the stand of «a virtual collective brain » modeling.

17

Fig. 19. Synchronous records of electroencephalograms of 4 person in 8 assignations in the making «a virtual collective brain».

Fig. 20. Schematic map of the intercentral communications « virtual collective brain » on an average level of correlation (A), based on EEGof4person(B).

This page is intentionally left blank

Parallel Computational Fluid Dynamics - Advanced Numerical Methods, Software and Applications B. Chetverushkin, A. Ecer, J. Periaux, N. Satofuka and P. Fox (Editors) © 2004 Elsevier B.V. All rights reserved.

19

A Review of Lattice Boltzmann Models for Compressible Flows A.T. Hsua, T. Yang3,1. Lopezb, and A. Ecera a

Department of Mechanical Engineering, Indiana University - Purdue University, Indianapolis Indianapolis, IN 46202-5132, USA e-mail: [email protected] b

NASA Glenn Research Center, Cleveland, Ohio 44135, USA

Keywords: lattice Boltzmann, compressible flow, three-dimension, Euler equation, NavierStokes equation. 1. Introduction This paper gives a review of the lattice Boltzmann method for compressible flows. The lattice Boltzmann method (LBM) is a relatively new numerical approach for simulating complex flow and transport phenomena in cases where direct solution of the Navier-Stokes equations is impractical or undesirable [1,2]. Unlike conventional CFD method based on macroscopic continuum equations, the LBM uses a microscopic equation, i.e., the Boltzmann equation, to determine macroscopic fluid dynamics. The LBM is flexible, has broad applicability, and may be easily adapted for parallel computing. The LBM originated from a Boolean model known as the lattice gas automata (LGA). In a LGA method, the local equilibrium distribution is described by the Fermi-Dirac statistics. As a result, LGA has several shortcomings: high statistical noises, the violation of Galilean transformation invariance in their resulting hydrodynamics equations, and the failure in high Reynolds number computations. To eliminate noise, the Boltzmann equation was used to simulate the lattice gas automata, however, other problems, i.e. non-Galilean invariance and low Reynolds number, remained. These difficulties led to the development of the LB method. The LB models have been successfully applied to various physical problems, such as single component hydrodynamics, multiphase and multi-component fluid flows, magnetohydrodynamics, reaction-diffusion systems, flows through porous media, and other complex systems at small Mach numbers. Unfortunately, as a new CFD tool, the general LB method developed in the past suffered from the constraint of small Mach number because the particle velocities belong to a finite set, and the resulting macroscopic velocity is always much smaller than the speed of sound calculated from the microscopic diffusion velocity. Efforts have been made to increase the allowable Mach number range and to incorporate the effects of temperature into lattice Boltzmann simulations in the past few years. In this paper, we

20

will review the work in this area by various researchers, and will introduce our own contribution in this area. 2. Conventional LB Model The standard LB model starts with the Boltzmann equation: ^ +|-V/ =Q dt

h

(1)

J

Where is the f{x,^,i) distribution function, with \ being the particle velocity, and Q. the collision function. In a typical LB model, the Boltzmann equation is discretized as f(x + £At,4,t + At) = f{x,£,t)+QAt

(2)

And the collision function is replaced by the Bhatnagar, Gross and Krook (BGK) simplified collision model

Where x is the relaxation time. The equilibrium distribution can be written as an approximation of the Maxwell-Boltzmann distribution [1]:

/.w=^(i+3fe.vKffe-vN^j

(4)

Which resulted in a constant speed of sound:

_S

_K

(5)

Since the non-dimensional macroscopic velocity is obtained from averaging £,, and is in general much less than 1, thus in standard LB simulations the Mach number is always less than 0.3, and flows can be assumed to be in the incompressible regime. 3. Adjustable Speed of Sound Models Alexander et al. [3] were the first to propose a Lattice Boltzmann model for compressible fluids, in which the speed of sound, bulk viscosity and kinematic viscosity can all be adjusted, by constructing a modified equilibrium distribution and introducing free parameters into the model. The equilibrium function introduced is:

21

/:q =ctt+S^ + ^pl-v + ^{l-vJ -^v2)p fora = l...6 1

(6)

U^p-^c.-S^-pv1

fora = 0

The distribution is analogous to that of traditional LB models given in equation (4), with two free parameters (cO and 5) introduced to make the speed of sound adjustable:

= s,isothermal

S

= JM

, U V " ^constant

(7)

V j '

With the adjustment of 8, the speed of sound is adjusted, and compressible effect can then be simulated. A shock formation governed by the one dimensional Burger's Equation was simulated by Alexander et al. A sinusoidal wave was used as initial conditions. Because of the non-negative requirement of the equilibrium function given in equation (6), the range values of cO and 8 are small and thus the range of the speed of sound is very limited. Simulations using this model is restricted to small Mach numbers. Yu and Zhao [4] introduced an attractive force to reduce the sound speed and to alleviate the small Mach number restriction:

Where the attractive force is given as 1

2-

Oa=-gcsea

(9) Vp

Chapman-Enskog expansion shows that the model has effective speed of sound as

cs=fi^cs

(10)

The speed of sound can thus be adjusted by selecting the parameter g. A simple wave induced by a density vibration at a single point in a uniform flow field was simulated, with the density at the singular point oscillating with a sine function. The parameter g can be as large as one, which causes a zero speed of sound; the smallest value for g is zero, providing a speed of sound that is the same as the original standard model. Thus the scheme is limited to only high Mach number flows. The models mentioned in this section, though able to handle some compressibility effects, are limited to isothermal flows, and no temperature variation is allowed.

22

4. Full thermal LB models Hinton, et al [5] proposal an advection with over-relaxation (AOR) algorithm in which the LB equation is written as fa{x + SalSt,t + M)-fa{x,t) =

~{f?{x,t)-f{x,t))

Where the modified distribution function is written as f'a{xj) = Pf{t-$aM,t-At) + (\-p)f{x-2iaM,t-2ht)

Through Chapman-Enskog expansion, it is shown that by adjusting (3 the second order dissipation coefficients (viscosity and thermal diffusivity) can be made arbitrary small, while fourth order dissipation coefficients (hyperviscosity and hyperdiffusivity) are maintained positive. For instance, the viscosity is given as (2-3/?/2)A/ (2-/?)

^

Simple sound wave propagation problems were simulated by the authors, and density and temperature changes in the wave were successfully evaluated. However, some lack of realism exists in the model: viscosity and thermal diffusivity decrease with increasing Mach number. Other restrictions include: Mach number must be less than one. Flows with M=0.1, 0.2 were simulated. Arguing that the poor stability of the traditional LB model was due to insufficient symmetry velocity set, Soe et al [6] proposed a model in which the velocity set has higher order symmetry than the traditional ones; and consequently the equilibrium distribution was represented by a higher order polynomial. The equilibrium distribution is approximated by the following polynomial functions

/ - = p(A(e)+ B(eil v)+ C{e{l vj + D(e)v2 -vj + G{e% -vjv2 +H{e)v4) + E(eil -vY+Fieil In a traditional LB model, the equilibrium distribution is approximated by the first four terms in the above equation. To achieve variable Pr number for the model, a multiple relaxation times collision were introduced:

23

j

u

php

Rayleigh-Benard Convection was simulated using the above model. Palmer and Rector [7] recently introduced a thermal LB model in which the internal energy is modeled as a scalar field that is governed by a second distribution (the other distribution governs mass and momentum of the flow as in traditional LB models). Thus two sets of LB equations are solved:

fa{x+Ati,t+At)-fa(x,t)=-—(fa(x,'hf:q(z4

Fa(x + to$a,t + to)-Fa(x,t) = ~{Fa{x,t)-F?(x,t)) with the second one governs the energy distribution function. A Rayleigh-Benard flow, a thermal conduction problem for two plates with variable conductivity in the medium between the two plates and an entry length behavior problem in a channel flow were successfully studied using this model. 5. Adaptive Velocity Compressible LB Model Recently, we proposed a locally adaptive LB model based on a large particle-velocity set so that mean flow may have a high velocity [8]. This model is suitable for a wide range of Mach numbers (from Mach numbers below 0.3 to hypersonic Mach numbers), and does not consume much computer resource. Compressible Navier-Stokes equations including the energy equation are derived from the BGK lattice Boltzmann equation; therefore, this model can simulate compressible viscous flows that include heat transfer. The basic idea of the present LBM for compressible flows is to allow particles to travel a distance that is much larger than the grid size of the lattices, and by doing so the velocity can now be arbitrarily larger (or smaller) than the speed of sound, which is determined by the molecular diffusion velocity. In order to achieve this goal, we symbolically decompose the discrete migrating velocity, c, of a particle into two components: c = v + c'.

(11)

where the first component v is the macroscopic fluid velocity, and the second component c' is the molecular diffusion velocity.

24

The macroscopic velocity, v, is a continuous quantity, which can be used to evaluate the momentum and kinetic energy carried by a particle, but it cannot be used to determine particle migration since in an LB model a particle must move from one node to another at a time step of a time marching procedure. To address this issue, we introduce a discrete macroscopic velocity, v*, as an approximation to v. Supposing that the macroscopic velocity, v, carries a particle from its originating node into a interior point of a lattice cube, we introduce a set of correction velocity vectors, v'* (k=l, 2,..., 8), that will carry fractions of the particle from the interior point to the eight nodes of the destination lattice cube (see Fig. 2). We then define the modified macroscopic velocity as the sum of the exact macroscopic velocity, v, and the correction velocity, \\:

\k = v + \'k . (12)

For high-speed flows the fluctuating velocities v'* are small compared to v. With this modification to the macroscopic velocity, Figure 1. Velocity set the molecular diffusion velocity can now easily be defined on a uniform lattice. We consider a symmetric vector set {c); j=\, ..., n) connecting a node to its equal distanced neighboring nodes, where n is the number of vectors. If we take At=l, then these vectors are equivalent to the length of the lattice sides. In the following description, At=l is implied, and velocities are used as distances without further Figure 2. Redistribution of particles explanation. We choose the number of vectors K=12, with two velocity levels and six directions for each level. The modulus of c'y is c'i forj=l,...,6 and c'2 forj=l,...,\2 (see Fig. 1). c'i and c'2 must be a multiple of the grid length. The symmetric particle velocity set {c'y} defined here is similar to a particle velocity set of the traditional lattice Boltzmann or lattice gas methods with multi velocity levels. With the above definitions of macroscopic and microscopic velocities, we define a discrete velocity set, So(Cyt), through the following relation:

c* = \k + c).

(13)

Figs. 1 and 2 demonstrate how the discrete particle velocities c# ensure the particle to move from one node to another during one time step.

25

The equilibrium distribution is defined as follows,

^

C

V

)

^ [

^

"

^

"

0

^

^

"

^

fOrC = C

* , for other c

(14)

where