Entropy and Free Energy in Structural Biology: From Thermodynamics to Statistical Mechanics to Computer Simulation Book 9780367406929, 9780367854782

Table of contents :
Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Table of Contents
Preface
Acknowledgments
Author
Section I: Probability Theory
1: Probability and Its Applications
1.1 Introduction
1.2 Experimental Probability
1.3 The Sample Space Is Related to the Experiment
1.4 Elementary Probability Space
1.5 Basic Combinatorics
1.5.1 Permutations
1.5.2 Combinations
1.6 Product Probability Spaces
1.6.1 The Binomial Distribution
1.6.2 Poisson Theorem
1.7 Dependent and Independent Events
1.7.1 Bayes Formula
1.8 Discrete Probability—Summary
1.9 One-Dimensional Discrete Random Variables
1.9.1 The Cumulative Distribution Function
1.9.2 The Random Variable of the Poisson Distribution
1.10 Continuous Random Variables
1.10.1 The Normal Random Variable
1.10.2 The Uniform Random Variable
1.11 The Expectation Value
1.11.1 Examples
1.12 The Variance
1.12.1 The Variance of the Poisson Distribution
1.12.2 The Variance of the Normal Distribution
1.13 Independent and Uncorrelated Random Variables
1.13.1 Correlation
1.14 The Arithmetic Average
1.15 The Central Limit Theorem
1.16 Sampling
1.17 Stochastic Processes—Markov Chains
1.17.1 The Stationary Probabilities
1.18 The Ergodic Theorem
1.19 Autocorrelation Functions
1.19.1 Stationary Stochastic Processes
Homework for Students
A Comment about Notations
References
Section II: Equilibrium Thermodynamics and Statistical Mechanics
2: Classical Thermodynamics
2.1 Introduction
2.2 Macroscopic Mechanical Systems versus Thermodynamic Systems
2.3 Equilibrium and Reversible Transformations
2.4 Ideal Gas Mechanical Work and Reversibility
2.5 The First Law of Thermodynamics
2.6 Joule’s Experiment
2.7 Entropy
2.8 The Second Law of Thermodynamics
2.8.1 Maximal Entropy in an Isolated System
2.8.2 Spontaneous Expansion of an Ideal Gas and Probability
2.8.3 Reversible and Irreversible Processes Including Work
2.9 The Third Law of Thermodynamics
2.10 Thermodynamic Potentials
2.10.1 The Gibbs Relation
2.10.2 The Entropy as the Main Potential
2.10.3 The Enthalpy
2.10.4 The Helmholtz Free Energy
2.10.5 The Gibbs Free Energy
2.10.6 The Free Energy, , H.(T,µ)
2.11 Maximal Work in Isothermal and Isobaric Transformations
2.12 Euler’s Theorem and Additional Relations for the Free Energies
2.12.1 Gibbs-Duhem Equation
2.13 Summary
Homework for Students
References
Further Reading
3: From Thermodynamics to Statistical Mechanics
3.1 Phase Space as a Probability Space
3.2 Derivation of the Boltzmann Probability
3.3 Statistical Mechanics Averages
3.3.1 The Average Energy
3.3.2 The Average Entropy
3.3.3 The Helmholtz Free Energy
3.4 Various Approaches for Calculating Thermodynamic Parameters
3.4.1 Thermodynamic Approach
3.4.2 Probabilistic Approach
3.5 The Helmholtz Free Energy of a Simple Fluid
Reference
Further Reading
4: Ideal Gas and the Harmonic Oscillator
4.1 From a Free Particle in a Box to an Ideal Gas
4.2 Properties of an Ideal Gas by the Thermodynamic Approach
4.3 The chemical potential of an Ideal Gas
4.4 Treating an Ideal Gas by the Probability Approach
4.5 The Macroscopic Harmonic Oscillator
4.6 The Microscopic Oscillator
4.6.1 Partition Function and Thermodynamic Properties
4.7 The Quantum Mechanical Oscillator
4.8 Entropy and Information in Statistical Mechanics
4.9 The Configurational Partition Function
Homework for Students
References
Further Reading
5: Fluctuations and the Most Probable Energy
5.1 The Variances of the Energy and the Free Energy
5.2 The Most Contributing Energy E*
5.3 Solving Problems in Statistical Mechanics
5.3.1 The Thermodynamic Approach
5.3.2 The Probabilistic Approach
5.3.3 Calculating the Most Probable Energy Term
5.3.4 The Change of Energy and Entropy with Temperature
References
6: Various Ensembles
6.1 The Microcanonical (petit) Ensemble
6.2 The Canonical (NVT) Ensemble
6.3 The Gibbs (NpT) Ensemble
6.4 The Grand Canonical (µVT) Ensemble
6.5 Averages and Variances in Different Ensembles
6.5.1 A Canonical Ensemble Solution (Maximal Term Method)
6.5.2 A Grand-Canonical Ensemble Solution
6.5.3 Fluctuations in Different Ensembles
References
Further Reading
7: Phase Transitions
7.1 Finite Systems versus the Thermodynamic Limit
7.2 First-Order Phase Transitions
7.3 Second-Order Phase Transitions
References
8: Ideal Polymer Chains
8.1 Models of Macromolecules
8.2 Statistical Mechanics of an Ideal Chain
8.2.1 Partition Function and Thermodynamic Averages
8.3 Entropic Forces in an One-Dimensional Ideal Chain
8.4 The Radius of Gyration
8.5 The Critical Exponent ν
8.6 Distribution of the End-to-End Distance
8.6.1 Entropic Forces Derived from the Gaussian Distribution
8.7 The Distribution of the End-to-End Distance Obtained from the Central Limit Theorem
8.8 Ideal Chains and the Random Walk
8.9 Ideal Chain as a Model of Reality
References
9: Chains with Excluded Volume
9.1 The Shape Exponent ν for Self-avoiding Walks
9.2 The Partition Function
9.3 Polymer Chain as a Critical System
9.4 Distribution of the End-to-End Distance
9.5 The Effect of Solvent and Temperature on the Chain Size
9.5.1 θ Chains in d = 3
9.5.2 θ Chains in d = 2
9.5.3 The Crossover Behavior Around
9.5.4 The Blob Picture
9.6 Summary
References
Section III: Topics in Non-Equilibrium Thermodynamics and Statistical Mechanics
10: Basic Simulation Techniques: Metropolis Monte Carlo and Molecular Dynamics
10.1 Introduction
10.2 Sampling the Energy and Entropy and New Notations
10.3 More About Importance Sampling
10.4 The Metropolis Monte Carlo Method
10.4.1 Symmetric and Asymmetric MC Procedures
10.4.2 A Grand-Canonical MC Procedure
10.5 Efficiency of Metropolis MC
10.6 Molecular Dynamics in the Microcanonical Ensemble
10.7 MD Simulations in the Canonical Ensemble
10.8 Dynamic MD Calculations
10.9 Efficiency of MD
10.9.1 Periodic Boundary Conditions and Ewald Sums
10.9.2 A Comment About MD Simulations and Entropy
References
11: Non-Equilibrium Thermodynamics—Onsager Theory
11.1 Introduction
11.2 The Local-Equilibrium Hypothesis
11.3 Entropy Production Due to Heat Flow in a Closed System
11.4 Entropy Production in an Isolated System
11.5 Extra Hypothesis: A Linear Relation Between Rates and Affinities
11.5.1 Entropy of an Ideal Linear Chain Close to Equilibrium
11.6 Fourier’s Law—A Continuum Example of Linearity
11.7 Statistical Mechanics Picture of Irreversibility
11.8 Time Reversal, Microscopic Reversibility, and the Principle of Detailed Balance
11.9 Onsager’s Reciprocal Relations
11.10 Applications
11.11 Steady States and the Principle of Minimum Entropy Production
11.12 Summary
References
12: Non-equilibrium Statistical Mechanics
12.1 Fick’s Laws for Diffusion
12.1.1 First Fick’s Law
12.1.2 Calculation of the Flux from Thermodynamic Considerations
12.1.3 The Continuity Equation
12.1.4 Second Fick’s Law—The Diffusion Equation
12.1.5 Diffusion of Particles Through a Membrane
12.1.6 Self-Diffusion
12.2 Brownian Motion: Einstein’s Derivation of the Diffusion Equation
12.3 Langevin Equation
12.3.1 The Average Velocity and the Fluctuation-Dissipation Theorem
12.3.2 Correlation Functions
12.3.3 The Displacement of a Langevin Particle
12.3.4 The Probability Distributions of the Velocity and the Displacement
12.3.5 Langevin Equation with a Charge in an Electric Field
12.3.6 Langevin Equation with an External Force—The Strong Damping Velocity
12.4 Stochastic Dynamics Simulations
12.4.1 Generating Numbers from a Gaussian Distribution by CLT
12.4.2 Stochastic Dynamics versus Molecular Dynamics
12.5 The Fokker-Planck Equation
12.6 Smoluchowski Equation
12.7 The Fokker-Planck Equation for a Full Langevin Equation with a Force
12.8 Summary of Pairs of Equations
References
13: The Master Equation
13.1 Master Equation in a Microcanonical System
13.2 Master Equation in the Canonical Ensemble
13.3 An Example from Magnetic Resonance
13.3.1 Relaxation Processes Under Various Conditions
13.3.2 Steady State and the Rate of Entropy Production
13.4 The Principle of Minimum Entropy Production—Statistical Mechanics Example
References
Section IV: Advanced Simulation Methods: Polymers and Biological Macromolecules
14: Growth Simulation Methods for Polymers
14.1 Simple Sampling of Ideal Chains
14.2 Simple Sampling of SAWs
14.3 The Enrichment Method
14.4 The Rosenbluth and Rosenbluth Method
14.5 The Scanning Method
14.5.1 The Complete Scanning Method
14.5.2 The Partial Scanning Method
14.5.3 Treating SAWs with Finite Interactions
14.5.4 A Lower Bound for the Entropy
14.5.5 A Mean-Field Parameter
14.5.6 Eliminating the Bias by Schmidt’s Procedure
14.5.7 Correlations in the Accepted Sample
14.5.8 Criteria for Efficiency
14.5.9 Locating Transition Temperatures
14.5.10 The Scanning Method versus Other Techniques
14.5.11 The Stochastic Double Scanning Method
14.5.12 Future Scanning by Monte Carlo
14.5.13 The Scanning Method for the Ising Model and Bulk Systems
14.6 The Dimerization Method
References
15: The Pivot Algorithm and Hybrid Techniques
15.1 The Pivot Algorithm—Historical Notes
15.2 Ergodicity and Efficiency
15.3 Applicability
15.4 Hybrid and Grand-Canonical Simulation Methods
15.5 Concluding Remarks
References
16: Models of Proteins
16.1 Biological Macromolecules versus Polymers
16.2 Definition of a Protein Chain
16.3 The Force Field of a Protein
16.4 Implicit Solvation Models
16.5 A Protein in an Explicit Solvent
16.6 Potential Energy Surface of a Protein
16.7 The Problem of Protein Folding
16.8 Methods for a Conformational Search
16.8.1 Local Minimization—The Steepest Descents Method
16.8.2 Monte Carlo Minimization
16.8.3 Simulated Annealing
16.9 Monte Carlo and Molecular Dynamics Applied to Proteins
16.10 Microstates and Intermediate Flexibility
16.10.1 On the Practical Definition of a Microstate
References
17: Calculation of the Entropy and the Free Energy by Thermodynamic Integration
17.1 “Calorimetric” Thermodynamic Integration
17.2 The Free Energy Perturbation Formula
17.3 The Thermodynamic Integration Formula of Kirkwood
17.4 Applications
17.4.1 Absolute Entropy of a SAW Integrated from an Ideal Chain Reference State
17.4.2 Harmonic Reference State of a Peptide
17.5 Thermodynamic Cycles
17.5.1 Other Cycles
17.5.2 Problems of TI and FEP Applied to Proteins
References
18: Direct Calculation of the Absolute Entropy and Free Energy
18.1 Absolute Free Energy from
18.2 The Harmonic Approximation
18.3 The M2 Method
18.4 The Quasi-Harmonic Approximation
18.5 The Mutual Information Expansion
18.6 The Nearest Neighbor Technique
18.7 The MIE-NN Method
18.8 Hybrid Approaches
References
19: Calculation of the Absolute Entropy from a Single Monte Carlo Sample
19.1 The Hypothetical Scanning (HS) Method for SAWs
19.1.1 An Exact HS Method
19.1.2 Approximate HS Method
19.2 The HS Monte Carlo (HSMC) Method
19.3 Upper Bounds and Exact Functionals for the Free Energy
19.3.1 The Upper Bound FB
19.3.2 FB Calculated by the Reversed Schmidt Procedure
19.3.3 A Gaussian Estimation of FB
19.3.4 Exact Expression for the Free Energy
19.3.5 The Correlation Between sA and FA
19.3.6 Entropy Results for SAWs on a Square Lattice
19.4 HS and HSMC Applied to the Ising Model
19.5 The HS and HSMC Methods for a Continuum Fluid
19.5.1 The HS Method
19.5.2 The HSMC Method
19.5.3 Results for Argon and Water
19.5.3.1 Results for Argon
19.5.3.2 Results for Water
19.6 HSMD Applied to a Peptide
19.6.1 Applications
19.7 The HSMD-TI Method
19.8 The LS Method
19.8.1 The LS Method Applied to the Ising Model
19.8.2 The LS Method Applied to a Peptide
References
20: The Potential of Mean Force, Umbrella Sampling, and Related Techniques
20.1 Umbrella Sampling
20.2 Bennett’s Acceptance Ratio
20.3 The Potential of Mean Force
20.3.1 Applications
20.4 The Self-Consistent Histogram Method
20.4.1 Free Energy from a Single Simulation
20.4.2 Multiple Simulations and The Self-Consistent Procedure
20.5 The Weighted Histogram Analysis Method
20.5.1 The Single Histogram Equations
20.5.2 The WHAM Equations
20.5.3 Enhancements of WHAM
20.5.4 The Basic MBAR Equation
20.5.5 ST-WHAM and UIM
20.5.6 Summary
References
21: Advanced Simulation Methods and Free Energy Techniques
21.1 Replica-Exchange
21.1.1 Temperature-Based REM
21.1.2 Hamiltonian-Dependent Replica Exchange
21.2 The Multicanonical Method
21.2.1 Applications
21.2.2 MUCA-Summary
21.3 The Method of Wang and Landau
21.3.1 The Wang and Landau Method-Applications
21.4 The Method of Expanded Ensembles
21.4.1 The Method of Expanded Ensembles-Applications
21.5 The Adaptive Integration Method
21.6 Methods Based on Jarzynski’s Identity
21.6.1 Jarzynski’s Identity versus Other Methods for Calculating ΔF
21.7 Summary
References
22: Simulation of the Chemical Potential
22.1 The Widom Insertion Method
22.2 The Deletion Procedure
22.3 Personage’s Method for Treating Deletion
22.4 Introduction of a Hard Sphere
22.5 The Ideal Gas Gauge Method
22.6 Calculation of the Chemical Potential of a Polymer by the Scanning Method
22.7 The Incremental Chemical Potential Method for Polymers
22.8 Calculation of µ by Thermodynamic Integration
References
23: The Absolute Free Energy of Binding
23.1 The Law of Mass Action
23.2 Chemical Potential, Fugacity, and Activity of an Ideal Gas
23.2.1 Thermodynamics
23.2.2 Canonical Ensemble
23.2.3 NpT Ensemble
23.3 Chemical Potential in Ideal Solutions: Raoult’s and Henry’s Laws
23.3.1 Raoult’s Law
23.3.2 Henry’s Law
23.4 Chemical Potential in Non-ideal Solutions
23.4.1 Solvent
23.4.2 Solute
23.5 Thermodynamic Treatment of Chemical Equilibrium
23.6 Chemical Equilibrium in Ideal Gas Mixtures: Statistical Mechanics
23.7 Pressure-Dependent Equilibrium Constant of Ideal Gas Mixtures
23.8 Protein-Ligand Binding
23.8.1 Standard Methods for Calculating .A0
23.8.2 Calculating .A0 by HSMD-TI
23.8.3 HSMD-TI Applied to the FKBP12-FK506 Complex: Equilibration
23.8.4 The Internal and External Entropies
23.8.5 TI Results for FKBP12-FK506
23.8.6 .A0 Results for FKBP12-FK506
23.9 Summary
References
Appendix
Index

Citation preview

Foundations of Biochemistry and Biophysics

ENTROPY AND FREE ENERGY IN STRUCTURAL BIOLOGY FROM THERMODYNAMICS TO STATISTICAL MECHANICS TO COMPUTER SIMULATION Hagai Meirovitch

Entropy and Free Energy in Structural Biology

Foundations of Biochemistry and Biophysics This textbook series focuses on foundational principles and experimental approaches across all areas of biological physics, covering core subjects in a modern biophysics curriculum. Individual titles address such topics as molecular biophysics, statistical biophysics, molecular modeling, single-molecule biophysics, and chemical biophysics. It is aimed at advanced undergraduate- and graduate-level curricula at the intersection of biological and physical sciences. The goal of the series is to facilitate interdisciplinary research by training biologists and biochemists in quantitative aspects of modern biomedical research and to teach key biological principles to students in physical sciences and engineering. New books in the series are commissioned by invitation. Authors are also welcome to contact the publisher (Lou Chosen, Executive Editor: [email protected]) to discuss new title ideas. System Immunology: An Introduction to Modelling Methods for Scientists Jayajit Das, Ciriyam Jayaprakash An Introduction to Single Molecule Biophysics Yuri L. Lyubchenko Light Harvesting in Photosynthesis Roberta Croce, Rienk van Grondelle, Herbert van Amerongen, Ivo van Stokkum (Eds.) An Introduction to Single Molecule Biophysics Yuri L. Lyubchenko (Ed.) Biomolecular Kinetics: A Step-by-Step Guide Clive R. Bagshaw Biomolecular Thermodynamics: From Theory to Application Douglas E. Barrick Quantitative Understanding of Biosystems: An Introduction to Biophysics Thomas M. Nordlund Quantitative Understanding of Biosystems: An Introduction to Biophysics, Second Edition Thomas M. Nordlund, Peter M. Hoffmann Entropy and Free Energy in Structural Biology: From Thermodynamics to Statistical Mechanics to Computer Simulation Hagai Meirovitch

https://www.crcpress.com/Foundations-of-Biochemistry-and-Biophysics/book-series/CRCFOUBIOPHY

Entropy and Free Energy in Structural Biology From Thermodynamics to Statistical Mechanics to Computer Simulation

Hagai Meirovitch

First edition published 2020 by CRC Press 6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742 and by CRC Press 2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN © 2021 Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, LLC Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, access www.copyright.com or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are not available on CCC please contact [email protected] Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. ISBN: 978-0-367-40692-9 (hbk) ISBN: 978-0-367-85478-2 (ebk) Typeset in Times by Lumina Datamatics Limited

To Eva, Avner, Ofra Alon, Amir, Haim, and Noga

Contents Preface...................................................................................................................................................... xv Acknowledgments.................................................................................................................................... xix Author...................................................................................................................................................... xxi

Section I  Probability Theory 1. Probability and Its Applications...................................................................................................... 3 1.1 Introduction.............................................................................................................................. 3 1.2 Experimental Probability......................................................................................................... 3 1.3 The Sample Space Is Related to the Experiment..................................................................... 4 1.4 Elementary Probability Space................................................................................................. 5 1.5 Basic Combinatorics................................................................................................................ 6 1.5.1 Permutations.............................................................................................................. 6 1.5.2 Combinations............................................................................................................. 7 1.6 Product Probability Spaces...................................................................................................... 9 1.6.1 The Binomial Distribution........................................................................................11 1.6.2 Poisson Theorem.......................................................................................................11 1.7 Dependent and Independent Events....................................................................................... 12 1.7.1 Bayes Formula......................................................................................................... 12 1.8 Discrete Probability—Summary........................................................................................... 13 1.9 One-Dimensional Discrete Random Variables...................................................................... 13 1.9.1 The Cumulative Distribution Function.....................................................................14 1.9.2 The Random Variable of the Poisson Distribution...................................................14 1.10 Continuous Random Variables...............................................................................................14 1.10.1 The Normal Random Variable................................................................................. 15 1.10.2 The Uniform Random Variable............................................................................... 15 1.11 The Expectation Value............................................................................................................16 1.11.1 Examples...................................................................................................................16 1.12 The Variance...........................................................................................................................17 1.12.1 The Variance of the Poisson Distribution.................................................................18 1.12.2 The Variance of the Normal Distribution.................................................................18 1.13 Independent and Uncorrelated Random Variables................................................................ 19 1.13.1 Correlation............................................................................................................... 19 1.14 The Arithmetic Average........................................................................................................ 20 1.15 The Central Limit Theorem................................................................................................... 21 1.16 Sampling................................................................................................................................ 23 1.17 Stochastic Processes—Markov Chains................................................................................. 23 1.17.1 The Stationary Probabilities.................................................................................... 25 1.18 The Ergodic Theorem............................................................................................................ 26 1.19 Autocorrelation Functions..................................................................................................... 27 1.19.1 Stationary Stochastic Processes............................................................................... 28 Homework for Students..................................................................................................................... 28 A Comment about Notations............................................................................................................. 28 References......................................................................................................................................... 29

vii

viii

Contents

Section II  Equilibrium Thermodynamics and Statistical Mechanics 2. Classical Thermodynamics............................................................................................................ 33 2.1 Introduction............................................................................................................................ 33 2.2 Macroscopic Mechanical Systems versus Thermodynamic Systems................................... 33 2.3 Equilibrium and Reversible Transformations........................................................................ 34 2.4 Ideal Gas Mechanical Work and Reversibility...................................................................... 34 2.5 The First Law of Thermodynamics....................................................................................... 36 2.6 Joule’s Experiment................................................................................................................. 37 2.7 Entropy................................................................................................................................... 39 2.8 The Second Law of Thermodynamics................................................................................... 40 2.8.1 Maximal Entropy in an Isolated System..................................................................41 2.8.2 Spontaneous Expansion of an Ideal Gas and Probability........................................ 42 2.8.3 Reversible and Irreversible Processes Including Work............................................ 42 2.9 The Third Law of Thermodynamics..................................................................................... 43 2.10 Thermodynamic Potentials.................................................................................................... 43 2.10.1 The Gibbs Relation.................................................................................................. 43 2.10.2 The Entropy as the Main Potential.......................................................................... 44 2.10.3 The Enthalpy............................................................................................................ 45 2.10.4 The Helmholtz Free Energy..................................................................................... 45 2.10.5 The Gibbs Free Energy............................................................................................ 45 2.10.6 The Free Energy, H ( T , µ). ....................................................................................... 46 2.11 Maximal Work in Isothermal and Isobaric Transformations................................................ 47 2.12 Euler’s Theorem and Additional Relations for the Free Energies......................................... 48 2.12.1 Gibbs-Duhem Equation........................................................................................... 49 2.13 Summary................................................................................................................................ 49 Homework for Students..................................................................................................................... 49 References......................................................................................................................................... 49 Further Reading................................................................................................................................. 49 3. From Thermodynamics to Statistical Mechanics.........................................................................51 3.1 Phase Space as a Probability Space........................................................................................51 3.2 Derivation of the Boltzmann Probability.............................................................................. 52 3.3 Statistical Mechanics Averages............................................................................................. 54 3.3.1 The Average Energy................................................................................................. 54 3.3.2 The Average Entropy............................................................................................... 54 3.3.3 The Helmholtz Free Energy..................................................................................... 55 3.4 Various Approaches for Calculating Thermodynamic Parameters....................................... 55 3.4.1 Thermodynamic Approach...................................................................................... 55 3.4.2 Probabilistic Approach............................................................................................ 56 3.5 The Helmholtz Free Energy of a Simple Fluid...................................................................... 56 Reference........................................................................................................................................... 58 Further Reading................................................................................................................................. 58 4. Ideal Gas and the Harmonic Oscillator........................................................................................ 59 4.1 From a Free Particle in a Box to an Ideal Gas....................................................................... 59 4.2 Properties of an Ideal Gas by the Thermodynamic Approach.............................................. 60 4.3 The chemical potential of an Ideal Gas................................................................................. 62 4.4 Treating an Ideal Gas by the Probability Approach.............................................................. 63 4.5 The Macroscopic Harmonic Oscillator................................................................................. 64 4.6 The Microscopic Oscillator................................................................................................... 65 4.6.1 Partition Function and Thermodynamic Properties................................................ 66 4.7 The Quantum Mechanical Oscillator.................................................................................... 68

Contents

ix

4.8 Entropy and Information in Statistical Mechanics................................................................ 71 4.9 The Configurational Partition Function................................................................................. 71 Homework for Students..................................................................................................................... 72 References......................................................................................................................................... 72 Further Reading................................................................................................................................. 72 5. Fluctuations and the Most Probable Energy................................................................................ 73 5.1 The Variances of the Energy and the Free Energy................................................................ 73 5.2 The Most Contributing Energy E*........................................................................................ 74 5.3 Solving Problems in Statistical Mechanics........................................................................... 76 5.3.1 The Thermodynamic Approach............................................................................... 77 5.3.2 The Probabilistic Approach..................................................................................... 78 5.3.3 Calculating the Most Probable Energy Term........................................................... 79 5.3.4 The Change of Energy and Entropy with Temperature........................................... 80 References......................................................................................................................................... 81 6. Various Ensembles.......................................................................................................................... 83 6.1 The Microcanonical (petit) Ensemble................................................................................... 83 6.2 The Canonical (NVT) Ensemble............................................................................................ 84 6.3 The Gibbs (NpT) Ensemble................................................................................................... 85 6.4 The Grand Canonical (μVT) Ensemble................................................................................. 88 6.5 Averages and Variances in Different Ensembles................................................................... 90 6.5.1 A Canonical Ensemble Solution (Maximal Term Method)..................................... 90 6.5.2 A Grand-Canonical Ensemble Solution................................................................... 91 6.5.3 Fluctuations in Different Ensembles....................................................................... 91 References......................................................................................................................................... 92 Further Reading................................................................................................................................. 92 7. Phase Transitions............................................................................................................................ 93 7.1 Finite Systems versus the Thermodynamic Limit................................................................. 93 7.2 First-Order Phase Transitions................................................................................................ 94 7.3 Second-Order Phase Transitions............................................................................................ 95 References......................................................................................................................................... 98 8. Ideal Polymer Chains...................................................................................................................... 99 8.1 Models of Macromolecules.................................................................................................... 99 8.2 Statistical Mechanics of an Ideal Chain................................................................................ 99 8.2.1 Partition Function and Thermodynamic Averages................................................ 100 8.3 Entropic Forces in an One-Dimensional Ideal Chain..........................................................101 8.4 The Radius of Gyration....................................................................................................... 104 8.5 The Critical Exponent ν....................................................................................................... 105 8.6 Distribution of the End-to-End Distance............................................................................. 106 8.6.1 Entropic Forces Derived from the Gaussian Distribution..................................... 107 8.7 The Distribution of the End-to-End Distance Obtained from the Central Limit Theorem..... 108 8.8 Ideal Chains and the Random Walk.................................................................................... 109 8.9 Ideal Chain as a Model of Reality........................................................................................110 References........................................................................................................................................110 9. Chains with Excluded Volume......................................................................................................111 9.1 The Shape Exponent ν for Self-avoiding Walks...................................................................111 9.2 The Partition Function..........................................................................................................112 9.3 Polymer Chain as a Critical System.....................................................................................113 9.4 Distribution of the End-to-End Distance..............................................................................114

x

Contents 9.5

The Effect of Solvent and Temperature on the Chain Size..................................................115 9.5.1 θ Chains in d = 3....................................................................................................116 9.5.2 θ Chains in d = 2....................................................................................................116 9.5.3 The Crossover Behavior Around θ.........................................................................117 9.5.4 The Blob Picture.....................................................................................................118 9.6 Summary...............................................................................................................................119 References........................................................................................................................................119

Section III Topics in Non-Equilibrium Thermodynamics and Statistical Mechanics 10. Basic Simulation Techniques: Metropolis Monte Carlo and Molecular Dynamics............... 123 10.1 Introduction.......................................................................................................................... 123 10.2 Sampling the Energy and Entropy and New Notations....................................................... 124 10.3 More About Importance Sampling...................................................................................... 125 10.4 The Metropolis Monte Carlo Method.................................................................................. 126 10.4.1 Symmetric and Asymmetric MC Procedures........................................................ 127 10.4.2 A Grand-Canonical MC Procedure....................................................................... 128 10.5 Efficiency of Metropolis MC............................................................................................... 129 10.6 Molecular Dynamics in the Microcanonical Ensemble.......................................................131 10.7 MD Simulations in the Canonical Ensemble....................................................................... 134 10.8 Dynamic MD Calculations...................................................................................................135 10.9 Efficiency of MD..................................................................................................................135 10.9.1 Periodic Boundary Conditions and Ewald Sums................................................... 136 10.9.2 A Comment About MD Simulations and Entropy................................................ 136 References....................................................................................................................................... 137 11. Non-Equilibrium Thermodynamics—Onsager Theory........................................................... 139 11.1 Introduction.......................................................................................................................... 139 11.2 The Local-Equilibrium Hypothesis..................................................................................... 139 11.3 Entropy Production Due to Heat Flow in a Closed System................................................. 140 11.4 Entropy Production in an Isolated System...........................................................................141 11.5 Extra Hypothesis: A Linear Relation Between Rates and Affinities...................................142 11.5.1 Entropy of an Ideal Linear Chain Close to Equilibrium........................................143 11.6 Fourier’s Law—A Continuum Example of Linearity.......................................................... 144 11.7 Statistical Mechanics Picture of Irreversibility....................................................................145 11.8 Time Reversal, Microscopic Reversibility, and the Principle of Detailed Balance.............147 11.9 Onsager’s Reciprocal Relations............................................................................................149 11.10 Applications......................................................................................................................... 150 11.11 Steady States and the Principle of Minimum Entropy Production......................................151 11.12 Summary...............................................................................................................................152 References........................................................................................................................................152 12. Non-equilibrium Statistical Mechanics.......................................................................................153 12.1 Fick’s Laws for Diffusion.....................................................................................................153 12.1.1 First Fick’s Law.......................................................................................................153 12.1.2 Calculation of the Flux from Thermodynamic Considerations............................. 154 12.1.3 The Continuity Equation.........................................................................................155 12.1.4 Second Fick’s Law—The Diffusion Equation....................................................... 156 12.1.5 Diffusion of Particles Through a Membrane......................................................... 156 12.1.6 Self-Diffusion......................................................................................................... 156

Contents

xi

12.2 Brownian Motion: Einstein’s Derivation of the Diffusion Equation................................... 158 12.3 Langevin Equation............................................................................................................... 160 12.3.1 The Average Velocity and the Fluctuation-Dissipation Theorem..........................162 12.3.2 Correlation Functions.............................................................................................163 12.3.3 The Displacement of a Langevin Particle.............................................................. 164 12.3.4 The Probability Distributions of the Velocity and the Displacement.................... 166 12.3.5 Langevin Equation with a Charge in an Electric Field...........................................168 12.3.6 Langevin Equation with an External Force—The Strong Damping Velocity........168 12.4 Stochastic Dynamics Simulations........................................................................................169 12.4.1 Generating Numbers from a Gaussian Distribution by CLT..................................170 12.4.2 Stochastic Dynamics versus Molecular Dynamics................................................171 12.5 The Fokker-Planck Equation................................................................................................171 12.6 Smoluchowski Equation.......................................................................................................174 12.7 The Fokker-Planck Equation for a Full Langevin Equation with a Force...........................175 12.8 Summary of Pairs of Equations............................................................................................175 References........................................................................................................................................176 13. The Master Equation.....................................................................................................................177 13.1 Master Equation in a Microcanonical System......................................................................177 13.2 Master Equation in the Canonical Ensemble.......................................................................178 13.3 An Example from Magnetic Resonance.............................................................................. 180 13.3.1 Relaxation Processes Under Various Conditions....................................................181 13.3.2 Steady State and the Rate of Entropy Production.................................................. 184 13.4 The Principle of Minimum Entropy Production—Statistical Mechanics Example............185 References........................................................................................................................................186

Section IV Advanced Simulation Methods: Polymers and Biological Macromolecules 14. Growth Simulation Methods for Polymers..................................................................................189 14.1 Simple Sampling of Ideal Chains.........................................................................................189 14.2 Simple Sampling of SAWs................................................................................................... 190 14.3 The Enrichment Method...................................................................................................... 192 14.4 The Rosenbluth and Rosenbluth Method............................................................................. 193 14.5 The Scanning Method.......................................................................................................... 195 14.5.1 The Complete Scanning Method........................................................................... 195 14.5.2 The Partial Scanning Method................................................................................ 196 14.5.3 Treating SAWs with Finite Interactions................................................................. 197 14.5.4 A Lower Bound for the Entropy............................................................................ 197 14.5.5 A Mean-Field Parameter........................................................................................ 198 14.5.6 Eliminating the Bias by Schmidt’s Procedure....................................................... 199 14.5.7 Correlations in the Accepted Sample.................................................................... 200 14.5.8 Criteria for Efficiency............................................................................................ 201 14.5.9 Locating Transition Temperatures......................................................................... 202 14.5.10 The Scanning Method versus Other Techniques................................................... 203 14.5.11 The Stochastic Double Scanning Method............................................................. 204 14.5.12 Future Scanning by Monte Carlo........................................................................... 204 14.5.13 The Scanning Method for the Ising Model and Bulk Systems.............................. 205 14.6 The Dimerization Method................................................................................................... 206 References....................................................................................................................................... 208

xii

Contents

15. The Pivot Algorithm and Hybrid Techniques.............................................................................211 15.1 The Pivot Algorithm—Historical Notes...............................................................................211 15.2 Ergodicity and Efficiency.....................................................................................................211 15.3 Applicability.........................................................................................................................212 15.4 Hybrid and Grand-Canonical Simulation Methods..............................................................213 15.5 Concluding Remarks.............................................................................................................214 References........................................................................................................................................214 16. Models of Proteins..........................................................................................................................217 16.1 Biological Macromolecules versus Polymers.......................................................................217 16.2 Definition of a Protein Chain................................................................................................217 16.3 The Force Field of a Protein.................................................................................................218 16.4 Implicit Solvation Models.....................................................................................................219 16.5 A Protein in an Explicit Solvent.......................................................................................... 220 16.6 Potential Energy Surface of a Protein................................................................................. 221 16.7 The Problem of Protein Folding.......................................................................................... 222 16.8 Methods for a Conformational Search................................................................................. 222 16.8.1 Local Minimization—The Steepest Descents Method......................................... 223 16.8.2 Monte Carlo Minimization.................................................................................... 224 16.8.3 Simulated Annealing............................................................................................. 225 16.9 Monte Carlo and Molecular Dynamics Applied to Proteins............................................... 225 16.10 Microstates and Intermediate Flexibility............................................................................ 226 16.10.1 On the Practical Definition of a Microstate........................................................... 227 References....................................................................................................................................... 227 17. Calculation of the Entropy and the Free Energy by Thermodynamic Integration.................231 17.1 “Calorimetric” Thermodynamic Integration....................................................................... 232 17.2 The Free Energy Perturbation Formula............................................................................... 232 17.3 The Thermodynamic Integration Formula of Kirkwood.................................................... 234 17.4 Applications......................................................................................................................... 235 17.4.1 Absolute Entropy of a SAW Integrated from an Ideal Chain Reference State...... 235 17.4.2 Harmonic Reference State of a Peptide................................................................. 237 17.5 Thermodynamic Cycles....................................................................................................... 237 17.5.1 Other Cycles........................................................................................................... 240 17.5.2 Problems of TI and FEP Applied to Proteins........................................................ 240 References....................................................................................................................................... 241 18. Direct Calculation of the Absolute Entropy and Free Energy................................................. 243 18.1 Absolute Free Energy from ....................................................................... 243 18.2 The Harmonic Approximation............................................................................................ 244 18.3 The M2 Method................................................................................................................... 245 18.4 The Quasi-Harmonic Approximation.................................................................................. 246 18.5 The Mutual Information Expansion.................................................................................... 247 18.6 The Nearest Neighbor Technique........................................................................................ 248 18.7 The MIE-NN Method.......................................................................................................... 249 18.8 Hybrid Approaches.............................................................................................................. 249 References....................................................................................................................................... 249 19. Calculation of the Absolute Entropy from a Single Monte Carlo Sample...............................251 19.1 The Hypothetical Scanning (HS) Method for SAWs............................................................251 19.1.1 An Exact HS Method..............................................................................................251 19.1.2 Approximate HS Method....................................................................................... 252

Contents

xiii

19.2 The HS Monte Carlo (HSMC) Method............................................................................... 253 19.3 Upper Bounds and Exact Functionals for the Free Energy................................................. 255 19.3.1 The Upper Bound F B............................................................................................. 255 19.3.2 FB Calculated by the Reversed Schmidt Procedure.............................................. 256 19.3.3 A Gaussian Estimation of FB................................................................................. 257 19.3.4 Exact Expression for the Free Energy................................................................... 258 19.3.5 The Correlation Between σA and FA...................................................................... 258 19.3.6 Entropy Results for SAWs on a Square Lattice..................................................... 259 19.4 HS and HSMC Applied to the Ising Model......................................................................... 260 19.5 The HS and HSMC Methods for a Continuum Fluid...........................................................261 19.5.1 The HS Method.......................................................................................................261 19.5.2 The HSMC Method................................................................................................ 262 19.5.3 Results for Argon and Water.................................................................................. 264 19.5.3.1 Results for Argon................................................................................... 264 19.5.3.2 Results for Water................................................................................... 266 19.6 HSMD Applied to a Peptide................................................................................................ 266 19.6.1 Applications........................................................................................................... 269 19.7 The HSMD-TI Method........................................................................................................ 269 19.8 The LS Method.................................................................................................................... 270 19.8.1 The LS Method Applied to the Ising Model.......................................................... 270 19.8.2 The LS Method Applied to a Peptide.................................................................... 272 References........................................................................................................................................274 20. The Potential of Mean Force, Umbrella Sampling, and Related Techniques......................... 277 20.1 Umbrella Sampling.............................................................................................................. 277 20.2 Bennett’s Acceptance Ratio................................................................................................. 278 20.3 The Potential of Mean Force............................................................................................... 281 20.3.1 Applications........................................................................................................... 284 20.4 The Self-Consistent Histogram Method.............................................................................. 285 20.4.1 Free Energy from a Single Simulation.................................................................. 286 20.4.2 Multiple Simulations and The Self-Consistent Procedure.................................... 286 20.5 The Weighted Histogram Analysis Method........................................................................ 289 20.5.1 The Single Histogram Equations........................................................................... 290 20.5.2 The WHAM Equations...........................................................................................291 20.5.3 Enhancements of WHAM..................................................................................... 293 20.5.4 The Basic MBAR Equation................................................................................... 295 20.5.5 ST-WHAM and UIM............................................................................................. 296 20.5.6 Summary................................................................................................................ 296 References....................................................................................................................................... 297 21. Advanced Simulation Methods and Free Energy Techniques.................................................. 301 21.1 Replica-Exchange................................................................................................................ 301 21.1.1 Temperature-Based REM...................................................................................... 301 21.1.2 Hamiltonian-Dependent Replica Exchange........................................................... 305 21.2 The Multicanonical Method................................................................................................ 308 21.2.1 Applications............................................................................................................311 21.2.2 MUCA-Summary...................................................................................................312 21.3 The Method of Wang and Landau........................................................................................312 21.3.1 The Wang and Landau Method-Applications.........................................................314 21.4 The Method of Expanded Ensembles...................................................................................315 21.4.1 The Method of Expanded Ensembles-Applications...............................................317 21.5 The Adaptive Integration Method........................................................................................317

xiv

Contents 21.6 Methods Based on Jarzynski’s Identity................................................................................319 21.6.1 Jarzynski’s Identity versus Other Methods for Calculating ΔF............................ 323 21.7 Summary.............................................................................................................................. 324 References....................................................................................................................................... 324

22. Simulation of the Chemical Potential...........................................................................................331 22.1 The Widom Insertion Method..............................................................................................331 22.2 The Deletion Procedure........................................................................................................332 22.3 Personage’s Method for Treating Deletion.......................................................................... 334 22.4 Introduction of a Hard Sphere............................................................................................. 336 22.5 The Ideal Gas Gauge Method.............................................................................................. 337 22.6 Calculation of the Chemical Potential of a Polymer by the Scanning Method................... 338 22.7 The Incremental Chemical Potential Method for Polymers................................................ 340 22.8 Calculation of μ by Thermodynamic Integration.................................................................341 References........................................................................................................................................341 23. The Absolute Free Energy of Binding......................................................................................... 343 23.1 The Law of Mass Action...................................................................................................... 343 23.2 Chemical Potential, Fugacity, and Activity of an Ideal Gas............................................... 344 23.2.1 Thermodynamics................................................................................................... 344 23.2.2 Canonical Ensemble.............................................................................................. 344 23.2.3 NpT Ensemble........................................................................................................ 345 23.3 Chemical Potential in Ideal Solutions: Raoult’s and Henry’s Laws.................................... 345 23.3.1 Raoult’s Law.......................................................................................................... 346 23.3.2 Henry’s Law........................................................................................................... 346 23.4 Chemical Potential in Non-ideal Solutions.......................................................................... 346 23.4.1 Solvent.................................................................................................................... 346 23.4.2 Solute...................................................................................................................... 347 23.5 Thermodynamic Treatment of Chemical Equilibrium........................................................ 347 23.6 Chemical Equilibrium in Ideal Gas Mixtures: Statistical Mechanics................................. 348 23.7 Pressure-Dependent Equilibrium Constant of Ideal Gas Mixtures..................................... 349 23.8 Protein-Ligand Binding....................................................................................................... 350 23.8.1 Standard Methods for Calculating ΔA0..................................................................352 23.8.2 Calculating ΔA0 by HSMD-TI............................................................................... 354 23.8.3 HSMD-TI Applied to the FKBP12-FK506 Complex: Equilibration.................... 356 23.8.4 The Internal and External Entropies..................................................................... 357 23.8.5 TI Results for FKBP12-FK506.............................................................................. 359 23.8.6 ΔA0 Results for FKBP12-FK506........................................................................... 359 23.9 Summary.............................................................................................................................. 362 References....................................................................................................................................... 362 Appendix................................................................................................................................................ 367 Index....................................................................................................................................................... 369

Preface Statistical mechanics is a well established basic discipline encompassing all the exact sciences; its theoretical foundations, laid down at the end of the nineteenth century, have been summarized in the early classical monographs of Tolman, as well as Landau & Lifshitz. In many later books, the scope of the theory has been extended to new systems and phenomena. Statistical mechanics received a big boost in the 1950s of the twentieth century with the advent of Monte Carlo (MC) and molecular dynamics (MD) simulation techniques. Thus, computer simulation, not only has become an organic part of statistical mechanics, but it is currently the main engine of development in this field, reflected by a vast literature including the early books of Binder, Allan & Tildesley, Frenkel & Smith, and more recent ones. The constant progress in this field requires timely reviews of new material, which is one aim of this book. The present book is an extension of a course given first to graduate students in the biophysics program at Bar Ilan University, Israel, and later to students in the graduate program of the Department of Computational and Systems Biology at the University of Pittsburgh School of Medicine. The book, which is structured as a course, consists of four parts: (1) probability theory, (2) equilibrium thermodynamics and statistical mechanics, (3) non-equilibrium problems, and (4) computer simulation of polymers and biological macromolecular systems. The 23 chapters of the book stand to a large extent by themselves. The book requires entry level mathematics—derivatives and integrals—and thus can benefit graduate and undergraduate students, as well as researchers from all the exact sciences (in particular, chemistry, chemical engineering, biophysics, and structural biology), who seek to get acquainted with statistical mechanics theory and computer simulation methodologies for treating polymers and biological macromolecules. We elaborate on the (somewhat vague) notion of entropy, pointing out the difficulties encountered in its computation and in the computation of the related properties—free energy, chemical potential, and the potential of mean force. Sophisticated methodologies (including ours) for the calculation of these properties are presented and classified with respect to performance and functionality. While the material covered in the book can be learned from the contents, we highlight below, in some detail, unique topics and focuses of the book, such as teaching aspects of probability theory, the nontraditional derivation of statistical mechanics, or our methods for calculation the entropy. In Chapter 1, which contains standard probability material, we elaborate (with examples) on the related notions of experimental probability, probability space, and the experimental probability on a computer, advocating probability space to be used as a framework for solving probability problems in a systematic way. The distinction between these three phases, which is sometimes confusing to students, is essential for devising simulation methods, in particular, for the entropy. We devote longer discussions than usual to the concept of product space, which constitutes the theoretical foundation for simulation and estimation theories; we also discuss the related “central limit theorem” (CLT)—a highly used tool throughout the book. Chapter 2 provides a short review of classical thermodynamics. An emphasis is given to the notion of entropy in equilibrium and non-equilibrium conditions, illustrated by solvable problems for an ideal gas, where its entropy, S, is shown to include probabilistic elements, S~-lnP, already on the thermodynamics level. This leads to our non-traditional derivation of statistical mechanics (Chapter 3), where thermodynamic parameters (e.g., energy) become statistical averages based on initially unknown probability density, P, and an entropy function, Sʹ. Based on lnP above, and experimental properties (e.g., extensivity), an Sʹ function is defined, becoming part of a free energy functional A(P), which is minimized with respect to P, leading to the Boltzmann probability density. In Chapter 4, we derive the statistical mechanics equations of the ideal gas and the harmonic oscillator on three levels: macroscopic, microscopic, and quantum mechanics. Elaborating on the differences among these models deepens the understanding of the probabilistic nature of statistical mechanics. Fluctuations (standard deviations), σ, are treated in Chapters  5 and 6. For  the exact free energy, A, σ(A) = 0—a completely unrecognized result, which can lead to the correct A by extrapolating to σ(A) = 0 xv

xvi

Preface

approximate results of A versus σ(A). The behavior of a large system, σ(E)/E → 0, provides a way for solving problems in statistical mechanics based on “the most probable energy term,” in addition to using thermodynamic derivatives and statistical averages. Likewise, different ensembles lead to the same averages (but not fluctuations). This versatility of alternative treatments (which is not always emphasized) is demonstrated by carrying out multisolutions of specific problems, in particular, for calculating the entropic forces required to stretch a one-dimensional ideal chain (Chapter 8). First- and second-order phase transitions, the Ising model, and the corresponding critical exponents are shortly discussed in Chapter 7, and self-avoiding walks (SAWs) with and without attractions are reviewed in Chapter 9, and the relation of their properties to the behavior of real macromolecules is discussed. Chapters  11–13 cover several topics in non-equilibrium thermodynamics and statistical mechanics close to equilibrium. However, being relaxation-type methods, MC/MD are presented first, and their efficiencies are discussed (Chapter 10) as a precursor to the relaxation phenomena studied in the subsequent chapters. In  Chapter  11, Onsager relations, microscopic reversibility, the notion of a steady state, and the principle of minimum entropy production are described in detail with solved problems. Chapter 12 presents complete derivations of the two Fick’s laws, the diffusion equation (and Einstein’s derivation for Brownian motion), Langevin and Fokker-Planck equations, and the stochastic dynamics simulation. Chapter 13 is devoted to the master equation, with a specific example from nuclear magnetic resonance (NMR). Finally, a statistical mechanics example of the principle of minimum entropy production is presented. The effects of the close to equilibrium condition and the local equilibrium hypothesis are carefully examined. Chapter 14 deals with methods that generate a SAW step-by-step (from nothing) with the help of transition probabilities (TPs); thus, unlike the case of MC, the construction probability, P = ΠkTPk, is known and S is known as well. “Simple sampling” is an exact, but inefficient method (due to “sample attrition”) since a direction, ν, is chosen “democratically” (pν = 1/2d) (N  0

P(A) > 0

(1.18)

(1.19)

therefore, P(A B) =



P(B A) P(A) P(B)

(1.20)

Examples for a die: Defining A = (2) and B = even = {2,4,6} → P(A∩ B) = P(2) = 1/6; using Equations (1.18) and (1.19) lead to:

P(A= B)

1/ 6 = 1/ 3 1/ 2

P(B = A)

1/ 6 = 1 1/ 6

Using Equation (1.20), one obtains as above, P(A/B) = 1/3 since:

P(A B) =

1× 1 / 6 = 1 / 3 1/ 2

An equivalent condition for independency: Event A is independent of B if:

P(A ∩ B) = P(A) P(B).

(1.21)

This stems from Equation (1.18), which lead to,

P(A= B) P(B) P= (A ∩ B) P(B) P(A)

or P(A/B) = P(A). Thus, in the example above, event A depends on B since: P(A∩ B) = 1/6 ≠ P(A)P(B) = 1/6 × 1/2 = 1/12. However, A = (1,2,3) is independent of B = (3,4) since P(A∩ B) = 1/6 is equal to P(A)P(B) = 1/2 × 1/3 = 1/6.

13

Probability and Its Applications

Based on the Bayes formula [Equation (1.18)], one can show that if an event A must result in mutually exclusive events, A1,…, An, that is, A = A∩A1 + A∩ A2 +… A∩An, then: P(A) = P(A1 ) P(A A1 ) +  + P(A n ) P(A A n )



(1.22)

Example Two cards are drawn successively from a deck. What is the probability that both are red (r)? Solution The elementary events are defined in the product space; they are: (r, r), (r, no), (no, r), and (no, no). The events of interest are: A ≡ (“first card is red,” which includes {(r, r); (r, no)}, and B ≡ (“second card is red,” which includes {(no, r); (r, r)}, thus:



P ( A ∩ B) = P ( r,r ) = P ( A ) P ( B A ) =

1  25   2  51 

1.8  Discrete Probability—Summary Thus far, we have dealt with discrete probability theory. We started by defining the experimental probability as a limit of relative frequency, and then described a model for the experimental reality called elementary probability space, where the probability P(EE) of any EE is known exactly. This space enables one addressing in a systematic way complicated problems without the need to carry out experiments. In solving such problems, it is crucial to make the distinction between the experimental world, the corresponding (modeled) probability space, and when simulations are performed, the experimental world on a computer. We have defined the notion of permutations, combinations, and conditional probability and demonstrated how related problems can be solved systematically within the framework of the probability space. A special emphasis has been given to product probability spaces, which constitute the basis of sampling theory. Next, we advance to the more complex theory of a continuous probability space, starting with a discussion on random variables.

1.9  One-Dimensional Discrete Random Variables Assume a probability space consisting of a set of elementary events, {ω}  =  Ω and the corresponding events. A random variable is a numerical function X = X(ω) from the elementary events, ω to the real line, −∞