Beginning Ubuntu LTS Server Administration: From Novice to Professional [2 ed.] 9781430210825, 9781430210818, 1430210826

Beginning Ubuntu LTS Server Administration, Second Editionis the touchstone companion book for anyone implementing Ubunt

903 152 6MB

English Pages 403 [412] Year 2008

Report DMCA / Copyright

DOWNLOAD FILE

Polecaj historie

Beginning Ubuntu LTS Server Administration: From Novice to Professional [2 ed.]
 9781430210825, 9781430210818, 1430210826

Table of contents :
Title Page......Page 2
Copyright Page......Page 3
Contents at a Glance......Page 5
Table of Contents......Page 6
About the Author......Page 14
About the Technical Reviewers......Page 15
How This Book Is Structured......Page 16
Contacting the Author......Page 17
Preparing for the Installation......Page 18
Starting the Ubuntu Server Installation Process......Page 19
Configuring the Server’s Hard Drive......Page 25
Completing the Installation......Page 43
Summary......Page 44
Working As root?......Page 45
Using Bash to Best Effect......Page 46
Managing Bash with Key Sequences......Page 49
Working with Directories......Page 50
Working with Files......Page 51
Viewing the Content of Text Files......Page 53
Finding Files That Contain Specific Text......Page 55
Piping......Page 57
Redirection......Page 58
Finding Files......Page 60
Working with an Editor......Page 61
Saving and Quitting......Page 62
Getting Help......Page 63
Using man to Get Help......Page 64
Getting Information on Installed Packages......Page 66
Summary......Page 67
Software Management......Page 68
Software Repositories and Package Databases......Page 69
Package Management Utilities......Page 70
Understanding apt......Page 71
Installing Software from Tarballs......Page 78
Configuring a Graphical User Interface......Page 79
Making File Backups with tar......Page 84
Making Device Backups Using dd......Page 87
Configuring Logging......Page 88
Configuring syslog......Page 89
Logging in Other Ways......Page 93
Rotating Log Files......Page 94
Summary......Page 96
Using the mount Command......Page 97
Unmounting Devices......Page 102
Automating Mounts with /etc/fstab......Page 103
Checking File System Integrity......Page 106
Working with Symbolic Links......Page 107
Comparing File Systems......Page 110
Creating File Systems......Page 119
Working with Logical Volumes......Page 122
Doing Magic on Your File Systems with dd......Page 127
Summary......Page 130
Commands for User Management......Page 131
Managing Passwords......Page 134
Behind the Commands: Configuration Files......Page 136
Creating Groups......Page 140
Behind the Commands: /etc/group......Page 141
Managing the User’s Shell Environment......Page 142
Read,Write, and Execute: The Three Basic Linux Permissions......Page 143
Permissions and the Concept of Ownership......Page 144
Working with Advanced Linux Permissions......Page 146
Setting Permissions......Page 148
Working with Access Control Lists......Page 150
Preparing the File System for ACLs......Page 151
Applying File Attributes......Page 154
Preparing the File System for Quota......Page 156
Setting Quota for Users and Groups......Page 157
Understanding Pluggable Authentication Modules......Page 158
Creating a Default Security Policy......Page 160
Discovering PAM Modules......Page 161
Configuring Administrator Tasks with sudo......Page 164
Summary......Page 166
Different Kinds of Processes......Page 167
Foreground and Background......Page 168
Managing Processes......Page 170
Other Tools to Monitor System Activity......Page 174
Executing Processes Automatically......Page 177
Configuring cron......Page 178
Tuning the Boot Procedure......Page 180
The GRUB Configuration File......Page 181
Installing GRUB......Page 184
Working with the GRUB Boot Menu......Page 185
Upstart......Page 186
Runlevels......Page 188
Making Service Management Easier......Page 189
Managing Hardware......Page 190
Kernel Management......Page 191
Installing Your Own Custom Kernel......Page 197
Hardware Management with udev......Page 200
Summary......Page 204
To Script or Not to Script?......Page 205
What Shell?......Page 206
Basic Elements of a Shell Script......Page 207
Making It Executable......Page 208
Making a Script Interactive......Page 210
Working with Arguments......Page 211
Command Substitution......Page 214
Substitution Operators......Page 215
Pattern-Matching Operators......Page 217
Performing Calculations in Scripts......Page 219
Using Flow Control......Page 223
Using if...then...else......Page 224
Case......Page 227
Using while......Page 228
Using until......Page 229
Using for......Page 230
Using a Stream Editor......Page 231
Working with Functions......Page 232
A Complex Scripting Example......Page 233
Summary......Page 235
Configuring the Network Card......Page 236
Using ifconfig......Page 238
Using the ip Tool......Page 241
Managing IPv6......Page 244
Managing Routes......Page 247
Configuring the DNS Resolver......Page 248
Configuring Network Card Properties with the ethtool Command......Page 250
Testing Connectivity......Page 253
Testing Routability......Page 255
Testing Availability of Services......Page 256
Monitoring the Network Interface......Page 260
Monitoring Network Traffic......Page 263
Connecting Remotely with SSH......Page 265
Working with Public/Private Key Pairs......Page 266
Working with Secure Shell......Page 267
Configuring SSH......Page 268
A Short Introduction to Cryptography......Page 270
Using Public/Private Key–Based Authentication in an SSH Environment......Page 271
Setting Up SSH for Key-Based Authentication......Page 272
Caching Keys with ssh-agent......Page 273
X Forwarding......Page 274
Generic TCP Port Forwarding......Page 275
Summary......Page 276
Methods of Name Resolution......Page 277
Structure of the DNS Hierarchy......Page 279
Configuring DNS......Page 283
Configuring Reversed Lookup......Page 289
Testing Your Name Server......Page 290
Understanding the DHCP Protocol......Page 291
The /etc/dhcp/dhcpd.conf Configuration File......Page 292
Advanced DHCP Configuration Options......Page 295
Configuring NTP......Page 298
How NTP Works......Page 299
Configuring a Stand-Alone NTP Time Server......Page 300
Configuring an NTP Client......Page 301
Checking NTP Synchronization Status......Page 302
Customizing Your NTP Server......Page 303
Starting Services with xinetd......Page 304
Setting up xinetd by Hand......Page 305
Tuning Access to Services with TCP Wrapper......Page 306
Summary......Page 309
Adding Printers......Page 310
Sharing Printers......Page 313
Managing Printers......Page 314
Accessing CUPS Printers......Page 315
Sharing Files with NFS......Page 316
Understanding How the NFS Works......Page 317
Configuring an NFS Server......Page 319
Configuring an NFS Client......Page 321
Sharing Files with Samba......Page 322
Configuring the Samba Server......Page 323
Integrating CUPS with Samba......Page 328
Setting Up Samba As a Domain Controller......Page 330
Client Access to the Samba Server......Page 332
Summary......Page 334
Setting Up Apache......Page 335
Starting, Stopping, and Testing the Apache Web Server......Page 336
Exploring the Configuration Files......Page 338
The Structure of the Apache Configuration Files......Page 339
Checking the Configuration......Page 340
Configuring Virtual Hosts......Page 342
Managing Access to the Web Server......Page 343
Configuring Host-Based Access Restrictions......Page 344
Configuring User-Based Access Restrictions......Page 345
Enabling HTTPS......Page 346
Creating a Self-Signed Certificate......Page 347
Configuring Apache to Use the Self-Signed Certificate......Page 348
Some Words on Apache Performance Tuning......Page 349
Using PHP......Page 350
Setting Up MySQL......Page 351
Configuring a Squid Proxy Server......Page 352
Configuring Squid Access Control Policies......Page 353
Configuring User Authentication......Page 355
Configuring the pure-ftpd Server......Page 357
Summary......Page 359
CHAPTER 12 Setting Up the Netfilter Firewall with iptables and ufw......Page 360
Netfilter Building Blocks......Page 361
Using iptables to Create a Firewall......Page 362
Firewall Management Made Easy:Uncomplicated Firewall......Page 369
Summary......Page 370
Virtualization Solutions......Page 372
Approaches to Virtualization......Page 374
Installing Virtual Machines with KVM......Page 375
Setting Up KVM on Ubuntu Server......Page 376
Installing Windows As a Guest Operating System on KVM......Page 377
Managing Virtual Machines with Virtual Manager......Page 379
Managing Virtual Machines with libvirt Tools......Page 383
Installing Virtual Machines Using Xen......Page 385
Setting Up Xen on Ubuntu Server......Page 386
Installing Windows as a Guest Operating System on Xen......Page 388
Installing Ubuntu Server As a Guest Operating System on Xen......Page 390
Using Xen Management Commands......Page 391
Ubuntu JeOS......Page 392
Summary......Page 393
Index......Page 394

Citation preview

Beginning Ubuntu LTS Server Administration From Novice to Professional, Second Edition

Sander van Vugt

Beginning Ubuntu LTS Server Administration: From Novice to Professional, Second Edition Copyright © 2008 by Sander van Vugt All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-1082-5 ISBN-13 (electronic): 978-1-4302-1081-8 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Java™ and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc., in the US and other countries. Apress, Inc., is not affiliated with Sun Microsystems, Inc., and this book was written without endorsement from Sun Microsystems, Inc. Lead Editor: Frank Pohlmann Technical Reviewers: Tim Hall, Samuel Cuella Editorial Board: Clay Andres, Steve Anglin, Ewan Buckingham, Tony Campbell, Gary Cornell, Jonathan Gennick, Matthew Moodie, Joseph Ottinger, Jeffrey Pepper, Frank Pohlmann, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Project Manager: Beth Christmas Copy Editor: Nancy Sixsmith Associate Production Director: Kari Brooks-Copony Production Editor: Liz Berry Compositor: Dina Quan Proofreader: Liz Welch Indexer: Odessa&Cie Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax 201-348-4505, e-mail [email protected], or visit http://www.springeronline.com. For information on translations, please contact Apress directly at 2855 Telegraph Avenue, Suite 600, Berkeley, CA 94705. Phone 510-549-5930, fax 510-549-5939, e-mail [email protected], or visit http://www.apress.com. Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at http://www.apress.com/info/bulksales. The information in this book is distributed on an “as is” basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at http://www.apress.com.

This book is dedicated to Florence.

Contents at a Glance About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv About the Technical Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER ■CHAPTER

1 2 3 4 5 6 7 8 9 10 11 12

Installing Ubuntu Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Getting the Most from the Command Line . . . . . . . . . . . . . . . . . . . . . . 29 Performing Essential System Administration Tasks . . . . . . . . . . . . . 53 Performing File System Management Tasks . . . . . . . . . . . . . . . . . . . . 83 Configuring Your Server for Security . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Setting the System to Your Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Running It Any Way You Like . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Making a Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Configuring Network Infrastructure Services . . . . . . . . . . . . . . . . . . 265 Using Ubuntu Server As a File and Print Server . . . . . . . . . . . . . . . . 299 Setting Up Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Setting Up the Netfilter Firewall with iptables and ufw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 ■CHAPTER 13 Multiplying Your Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

■INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385

v

Contents About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv About the Technical Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

■CHAPTER 1

Installing Ubuntu Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Preparing for the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Starting the Ubuntu Server Installation Process . . . . . . . . . . . . . . . . . . . . . . . 2 Configuring the Server’s Hard Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Completing the Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

■CHAPTER 2

Getting the Most from the Command Line

. . . . . . . . . . . . . . . . 29

Working As root? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Working with the Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Using Bash to Best Effect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Managing Bash with Key Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Performing Basic File System Management Tasks . . . . . . . . . . . . . . . . . . . 34 Working with Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Working with Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Viewing the Content of Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Finding Files That Contain Specific Text . . . . . . . . . . . . . . . . . . . . . . . 39 Creating Empty Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Piping and Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Piping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Finding Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Working with an Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Vi Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Saving and Quitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Cut, Copy, and Paste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Deleting Text. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

vii

viii

■CONTENTS

Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Using man to Get Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Using the --help Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Getting Information on Installed Packages . . . . . . . . . . . . . . . . . . . . . 50 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

■CHAPTER 3

Performing Essential System Administration Tasks. . . . . . 53 Software Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Software Repositories and Package Databases . . . . . . . . . . . . . . . . . 54 Package Management Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Understanding apt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Installing Software from Tarballs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Configuring a Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . . . 64 Creating Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Making File Backups with tar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Making Device Backups Using dd. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Configuring Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Configuring syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Logging in Other Ways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Rotating Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

■CHAPTER 4

Performing File System Management Tasks . . . . . . . . . . . . . . 83 Mounting Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Using the mount Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Unmounting Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Automating Mounts with /etc/fstab . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Checking File System Integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Working with Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Why Use Links? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Working with Symbolic Links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Working with Hard Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Configuring Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Comparing File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Creating File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Working with Logical Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Doing Magic on Your File Systems with dd. . . . . . . . . . . . . . . . . . . . . . . . . 113 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

■CONTENTS

■CHAPTER 5

Configuring Your Server for Security . . . . . . . . . . . . . . . . . . . . . 117 Setting Up User Accounts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Commands for User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Managing Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Modifying and Deleting User Accounts . . . . . . . . . . . . . . . . . . . . . . . 122 Behind the Commands: Configuration Files . . . . . . . . . . . . . . . . . . . 122 Creating Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Commands for Group Management . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Behind the Commands: /etc/group . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Using Group Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Managing the User’s Shell Environment . . . . . . . . . . . . . . . . . . . . . . 128 Configuring Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Read, Write, and Execute: The Three Basic Linux Permissions . . . 129 Permissions and the Concept of Ownership . . . . . . . . . . . . . . . . . . . 130 Working with Advanced Linux Permissions . . . . . . . . . . . . . . . . . . . . . . . . 132 Setting Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Using umask to Set Default Permissions for New Files . . . . . . . . . 136 Working with Access Control Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Preparing the File System for ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . 137 ACL Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Applying File Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Apply Quota to Allow a Maximum Amount of Files . . . . . . . . . . . . . . . . . . 142 Installing the Quota Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Preparing the File System for Quota . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Initializing Quota. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Setting Quota for Users and Groups . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Understanding Pluggable Authentication Modules . . . . . . . . . . . . . . . . . . 144 Creating a Default Security Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 Discovering PAM Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Configuring Administrator Tasks with sudo . . . . . . . . . . . . . . . . . . . . . . . . 150 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

■CHAPTER 6

Setting the System to Your Hand . . . . . . . . . . . . . . . . . . . . . . . . . 153 Process Monitoring and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Different Kinds of Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Foreground and Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Managing Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Other Tools to Monitor System Activity . . . . . . . . . . . . . . . . . . . . . . . 160 Setting Process Priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

ix

x

■CONTENTS

Executing Processes Automatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Configuring cron. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 Executing Once with at . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Tuning the Boot Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Managing the GRUB Boot Loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 The GRUB Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Installing GRUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 Working with the GRUB Boot Menu . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Upstart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 Runlevels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Making Service Management Easier . . . . . . . . . . . . . . . . . . . . . . . . . 175 Managing Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 Kernel Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Installing Your Own Custom Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Hardware Management with udev . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

■CHAPTER 7

Running It Any Way You Like

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Before You Even Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 To Script or Not to Script? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 What Shell? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 Basic Elements of a Shell Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Making It Executable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Making a Script Interactive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 Working with Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Working with Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Command Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Changing Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Substitution Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Pattern-Matching Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Performing Calculations in Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 Using Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Using if...then...else . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Using while . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Using until . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Using for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

■CONTENTS

Using a Stream Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Working with Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 A Complex Scripting Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

■CHAPTER 8

Making a Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Configuring the Network Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Using ifup, ifdown, and Related Tools . . . . . . . . . . . . . . . . . . . . . . . . 225 Using ifconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Using the ip Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Managing IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Managing Routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Configuring the DNS Resolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Configuring Network Card Properties with the ethtool Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Troubleshooting Network Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Testing Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Testing Routability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Testing Availability of Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Monitoring the Network Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Monitoring Network Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Connecting Remotely with SSH. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Working with Public/Private Key Pairs . . . . . . . . . . . . . . . . . . . . . . . . 253 Working with Secure Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 Configuring SSH. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Using Key-Based Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 A Short Introduction to Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . 257 Using Public/Private Key–Based Authentication in an SSH Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Setting Up SSH for Key-Based Authentication . . . . . . . . . . . . . . . . . 259 Caching Keys with ssh-agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Tunneling Traffic with SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 X Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Generic TCP Port Forwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

xi

xii

■CONTENTS

■CHAPTER 9

Configuring Network Infrastructure Services . . . . . . . . . . . . 265 Configuring DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Methods of Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Structure of the DNS Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Introducing Forward and Reverse DNS . . . . . . . . . . . . . . . . . . . . . . . 271 Configuring DNS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Configuring Reversed Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Testing Your Name Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 Configuring DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Understanding the DHCP Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Creating the DHCP Server Configuration . . . . . . . . . . . . . . . . . . . . . . 280 The DHCP Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 The /etc/dhcp/dhcpd.conf Configuration File . . . . . . . . . . . . . . . . . . 280 Advanced DHCP Configuration Options . . . . . . . . . . . . . . . . . . . . . . . 283 The DHCP Relay Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Configuring NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 How NTP Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Configuring a Stand-Alone NTP Time Server . . . . . . . . . . . . . . . . . . 288 Pulling or Pushing the Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Configuring an NTP Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Checking NTP Synchronization Status . . . . . . . . . . . . . . . . . . . . . . . . 290 Customizing Your NTP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Applying NTP Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Starting Services with xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Setting up xinetd by Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Tuning Access to Services with TCP Wrapper . . . . . . . . . . . . . . . . . 294 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

■CHAPTER 10 Using Ubuntu Server As a File and Print Server. . . . . . . . . . 299 Setting Up a CUPS Print Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Adding Printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Sharing Printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 Managing Printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Accessing CUPS Printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Sharing Files with NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Using the NFS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Understanding How the NFS Works . . . . . . . . . . . . . . . . . . . . . . . . . . 306 Configuring an NFS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308

■CONTENTS

Configuring an NFS Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 Monitoring the NFS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Sharing Files with Samba . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Samba Server Possibilities and Impossibilities . . . . . . . . . . . . . . . . . 312 Configuring the Samba Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 Integrating CUPS with Samba . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Setting Up Samba As a Domain Controller . . . . . . . . . . . . . . . . . . . . 319 Client Access to the Samba Server. . . . . . . . . . . . . . . . . . . . . . . . . . . 321 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

■CHAPTER 11 Setting Up Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Setting Up Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Apache Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 Starting, Stopping, and Testing the Apache Web Server . . . . . . . . 326 Exploring the Configuration Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 The Structure of the Apache Configuration Files . . . . . . . . . . . . . . . 329 Checking the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Working with Virtual Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 Configuring Virtual Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 Managing Access to the Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 Configuring Host-Based Access Restrictions . . . . . . . . . . . . . . . . . . 334 Configuring User-Based Access Restrictions . . . . . . . . . . . . . . . . . . 335 Enabling HTTPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Creating a Self-Signed Certificate. . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Configuring Apache to Use the Self-Signed Certificate . . . . . . . . . . 338 Some Words on Apache Performance Tuning . . . . . . . . . . . . . . . . . . . . . . 339 Using PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Setting Up MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Setting the MySQL Root Password . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Creating a MySQL Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Configuring a Squid Proxy Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Installing a Squid Proxy Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 Configuring Squid Access Control Policies . . . . . . . . . . . . . . . . . . . . 343 Configuring User Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 Setting Up FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Configuring the pure-ftpd Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

xiii

xiv

■CONTENTS

■CHAPTER 12 Setting Up the Netfilter Firewall with

iptables and ufw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Netfilter Building Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352 Using iptables to Create a Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Firewall Management Made Easy: Uncomplicated Firewall . . . . . . 360 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361

■CHAPTER 13 Multiplying Your Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Understanding Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Virtualization Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Approaches to Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 Installing Virtual Machines with KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Preparing Your Server for KVM Virtualization: Networking . . . . . . . 367 Setting Up KVM on Ubuntu Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 Installing Windows As a Guest Operating System on KVM . . . . . . . 368 Installing Ubuntu Server As a Guest Operating System on KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 Managing Virtual Machines with Virtual Manager . . . . . . . . . . . . . . 370 Managing Virtual Machines with libvirt Tools . . . . . . . . . . . . . . . . . . 374 Installing Virtual Machines Using Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 Setting Up Xen on Ubuntu Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377 Installing Windows as a Guest Operating System on Xen. . . . . . . . 379 Installing Ubuntu Server As a Guest Operating System on Xen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 Using Xen Management Commands . . . . . . . . . . . . . . . . . . . . . . . . . 382 Ubuntu Server in a VMware Environment . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Ubuntu JeOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384

■INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385

About the Author ■SANDER VAN VUGT is an independent trainer and consultant who lives in the Netherlands and works in the extended EMEA (Europe, Middle East, and Africa) area. He specializes in Linux high availability, storage solutions, and performance problems; and has successfully implemented Linux clusters across the globe. Sander has written several books about Linux-related subjects, including The Definitive Guide to SUSE Linux Enterprise Server (Apress, 2006). Sander’s articles can be found on several international web sites and in magazines such as SearchEnterpriseLinux.com, Linux Journal, and Linux Magazine. He works as a volunteer for the Linux Professional Institute (LPI), contributing topics for different certification levels. Most important, Sander is the father of Alex and Franck, and is the loving husband of Florence. For more information, consult Sander’s web site: www.sandervanvugt.com. Sander can be reached by email at [email protected].

xv

About the Technical Reviewers ■SAMUEL CUELLA was born in 1985. He is currently an IT student and also works as a Linux/Solaris trainer. Samuel taught the complete Mandriva certification program in China (JUST University) and also teaches Linux for LPI certification training. ■TIM HALL currently provides front-line support for 64 Studio. He has also written newbie tutorials for Linux User & Developer magazine in between more mundane system admin and web authoring jobs. Tim has released albums and performed as a musician and songwriter, both solo and in collaboration with other artists. He has been further honored as the holder of the Bardic chair of Glastonbury between 2005 and 2007.

xvii

Introduction B

eginning Ubuntu LTS Server Administration: From Novice to Professional, Second Edition provides a complete introduction to Ubuntu Server. I wrote it for people who are new to Ubuntu Server administration but have a solid foundation in IT. The target readers are Windows administrators as well as people who are used to managing other flavors of Linux (or UNIX). It was the goal of this book to give a no-nonsense introduction to working with Ubuntu Server, so it provides all the basics that are needed to get you going. It also includes many useful tips that help you do your work in a more efficient manner. Many books about Ubuntu are presently available, but you can’t do Ubuntu Server justice by covering both the desktop and the server version in one book. The needs of a server administrator are vastly different from the needs of a desktop administrator. So I chose an approach that makes sense for the server administrator, and all topics are selected and organized to make sense for your day-to-day work as a server administrator.

Who This Book Is For This book is written for Linux administrators, whether novice or experienced, who are looking for a quick, thorough, and authoritative introduction to daily Ubuntu Server management.

How This Book Is Structured The book starts by describing Ubuntu Server, with a special focus on storage configuration, which is an especially important concern when dealing with server environments. You’ll then find a quick introduction to driving Ubuntu Server from the command line, in case you haven’t done this before. The third chapter tackles some of the common generic tasks of a server administrator, including managing software packages and configuring a graphical user interface. Next are chapters about file system management, Ubuntu Server security, managing processes, and the boot procedure. The last chapter, which deals with stand-alone server functionality, explains Bash shell scripting—in fewer than 30 pages, you’ll learn everything you ever need to know about this complex topic. The second part of the book teaches you all about network services. First, you’ll learn how to configure and troubleshoot a network interface. Next, you’ll read how to set up infrastructure services such as time services, name services, and DHCP. The following chapters discuss managing file services, the Apache web server (including performance tuning hints and a section on virtual hosts), and related packages such as MySQL. Finally, the last chapter provides an overview of the approaches to running virtualization on Ubuntu Server.

xix

xx

■INTRODUCTION

Prerequisites To get the most out of this book, you should have a computer that you can use to install Ubuntu Server. Any Pentium-based system with 128 MB of RAM and a hard disk with at least 2 GB of free space will do fine. You also need the Ubuntu Server software, which you can download from www.ubuntu.com. Apart from these simple elements, there are no further prerequisites. This book assumes no preliminary knowledge of Linux or Ubuntu.

Downloading the Code The source code for this book is available at www.apress.com in the Downloads section of this book’s home page. Please feel free to visit the Apress web site and download all the code there. You can also check for errata and find related Apress titles.

Contacting the Author The author can be reached via his web site (www.sandervanvugt.com) and by mail at [email protected].

CHAPTER

1

Installing Ubuntu Server Y

ou probably chose Ubuntu as a server solution because of either your gratifying experience using it on the desktop or the raves you’ve heard from others about its user-friendly approach. Accordingly, you might expect the general Ubuntu Server installation process to be fairly easy, and indeed it is. Nevertheless, because your ultimate goal is to deploy the server in a production environment, it’s a good idea to follow some key aspects of the installation process with rigor, and this chapter is intended to help you do exactly that. To keep things as simple as possible, you’ll read how to complete the installation on a real server, with no virtualization involved. You’ll explore the different options presented to you while installing Ubuntu, as well as the best choice to make to ensure that your installation is successful.

Preparing for the Installation Before starting the installation, you have to do a bit of preparation. First, you must make sure that the required hardware is available. At the most basic, any PC will do, but, if you are interested in putting a real server to work, I recommend using server-grade hardware because that kind of hardware is optimized for the tasks that servers typically perform. On such hardware, you can install Ubuntu directly or virtualized. If you don’t have server-grade hardware available, a standard PC is fine. In the end, it all depends on what you plan to do with your Ubuntu Server. An Apache web server at home does have some other requirements—as a corporate database server, for example. In this chapter you won’t learn how to install Ubuntu Server on a computer that already has some Windows installation. The reason for this is simple: on a real server you want only your server operating system and nothing else. Creating a dual-boot machine is cool for a desktop operating system, but you just don’t want that for a production server. Because your Ubuntu Server probably has to be available at all times, it won’t have any time to run anything but Ubuntu Server. So at this point, make sure that you have the hardware available to start the installation of a dedicated server. Also make sure that you have the installation CD, which can be downloaded from www.ubuntu.com. (Make sure that you select the server version of Ubuntu.) In this book, I’m working with Ubuntu Server 8.04 LTS. The letters LTS stand for Long Term Support, which means that this version of Ubuntu Server will have five years of support. That makes it a perfect server operating system to run in an enterprise environment, in which support is of the highest importance. 1

2

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Are you looking for the latest that Ubuntu Server has to offer? You can download the latest version of Ubuntu Server from www.ubuntu.com.

Starting the Ubuntu Server Installation Process Have everything ready? Time to go! I assume that you already found the CD image at www.ubuntu.com and burned it as an image to a CD. Insert the installation CD in your server’s optical drive and boot your server. Make sure the server boots from the CD-ROM and follow these steps to complete the installation. 1. In the installation menu that appears once the CD spins up, specify what you want to do. Often, it will be enough to select Install to the hard disk, but in certain cases other options are required as well. This is especially the case if you want to install in a language other than English and you’re using a keyboard different from a US keyboard. If this is the case, use the F2 and the F3 keys to specify your language settings. The other options are rarely used, although the F6 key hides an option that some will like: Free software only in this menu ensures that no commercial packages are installed on your server. Make sure that you have everything you need, select Install Ubuntu Server, as shown in Figure 1-1, and then press the Enter key to start the installation.

Figure 1-1. In many situations, you just have to press the Enter key to start the installation.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

■Note If your graphical hardware doesn’t support displaying the full graphical menu, you might get an installation screen that looks a little different. In that case, press F1 to see the options that are mentioned before.

2. The next screen shows a menu-driven interface. Ubuntu Server does not have a graphical user interface (GUI) by default because it runs on servers that are hidden in a data center, anyway. Using a GUI would be a waste of precious system resources, so the installation procedure itself is also text-based. In case you did not choose your installation language in the first step of this procedure, you get another chance in the second screen. In this book we’ll use English; if you want to install in another language, select it from the menu that you see in Figure 1-2.

Figure 1-2. If you did not specify the installation language in the boot screen, you have another chance of selecting the language here. 3. Based on the language that you selected, you’ll see a list of countries (see Figure 1-3). Select your country to make sure that other local settings are applied automatically. If your country is not in the default list, browse to the bottom of the list and select Other, which supplies a larger list.

■Tip Ubuntu Server makes some choices for you automatically. If you want to make these choices yourself, use the Go Back button that appears in almost every screen of the installer. This will display a more detailed list of options that are relevant to that particular stage of the installation, and you can choose what you want to do yourself.

3

4

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-3. If your country doesn’t appear in the default list of countries, select Other to choose from a larger list of countries. 4. Next, you can have the installer automatically detect the keyboard that you are using. After selecting this option, the installer will ask you to use some keys on your keyboard; based on the keys you use, the installer will determine the correct keyboard setting. If you don’t want to use this feature, click No from the screen that you see in Figure 1-4, and select your keyboard type from the list displayed next.

Figure 1-4. The installation program can automatically detect the keyboard layout that you are using.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

5. If you want the program to detect the keyboard automatically, select Yes. Next, the installer will ask you to hit a specified key (see Figure 1-5), by which it can detect the right keyboard layout in a matter of seconds. If you don’t want to do the automatic detection, that’s fine. You can just select the keyboard from a list of keyboards.

Figure 1-5. Based on the keys that you pressed, the installation program will quickly detect the proper keyboard layout. 6. After the keyboard is configured, most hardware is detected and configured automatically. Some hardware—such as WiFi network cards or graphical adapters—may require later configuration. Among the most important settings is the network configuration. If a DHCP server is available in the network to automatically assign IP addresses, your server will be provided with an IP address and ask you only for a name for the server; you’ll see nothing that is related to the configuration of the network board at all! If you don’t have a DHCP server, the network configuration program will start automatically. For a server, it is always a good idea to work with a fixed IP address, because you wouldn’t want your services to suddenly switch to a different IP address suddenly. To start the manual network configuration, use the Go Back button now and manually configure the network card. You’ll see a list of the available options. In the next step, you manually configure the IP address of your server. 7. After selecting the Go back option, move your cursor to Configure network manually (see Figure 1-6) and press Enter.

5

6

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-6. If you don’t use the Go Back option, you can’t configure your network manually. 8. Enter the IP address that you want to use for your server, select Continue, and press Enter. Not sure what information you need here? Then either return to step 6 and have DHCP automatically assign an IP address, or ask your service provider or network administrator for the proper address configuration. 9. Every IP address needs a netmask that explains to your server what network it is in. Most IP addresses can use the netmask that is assigned to them by default. If this doesn’t work in your situation, enter the proper netmask in the screen shown in Figure 1-7, select Continue, and press Enter. 10. Now you’re asked to enter the IP address of the default gateway. This is the IP address of the router that is connected to the rest of the Internet. If you don’t know what to enter, ask your network administrator what to use here and then proceed to the next step. 11. Enter the IP address of the DNS server that you want to use (see Figure 1-8). This server allows you to reach other computers on the Internet by their names instead of their IP addresses. If you are on a small network, this is probably the address of a server at your Internet service provider. If you are on a larger network, the network administrator may have configured a separate DNS server. Ask what IP address you should use and then proceed to the next step. You would normally enter two IP addresses for DNS name servers to ensure that names will still be resolved if the first DNS server is down. To enter a second IP address for a DNS server, just enter the address with a space as the separator between the two addresses. Use the actual IP addresses here, not names (because using names requires a means for them to be resolved, and setting up that mechanism is just what you’re doing here).

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-7. On most networks, the default netmask can be used.

Figure 1-8. The IP address of the DNS server is necessary to contact other computers on the Internet by their names instead of their IP addresses.

7

8

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

12. Now you are asked for the name you want to use for your computer. By default, the installer assigns a host name automatically (depending on the hardware configuration you’re using, it will usually be Ubuntu). There’s nothing wrong with using this assigned name, but you may want to use a name that provides a little more information or individuality. Also, the name typically has to conform to the naming scheme of your network.

Figure 1-9. The default host name is set to Ubuntu. You may want to change that to something that provides more information. 13. As the last part of the network configuration, you have to enter a domain name now. If there’s a DHCP server on your network, you’ll see that a domain name was entered automatically. (You’re free to change that automatically assigned name to any other name if it doesn’t fit what you want to do with your server.) 14. Now you have to select your server’s time zone. It is based on the country that you selected earlier, so if you don’t see the time zone you want to use, go back and change the country that you selected or change the time zone after your server has been installed completely.

Configuring the Server’s Hard Drive You’ve now completed the first part of the installation, but basically nothing has changed on your computer yet. So, if you want to stop the installation of Ubuntu Server and install Windows NT anyway, you can. If you want to continue, it’s time to do some thinking. The installer is going to ask you how you want to lay out your server’s hard drive. You have a couple of choices here, and you’d better make them now because they’ll be very difficult to change later. So wait before you press Enter and read this section first.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

The first choice you have to make is between using just one partition or volume to install your server, or using several of them. Using more than one storage unit can make your server more flexible and more secure. If, for example, you put all data on one large storage unit (like one root partition), a user or a process can fill that partition completely by accident, thus making your server completely unusable. It’s useful to use more than one storage unit for the following reasons, as well: • Working with more than one partition or logical volume makes it possible to mount them with different properties while mounting. For example, a partition where things normally wouldn’t change can be mounted as read-only, thus increasing the security of your server. • Using more than one partition makes it easier to work with external storage like a storage area network (SAN). For example, you could put all the system data on the server’s local hard drive, and all the user data could reside on the SAN. • Working with more than one partition is necessary in some situations. For example, to make sure that your server will always be able to start up, you could create a separate partition to boot from. Next, you have to decide between using logical volumes or partitions. Partitions are fixedsize slices of disk space. It is very hard (although not impossible) to change them later. Logical volumes from the Logical Volume Manager (LVM) system are much more flexible. It’s very easy to resize them, and they offer some other cool advanced features as well (about which you’ll read in Chapter 4 of this book). In the next subsections you’ll learn more about partitions and volumes.

Working with Traditional Partitions Partitions have been used to divide server hard disks in several usable slices since the very first days of the personal computer. To create partitions, you use the partition table in the master boot record of your server’s hard disk. Because this partition table is only 64 bytes, you can create only four partitions here. Originally, these were so-called primary partitions. In some cases, four is not enough, and that’s what the extended partition was invented for. A primary partition can contain a file system directly. This is not the case for extended partitions. An extended partition functions like an empty box that allows you to create logical partitions inside of it. So the only purpose of extended partitions is to allow you to create logical partitions. No logical partitions without extended partitions. The number of logical partitions that can be created depends on the hardware and software that you are using, but it is never more than 16. So, using the traditional partitioning scheme, a maximum of 20 partitions can be created. This number may seem enough, but in some situations it isn’t. The next characteristic of a traditional partition is that it is not very flexible. If, after some time, you learn that one of the partitions is almost full, it is very difficult in a traditional partitioning scheme to increase the size of one partition while decreasing the size of another partition. It can be done, but the work is really best left to the experts, because you could lose all data on all partitions involved.

9

10

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

■Caution You think you’re an expert? Okay. Here’s how it works: first shrink the size of the file system in the partition (check Chapter 4 for more information). After that, delete the partition you want to reduce in size and create it again, using the same start cylinder and the new size you want to use for the partition. But remember when you do this, you’re only one small step away from destroying all files on your partition, so be careful and don’t blame me if it goes wrong. Better wait until you read Chapter 4 before you try to perform a potentially dangerous action like this.

Advantages of Logical Volumes The LVM system can be used to avoid the disadvantages of traditional partitions. If you use an LVM layout, you format the logical volumes instead of the partitions. The logical volume has more or less the same functionality as the partition, but LVMs have some important benefits: • You can create as many as 256 LVM logical volumes. • Logical volumes are very easy to resize. • A logical volume does not have a one-to-one relationship with the storage device that it’s created on. Thus, it’s possible to create a logical volume that uses three hard disks at the same time. (Although this process is certainly not recommended because if you lost one hard disk, you would lose your complete volume, but it’s cool that you can do it, and there are ways of coping with hard disk failure.) • Logical volumes support working with snapshots. A snapshot allows you to freeze the state of a volume at a given point in time, which makes backing up data on a logical volume very convenient. This is done in a very clever way, so that the snapshot uses only a fraction of the disk space of the original volume.

■Note Really want to understand how LVM is working? The LVM-HOWTO at http://tldp.org/HOWTO/ LVM-HOWTO has some good in-depth information. Chapter 4 has some more information as well.

Apart from all the good news, LVMs have one drawback: it is very hard to boot from a logical volume. Therefore, even if you’re using LVMs, you’ll always need at least one traditional partition to boot your server.

Creating an Efficient Hard Disk Layout on the Server When installing a Linux server, it’s common not to put all files on one huge partition or logical volume for the reasons just discussed. Because Linux servers normally contain many different files, it is a good idea to create some partitions or volumes to store these files. Each of these partitions or volumes is assigned to (mounted on) a specific directory. Of course, you can put everything on one partition only, but you may run into troubles later, such as if a user completely fills this partition. Before starting the actual installation of your server, you should decide on the most appropriate way to use your hard drive.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

■Tip You don’t want to go for the complicated setup straight away—just a simple Ubuntu Server? Use the first Guided partitioning option from the menu in Figure 1-10 (in the section “Using the Guided Partitioning Procedure”) and press Enter. In that case, you can also skip the next couple of pages of this chapter. Want to have not just a Ubuntu Server but also a well-performing Ubuntu Server? Read on, please.

It is a good idea to give the directories their own partition or logical volume on your server. • /boot: Because the information in the /boot directory is needed to start a server, it’s a rather important directory. For that reason and especially to protect it from everything else that is used on your server, /boot often has its own partition. This directory cannot be on a logical volume because booting from logical volumes is currently not supported out of the box. Because this directory is the first thing that is needed when booting a server, it’s a very good idea to put it at the beginning of your server’s hard drive. Doing so will prevent time-out issues while booting the server. It will also make troubleshooting a lot easier. And if these reasons are not enough, it is a good idea to have a separated /boot partition if working on a server with an older BIOS because it should be at the beginning of your hard disk. Typically, it is more than enough to allocate the /boot directory to a 100 MB partition. • /: The root directory of the file system always has its own file system, which is also referred to as the root file system. The root file system is rather simple: it contains everything that hasn’t been split off to another partition. If no data files are stored here, 8 GB is typically large enough. • /var: The /var directory is used by lots of processes that need to create files on your server dynamically (such as printer spool files or the cache that is created by your proxy cache server). However, because the /var directory is so very dynamic, it has an increased chance of problems. So it’s always a good idea to put it on its own partition. In a normal environment, 4 GB is a reasonable amount of disk space to assign to this partition, but in some specific environments, you’ll need lots more.

■Caution A badly configured log system can eat up available disk space very fast. So don’t just create a /var volume and never look at it again; make sure that you tune your logging services as well. (More on that

in Chapter 3.)

• /home: The /home directory belongs to the user and is where he or she will normally store files if the server is a file server. Because it also is very dynamic, and users are accessing it all the time, make sure that it also has its own partition. The amount of disk space you reserve for this partition depends on how much space you want to grant to your users.

11

12

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

• /srv: The /srv directory is used by servers such as the Apache web server and the FTP server to store data. Because files in this directory can be accessed by users that make a connection from the Internet, it should have some extra security. A simple way to do this is to place it in its own partition or volume. The amount of disk space on this partition depends on how you are going to use your server. If it is a public FTP server, assign it the maximum space; if your servers serve web or FTP files only occasionally, you can keep the disk space in this directory quite moderate.

File Systems Because it’s a Linux server, Ubuntu offers a choice from many file systems. When creating disk partitions or volumes, you have to tell the partitioning utility what type of file system you want to use on that volume. The following file systems are available for Ubuntu Server: • Ext3: This is the default file system on almost all Linux distributions. Although it is a very stable file system with many debug tools available, there is a major drawback: Ext3 isn’t the best file system to handle many files on one volume. It also isn’t the fastest if you have to write many small files to your volume. • Ext2: Ext2 and Ext3 are largely the same, except that Ext3 uses a journal to make it easier to recover a corrupted file system. This isn’t the case for Ext2. Despite the absence of a journal, Ext2 is still a good choice for small volumes where the services of a journal aren’t necessarily needed (because, for example, the files are not supposed to be opened for writing anyway). For instance, if you create a 100 MB /boot partition, the Ext2 file system is an excellent choice for it. • ReiserFS: ReiserFS is a very advanced file system with great features. These features include journaling, advanced indexing, and many others. ReiserFS is particularly strong if many small files have to be written. However, it has two drawbacks: its main developer is currently facing myriad legal issues, and the file system is not particularly known for its stability and active community support. Use it if you want to write intensively or if you want to store many files in one directory, but make sure that you make a good backup at the same time. Want something that offers the same functionality but is more stable? Better use XFS. • XFS: XFS was developed by SGI as a special-purpose open source file system. It is especially meant to be used when the server will see lots of usage or when the files are very large. So use it if you want to stream lots of media files or if you have an FTP server with multiple terabytes of data. In its most recent versions, XFS offers a good alternative for ReiserFS. • Ext4: As you can probably guess from its name, Ext4 is the next generation of the Ext file systems. At the time of this writing, the first code was just available and it was far from being a usable file system. It is not in Ubuntu Server 8.04, but you will probably see it soon in future releases of Ubuntu Server. • FAT: FAT vfat and NTFS file systems allow you to create a multiboot environment for a computer on which both Windows and Linux are installed. The purpose of these file systems is to access files stored in Windows partitions. Because there’s no Windows on your Ubuntu Server, you don’t need it there.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Continuing the Installation of Ubuntu Server Now that you know some more about the choices that are offered when installing Ubuntu Server, let’s continue. You now have to specify how you want to partition your server’s hard disk. Because the partitioning of a hard disk is one of the most important parts of the server installation process, we will cover all choices. The choices are divided into three main categories: • Guided - use entire disk: This is the easiest option. It offers a guided installation of your hard disk, based on traditional partitions. If you want to keep the partitions that already are on your server’s hard drive, use Guided – resize instead. This option will shrink any installed operation system and use the space that is freed for installing Ubuntu Server. There is also an option that allows you to use the largest continuous free space, but this option is usable only if you have some free, unpartitioned disk space on your server’s hard drive. Because the last of these two options is useful only for multiboot configurations, it is not covered in this chapter. • Guided - use entire disk and set up LVM: This configuration option is a bit more complex. It offers you a wizard that allows you to create an LVM-based disk configuration. If you want to put up a well-secured environment, you can even choose to set up encrypted LVM instead. This is a good idea if you want to be sure that unauthorized people will never be able to access the data on your volume. Be aware, though, that an encrypted LVM volume will never be accessible if the related user credentials get lost. Also, there is a performance price for using encrypted volumes, so make sure that you really want them before applying them. • Manual: Use this procedure if you’re sure you know what you are doing and you don’t need the help of any wizard.

Using the Guided Partitioning Procedure Let’s first talk about the guided procedure to set up a server hard disk. Your starting point is the screen shown in Figure 1-10. (At least, something that looks like it because the exact offering depends on what you already have on your server’s hard drive.)

13

14

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-10. You have several choices for configuring your server’s hard disk. 1. From the screen shown in Figure 1-10, select Guided - use entire disk. 2. The installation shows an overview of all the available hard disks (see Figure 1-11). Choose the disk that you want to use and press the Enter key.

Figure 1-11. Choose the hard disk that you want to partition.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

3. Now the installation program shows you what it intends to do with your server’s hard disk (see Figure 1-12). The program isn’t very verbose about this, as it just shows that it wants to create a swap partition and an Ext3 partition. But you probably don’t care because this option is meant to offer a simple partitioning for your server. So select Yes and then press Enter to continue.

Figure 1-12. The default partitioning scheme is rather basic.

Using the Guided LVM-Based Setup The procedure for an LVM-based disk layout is a lot like the simple guided disk setup. Choosing the guided LVM-based setup also brings you to a screen from which you can select the disk or disks that you want to use. Press Enter to select your disk. The partitioning program next tells you that it wants to write a basic partitioning scheme to disk before it can continue (see Figure 1-13). This is needed because an LVM environment is created on top of a traditional partition. Be aware that this is the point of no return; after you’ve written this basic partitioning scheme to hard disk, you can’t undo your installation.

15

16

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-13. Before the logical volumes can be created, some traditional partition setup has to be written to disk. Once the default partitioning has been set up, the installation program makes a proposition for two logical partitions that are set up on top of that (see Figure 1-14). By default, this is a root partition, formatted as Ext3 and a swap partition. Select Finish partitioning and write changes to disk and press Enter to continue with the installation.

Figure 1-14. By default, two logical volumes are created on top of the partitions.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Manually Setting Up Your Hard Drive If you want to set up your server’s hard drive manually, that’s perfectly fine, but you need to do some thinking before you start. First, you need to decide if you want to use LVM or traditional partitions only. Once you have made this decision, you need to choose between the different file systems that are available for Linux. I recommend making a small overview like the one in Table 1-1. While making such an overview, don’t forget to assign some swap space as well. In Linux, swapping happens to a partition or volume, so you must consider it while setting up your server. In general, there is no need to make your swap space larger than 1 GB, with the exception of servers with special applications such as Oracle. If that is the case for your environment, consult your application documentation to find out what amount of swap space is reasonable for your situation. Table 1-1. Hard Disk Configuration Overview

Directory

Type

File System

Size

/boot

Primary partition

Ext2

100 MB

/var

LVM

XFS

4 GB

/home

LVM

XFS

200 GB

/

LVM

Ext3

50 GB

swap

LVM

Swap

1 GB

Once you have made up your mind about the hard disk usage, follow these steps to apply your decision. 1. From the Partition disks interface, select Manual. 2. You now see a screen like the one in Figure 1-15. In this screen, select the hard disk that you want to configure first.

Figure 1-15. Select the hard disk on which you want to create partitions and volumes.

17

18

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

3. Because you have just selected an entire hard disk to configure, the installation program warns you, stating that continuing will remove all existing partitions on the hard drive. If you are sure you want to do this, select Yes and press the Enter key. If there’s more than one hard disk on which you want to create partitions, select this additional hard disk and repeat these steps. 4. You now see an overview of all available unconfigured disk space on the selected hard drive (see Figure 1-16). Select this free space and press Enter.

Figure 1-16. Select the available disk space and press Enter. 5. Now the installer asks how to use the available disk space. To create the setup detailed in Table 1-1, you first have to set up two partitions. One of them will be used by the /boot partition, and the other will contain all available disk space on the hard drive. This second partition is used to create a partition of the type 0x8e (LVM), which will be used to set up logical volumes later. To set up the /boot partition first, select Create a new partition (see Figure 1-17) and press Enter.

■Note Partition types are written as hexadecimal numbers. You know that they are hexadecimal numbers because you can often see them as 0x8e (the 0x indicates that it is a hexadecimal number). In most cases, however, there is no problem in omitting this 0x, so 0x8e and 8e can both be used to refer to the partition ID.

6. Next, enter the size that you want to assign to the partition. You can enter a percentage of disk space that you want to use or just type max if you want to use the maximum available size. Next, select Continue and press the Enter key.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

7. Now you have to enter the type of partition you need, and the installation program offers a choice between a primary and a logical partition. If you choose a logical partition, the program will automatically create the necessary extended partition. Because you need only two partitions in this scenario, you can choose the primary partition type for both of the partitions.

Figure 1-17. You first have to create two traditional partitions, even if you want to create an LVM-based setup. 8. Now specify where the new partition should start. Choose Beginning to create the partition at the beginning of the available disk space, or choose End to create it at the end of the available disk space. It makes sense to create the first partition at the beginning, so select Beginning and then press the Enter key. 9. Next, you see a screen that contains all the default properties for the new partition (see Figure 1-18). Assuming that this really is the partition that you want to use for /boot, make sure you enter the following values, select Done setting up the partition, and press the Enter key to continue. • Use as: Ext2 file system. You are going to create a very small file system with files that will rarely change, so it doesn’t make sense to use a journaling file system here. • Mount point: /boot • Mount options: Use the options that are selected by default • Label: none • Reserved blocks: 5% • Typical usage: standard • Bootable flag: off

19

20

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-18. Make sure your boot partition uses these settings. 10. In the screen shown in Figure 1-19, select the available free space to create the LVM partition.

Figure 1-19. Select the available free space again to create the LVM partition.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

11. Select Create a new partition and accept the default in which all available disk space is assigned to the new partition. Then specify that the new partition should be a primary partition. Next, in the screen with the partition settings, make sure you set the following options as shown: • Use as: physical volume for LVM • Bootable flag: off

■Tip Did something not work out the way it should have? Take a look at the syslog screen. You’ll be able to see exactly what the installation program is trying to do and whether it succeeds. You can access the syslog screen by using Alt+F4. To return to the main installation screen, use Alt+F1.

12. Now select Done setting up the partition, and press Enter. 13. Once back in the main screen (see Figure 1-20), select Configure the Logical Volume Manager and press Enter.

Figure 1-20. After setting up the partitions, you must create the LVM environment. 14. You’ll now get a message (see Figure 1-21) that the current partitioning scheme has to be written to disk before setting up LVM. Select Yes and press Enter.

21

22

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-21. You must write the changes in the partitioning to hard disk before you can create logical volumes. 15. As the first step in the setup of an LVM environment, you must now assign all usable disk space to a volume group. From the screen shown in Figure 1-22, select Create volume group.

Figure 1-22. An LVM setup is based on one or more volume groups.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

16. Next, enter a name for the volume group. In this example setup, I’ll use “system”. After specifying the name, select Continue and press Enter. You next see a list of all devices that are available for the LVM environment, which in this case is just one device that probably has the name /dev/sda2 (see Figure 1-23). Select the device and select Continue once more to return to the main screen of the LVM setup program.

Figure 1-23. In most situations, you’ll select the device /dev/sda2 to be used by the Logical Volume Manager. 17. From the LVM main screen, select Create logical volume. Next, select the volume group that you have just created and enter a name for the first logical volume that you want to use (see Figure 1-24). I recommend using the name of the file system you are going to mount on the logical volume, so root is a decent name for the root file system, var is good if you are going to mount the /var directory on it, and so on. I’ll show you how to create the root logical volume in this and the following step. Make sure to repeat these steps for all other volumes that you want to create.

23

24

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-24. Every logical volume needs a unique name. 18. Now enter the size that you want to assign to the logical volume. Even if logical volumes are quite flexible, you should try to specify a realistic size here. Next, specify the file system sizes that you want to use on your logical volumes and finalize the LVM setup procedure by clicking Finish.

■Tip If you run into problems while writing the new partitioning scheme to disk, this is probably due to a conflict with some already existing setup. In this case, it may be a good idea to wipe your server’s master boot record (MBR). From the installation program, use Alt+F2 to display a console window. Press Enter to activate the console and enter the following command: dd if=/dev/zero of=/dev/sda bs=512 count=1. This will wipe your server’s MBR so that you can start all over again. You’ll have to restart the installation as well.

19. Back in the main partitioning screen, you now see a list of all the logical volumes that you have created; they’re just on top of the list (see Figure 1-25). You now have to put a file system on every one of them. To do that, select the logical volume from the list and press Enter.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

Figure 1-25. Select the logical volume you want to format and press Enter to put a file system on it. 20. You now see the Partition Settings screen, in which the option Use as shows do not use. Select this option and press Enter. Now from the list of available file systems, select the file system that you want to use on this volume; for example, Ext3 if this is the root volume (see Figure 1-26).

Figure 1-26. Select Use as to specify the file system that you want to use on your logical volume.

25

26

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

21. Select the Mount Point option and select the directory to which you want to assign this volume (/ for the root volume). Select any specific options you want to use on this file system as well and then select Done setting up the partition. 22. Repeat this procedure to put a file system on all the remaining logical volumes. When finished, from the Partition disks main screen (refer to Figure 1-25), select Finish partitioning and write changes to disk. Select Yes when asked if you want to write the changes to disk. This brings you to the next stage of the installation, in which all relevant software packages are copied to your server.

Completing the Installation Now that you have created the partitioning scheme for your server’s hard drive, it’s time to finalize the installation of your server. In this part of the installation, you enter some generic properties (like some user information), and you specify what software packages to install. 1. Next, the installer asks you to create a user account. This is the user account that you will normally be working with, instead of using the root user account by default. Enter the name of the user account and then enter (twice) the password for this user. The installation of the base system now begins. In this phase, some basic packages that are needed at all times are copied to your server. Hang on because this can take a couple of minutes. 2. After the core system is installed, the installer asks if you want to use an HTTP proxy. If this is the case, enter its details now, or leave the information bar empty and select Continue to go on. 3. Next, you can choose additional software to install. The default choices allow you to select between some of the most important server tasks (see Figure 1-27): • DNS server: A DNS server allows your server to participate in the DNS hierarchy and translate computer names into IP addresses. • LAMP server: Selecting the LAMP (Linux, Apache, MySQL, and PHP server) option will set up a versatile web server for you. • Mail server: Want to set up your server as an MTA? Select this option to make it a mail server that allows you to send and receive e-mail messages for all users in your company. • OpenSSH server: This is the option you’ll want to use in almost all situations. It allows you to access your server remotely, which is very useful for doing maintenance work on it. • PostgreSQL: Use this option if you want to install your server as an application server, hosting a PostgreSQL database. • Print server: This option installs the Common UNIX Print System (CUPS). Select it if you want your server to control access to shared printers.

CHAPTER 1 ■ INSTALLING UBUNTU SERVER

• Samba File server: Need a server to offer shared files to the Windows users in your network? Check this option. The Samba file server offers the best-performing CIFS-based file server in the world, and it even is a good solution if you want to offer shared files to Linux and Apple users! That’s it for installation choices! If you need additional software to be started on your server, you need to install that software later. See Chapter 3 for more details on that.

Figure 1-27. The installation program lets you choose between different server profiles. 4. Once all software packages have been copied to your server, the system is ready for use. You just have to press the Enter key once more to restart the server, and it will be usable. Once the server has been restarted, you will see the text-based login prompt of your server. You are supposed to enter your user name and password here. A text-based prompt? Yes, this is a server, and a server is generally locked behind doors in an air-conditioned room. Therefore, there is no need to set up a graphical user environment in most cases. But because many people consider a GUI quite useful anyway, you’ll learn how to set it up in Chapter 3. For now, though, you’ll learn in Chapter 2 how to manage Ubuntu Server from the command line.

Summary You learned in this chapter how to set up Ubuntu Server. Because the file system layout is a very important part of a server configuration, special attention was paid to configuring your server’s file system with LVM or traditional partitions. At the end of this chapter, you ended up with a text-based console that’s not so user friendly. In Chapter 2, you will learn to work with the most important commands needed for Linux server administration.

27

CHAPTER

2

Getting the Most from the Command Line Y

ou may know the workstation versions of Ubuntu as very accessible graphical desktops. This is not the case, however, for Ubuntu Server! You can’t manage a server properly without using the command line, so it’s absolutely necessary that you can find your way around the Bash interface. Once you have mastered the command line, you will find it so much more powerful and flexible that you may not miss the graphical interface at all. For command-line newbies, this chapter offers an introduction.

Working As root? By default, every Linux system installation creates a user with the name root. Many Linux distributions ask you to enter a password for this user during the installation. Ubuntu Server doesn’t, and it instead takes a radically different approach to performing privileged tasks. Ubuntu Server takes a different approach for several good reasons. The powers of the user root are limitless within the confines of that operating system. As root, you can bypass all system security and do anything at all. And you will not be given a warning screen if, for instance, you log in as root and then mistakenly type in a command that destroys all the files. This is why Ubuntu Server handles privileged access in a different way. By default, the user root does not have a password, so you cannot log in and work as root in the conventional way, but you still need to perform many tasks that require root privileges. For this purpose, Ubuntu offers the sudo mechanism, which is explained in detail in Chapter 5. With sudo, normal users can perform tasks that require root privileges. And it’s very simple: for every command that needs root permissions, you type sudo first. For example, whereas user root could just type passwd linda to change the password of user linda, a normal user enters sudo passwd linda.

■Note Want to work as root? Use the command sudo su, and you’ll be root. Alternatively, you can change the password of user root as well, which allows you to log in as user root directly. Don’t ever want the possibility to log in as root? In that case, you should change the default shell for this user to /bin/false. In Chapter 5, you’ll read how to do that. 29

30

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

In a default installation, any user can use sudo to perform tasks as root. As you can guess, this doesn’t make for a very secure situation. So you should limit this privilege. In Chapter 5, you can read how to do that. Before you do this, however, you need more understanding of Linux commands.

■Tip You don’t like sudo and want to work as root anyway? You can do that, but you need to first set a password for user root. To give root a password, as a normal user, use the command sudo passwd root. Next, you can enter the new password that you want to set for the user root.

Working with the Shell Ubuntu Server uses the kernel to address and control the machine’s hardware. The kernel can be considered the heart of the Linux operating system. Ubuntu Server gives users the shell interface to tell this kernel and the services running on top of it what they should do. Typically, the shell is a command-line interface in which users can enter their commands. This interface interprets the commands that users type and translates them to machine code. Several shells are available. The very first shell that was ever created for UNIX, back in the 1970s, was the Bourne shell. In Ubuntu Server you also have /bin/sh, but it’s not the real original sh; it’s just a link to the /bin/dash shell. Another popular shell is Bash (short for the Bourne Again Shell). The Bash shell is completely compatible with the original Bourne shell, but it has many enhancements. Most system scripts on your server are executed with dash as well; dash is used as the default shell for all users. The user root, however, has Bash as its default shell.* Some people prefer using other shells, three of which follow: • tcsh: A shell with a scripting language that works like the C programming language (and is thus fairly popular with C programmers). • zsh: A shell that is based on the Bash shell, but offers more features. Because of these additional features, you can run Bash scripts in zsh, but you can’t run zsh scripts in a Bash environment. • sash: The standalone shell. This is a very minimal shell that runs in almost all environments. It is thus very well suited for troubleshooting systems.

Using Bash to Best Effect Basically, in the Bash environment, an administrator is working with text commands. An example of such a command is ls, which can be used to display a list of files in a directory. Bash has some useful features to make working with these line commands as easy as possible.

* Because it has many more features and ensures better compatibility, I prefer working with Bash. For that reason, I’ll focus on discussing Bash rather than dash features in this book. When administering your server as root,you will be working with Bash anyway.

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

Some shells offer the option to complete a command automatically. Bash has this feature, but it does more than just complete commands. Bash can complete almost everything: not just commands, but also file names and shell variables.

Using Automatic Command Completion Using this feature is as simple as pressing the Tab key. For example, the cat line command is used to display the contents of an ASCII text file. The name of this file, which is in the current directory, is this_is_a_file. So, to open this file, the user can type cat thi and then press the Tab key. If the directory has only one file that starts with the letters t-h-i, Bash automatically completes the name of the file. If the directory has other files that start with the same letters, Bash completes the name of the file as far as possible. For example, let’s say that there is a file in the current directory with the name this_is_a_text_file and another named thisAlsoIsAFile. Because both files start with the text this, Bash will complete only up to this and no further. To display a list of possibilities, you then press the Tab key again. This allows you to manually enter more information. Of course, you can then use the Tab key again to use the completion feature once more.

■Tip Working with the Tab key really makes the command-line interface much easier. Imagine that you need to manage logical volumes on your server and you remember only that the command for that starts with lv. In this case, you can type lv and press the Tab key twice. The result will be a nice list of all commands that start with lv, from which you’ll probably recognize the command that you need.

Working with Variables A variable is simply a common value that is used often enough by the shell that it is stored with a name. An example of such a variable is PATH, which stores a list of directories that should be searched when a user enters a command. To refer to the contents of a variable, prefix a $ sign before the name of the variable. For example, the command echo $PATH displays the content of the current search path that Bash is using. On any Linux system, you’ll get quite a few variables automatically. For an overview of all of them, you can use the env (short for environment) command. Listing 2-1 shows the result of this command. Listing 2-1. The env Command Shows All Variables That Are Defined in Your Shell Environment root@RNA:~# env TERM=xterm SHELL=/bin/bash SSH_CLIENT=192.168.1.71 1625 22 SSH_TTY=/dev/pts/1 USER=root LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd\ =40;33;01: or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz\

31

32

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

=01;31:*.arj=0 1;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;\ 31:*.deb=01;31:*. rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;\ 35:*.pgm=01;35: *.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;\ 35:*.mov=01;3 5:*.mpg=01;35:*.mpeg=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;\ 35:*.xwd=01;35: *.flac=01;35:*.mp3=01;35:*.mpc=01;35:*.ogg=01;35:*.wav=01;35: MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games PWD=/root LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=root VISUAL=vi SSH_CONNECTION=192.168.1.71 1625 192.168.1.70 22 LESSOPEN=| /usr/bin/lesspipe %s LESSCLOSE=/usr/bin/lesspipe %s %s _=/usr/bin/env Normally, as a user, you’ll get your variables automatically when logging in to the system. The most important source of new variables is the /etc/profile file, a script that is processed for every user who logs in to the system. Want to add a new variable? Add it to the bottom of the /etc/profile file to make sure it is available for all users. If you have some code that you want to apply to /etc/profile, you can put it in a separate file and put that file in the /etc/profile.d directory as well. The master script /etc/profile will make sure that these commands also are executed automatically. Don’t worry about naming conventions for this file because there are none. The only requirement is that the script you put in here contains valid shell code.

Working with Bash History Another useful feature of the Bash shell is the history feature, which remembers and lets you reuse commands you have recently used. By default, the last 1,000 commands are remembered. This feature is useful for sessions beyond even the current one. A file named .bash_history is created in the home directory of every user, and this file records the last 1,000 commands that the user has entered. You can see an overview of these commands by typing history at the Bash prompt. Listing 2-2 is an example of this list.

■Note In addition to the history command, you can also use the up/down arrow keys, page up/down keys, and Ctrl+p/Ctrl+n to browse the history.

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

Listing 2-2. The history Command Shows a List of All Commands That You Recently Used sander@RNA:~$ history 1 clear 2 dpkg -l "*" | grep ^un 3 aptitude search xen 4 aptitude show xen-source 5 aptitude show xen-source-2.6.16 6 exit 7 apt-get install xen 8 sudo apt-get install xen This is where the history feature becomes especially useful because you can reissue any command from this list without typing it all over again. If you want to run any of the listed (and numbered) commands again, simply type its number preceded by an exclamation mark. In this example, typing !5 would run aptitude show xen-source-2.6.16 again. Users can also erase their history by using the history command. The most important option offered by this Bash internal command is the option -c, which clears the history list for that user. This is especially useful because everything that a user types at the command line— such as passwords—is recorded. So use history -c to make sure your history is cleared if you’d rather not have others knowing what you’ve been up to. Once using this option, however, you can’t use the up arrow key to access previous commands because those commands are all erased. Because everything you enter from the command line is saved, I recommend never typing a plain-text password in the first place, even if you regularly erase the history. Most commands that do require you to enter a password will prompt you anyway if you don’t enter one right away.

Managing Bash with Key Sequences Sometimes, you’ll enter a command from the Bash command line and either nothing happens at all or else something totally unexpected happens. In such an event, it’s good to know that some key sequences are available to perform basic Bash management tasks. Here are some of the most useful key sequences: • Ctrl+C: Use this key sequence to quit a command that is not responding (or simply takes too long to complete). This key sequence works in most scenarios where the command is operational and producing output to the screen. In general, Ctrl+C is also a good choice if you absolutely don’t have a clue as to what’s happening and you just want to terminate the command that’s running in your shell. If used in the shell itself, it will close the shell as well. • Ctrl+D: This key sequence is used to send the end of file (EOF) signal to a command. Use this sequence when the command is waiting for more input, which is indicated by the secondary prompt (>). You can also use this key sequence to close a shell session.

33

34

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

• Ctrl+R: This is the reversed search feature. It will open the “reversed I-search” prompt, which helps you locate commands that you used previously. The Ctrl+R key sequence searches the Bash history, and the feature is especially useful when working with longer commands. As before, type the first characters of the command and you will see the last command you’ve used that started with the same characters. • Ctrl+Z: Some people use Ctrl+Z to stop a command that is running interactively on the console (in the foreground). Although it does stop the command, it does not terminate it. A command that is stopped with Ctrl+Z is merely paused, so that you can easily start it in the background using the bg command or in the foreground again with the fg command. To start the command again, you need to refer to the job number that the program is using. You can see a list of these job numbers using the jobs command.

Performing Basic File System Management Tasks On a Linux system such as Ubuntu, everything is treated as a file. Even a device like your hard disk is addressed by pointing to a file (which, for your information, has the name /dev/sda in most cases). Therefore, working with files is the most important task when administering Linux. In this section, you’ll learn the basics of managing a file system. The following subjects are covered: • Working with directories • Working with files • Viewing text files • Creating empty files

Working with Directories Because files are normally organized in directories, it is important that you know how to handle these directories. This involves a few commands: • cd: This command changes the current working directory. When using cd, make sure to use the proper syntax. First, names of commands and directories are case sensitive; therefore, /bin is not the same as /BIN. Next, you should be aware that Linux uses a forward slash instead of a backslash for directory paths. So use cd /bin and not cd \bin to change the current directory to /bin.

■Tip Switching between directories? Use cd - to return to the last directory you were in.

• pwd: The pwd command stands for print working directory. Although you can usually see the directory you are currently in from the command-line prompt (this is a Bash shell setting), sometimes you can’t. If this is the case, pwd offers help.

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

• mkdir: If you need to create a new directory, use mkdir. With mkdir you can create a complete directory structure in one command as well, which is something you can’t do on other operating systems. For example, the command mkdir /some/directory will fail if /some does not already exist. In that case, you can force mkdir to create /some as well: do this by using the mkdir -p /some/directory command. • rmdir: The rmdir command is used to remove directories. However, this isn’t the most useful command because it works only on directories that are already empty. If the directory still has files and/or subdirectories in it, use rm –r or (eveb better) rm –rf, which makes sure that you’ll never get a prompt for confirmation. You should be sure that you know what you’re doing when using this option.

Working with Files An important task from the command line is managing the files in the directories. Four important commands are used for this purpose: • ls lists files. • rm removes files. • cp copies files. • mv moves files.

Listing Files with ls Before you can manage files on your server, you must first know what files are there; to do that you can use the ls command. If you just use ls to show the contents of a given directory, it displays a list of files. Of course, these files have properties as well, such as a user who is the owner of the file, some permissions, and the size of the file. To list all the files along with their properties, use ls -l. See Listing 2-3 for an example. Listing 2-3. Example Output of ls -l root@RNA:/boot# total 10032 -rw-r--r-- 1 -rw-r--r-- 1 drwxr-xr-x 2 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1

ls -l root root root root root root root root

root root root root root root root root

414210 83298 4096 6805645 94600 812139 1763308 240567

2007-04-15 2007-04-15 2007-07-29 2007-06-05 2006-10-20 2007-04-15 2007-04-15 2007-03-24

02:19 00:33 02:51 04:15 05:44 02:20 02:19 10:03

abi-2.6.20-15-server config-2.6.20-15-server grub initrd.img-2.6.20-15-server memtest86+.bin System.map-2.6.20-15-server vmlinuz-2.6.20-15-server xen-3.0-i386.gz

Apart from the option -l, ls has many other options as well. An especially useful one is the -d option, and the following example shows why. When working with the ls command, wildcards can be used. So, ls * will show a list of all files in the current directory, ls /etc/*a.*

35

36

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

will show a list of all files in the directory /etc that have an “a” followed by a dot somewhere in the file name, and ls [abc]* will show a list of all files whose names start with either an “a,” “b,” or “c” in the current directory. But something strange happens without the option -d. If a directory matches the wildcard pattern, the entire contents of that directory are displayed as well. This doesn’t really have any useful application, so you should always use the -d option with ls when using wildcards.

■Tip If you really are sure that you want to use a given option every time you issue a certain command, you can redefine the command by making an alias for it. If you put the definition of this alias in the system generic “login script” /etc/profile, it will be available to all users after they log in. To do this, open the profile file for editing with a command like sudo vi /etc/profile. Next, use the o command to open a new line and enter alias ls='ls -d' on that line. Now press Esc to return to Vi command mode and use the :wq! command to save your changes. The redefined ls command will now be available to all users who log in at your server. If the alias is intended for only one user, you can also make sure that it is executed when logging in by including it in the file .bash_profile in the user’s home directory.

One last thing you should be aware of when using ls is that it will normally not show any hidden files. If you want to see hidden files as well, use the -a option.

■Note A hidden file is a file whose name starts with a period. Most configuration files that are stored in user home directories are created as hidden files to prevent the user from deleting the files by accident.

Removing Files with rm Cleaning up the file system is another task that needs to be performed regularly, and for this you’ll use the rm command. For example, rm /tmp/somefile removes somefile from the /tmp directory. If you are root or if you have all the proper permissions on the file, you will succeed without any problem. (See Chapter 5 for more on permissions.) Removing files can be a delicate operation (imagine removing the wrong files), so it may be necessary to push the rm command a little to convince it that it really has to remove everything. You can do this by using the -f (force) switch (but only if you really are quite sure). For example, use rm -f somefile if the command complains that somefile cannot be removed for some reason. Conversely, to stay on the safe side, you can also use the -i option to rm, which makes the command interactive. When using this option, rm will ask for every file that it is about to remove if you really want to remove it. The rm command can be used to wipe entire directory structures as well; in this case the -r option has to be used. If this option is combined with the -f option, the command will become very powerful and even dangerous. For example, use rm -rf /somedir to clear out the entire content of /somedir, including the directory /somedir itself. Obviously, you should be very careful when using rm this way, especially because a small typing mistake can have serious consequences. Imagine, for example, that you type

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

rm -rf / somedir (with a space between / and somedir) instead of rm -rf /somedir. The rm command will first remove everything in /. When the rm command is finished with /, it will remove somedir as well. Hopefully you understand that the second part of the command is no longer required once the first part of the command has completed.

■Caution Be very careful using potentially destructive commands like rm. There is no good undelete mechanism for the Linux command line, and, if you ask Linux to do something, it doesn’t ask whether you are sure (unless you use the -i option).

Copying Files with cp If you need to copy files from one location in the file system to another, use the cp command. This command is straightforward and easy to use; for example, use cp ~/* /tmp to copy all files from your home directory to the /tmp directory. As you can see, in this example I introduced a new item: the tilde (~). The shell interprets that as a way to refer to the current user’s home directory (normally /home/username for ordinary users and /root for the user root. If subdirectories and their contents need to be included in the copy command as well, use the option -r. You should, however, be aware that cp normally does not copy hidden files. If you need to copy hidden files as well, make sure to use a pattern that starts with a dot; for example, use cp ~/.* /tmp to copy all files whose names start with a dot from your home directory to the /tmp directory.

Moving Files with mv As an alternative to copying files, you can move them. This means that the file is removed from its source location and placed in the target location, so you end up with just one copy instead of two. For example, use mv ~/somefile /tmp/otherfile to move the somefile file to /tmp. If a subdirectory with the name otherfile already exists in the /tmp directory, somefile will be created in this subdirectory. If /tmp has no directory with this name, the command will save the contents of the original somefile under its new name otherfile in the /tmp directory. The mv command also does more than just move files. You can use it to rename files or directories, regardless of whether there are any files in those directories. If, for example, you need to rename the directory /somedir to /somethingelse, use mv /somedir /somethingelse.

Viewing the Content of Text Files When administering your server, you will find that you often need to modify configuration files, which take the form of ASCII text files. Therefore, it’s very important to be able to browse the content of these files. You have several ways of doing this: • cat: Displays the contents of a file • tac: Does the same as cat, but displays the contents in an inverse order

37

38

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

• tail: Shows just the last lines of a text file • head: Displays the first lines of a file • less: Opens an advanced file viewer • more: Like less, but not as advanced First is the cat command. This command just dumps the contents of a file on the screen (see Listing 2-4). This can be useful, but if the contents of the file do not fit on the screen, you’ll see some text scrolling by and, when it stops, you’ll only see the last lines of the file displayed on the screen. As an alternative to cat, you can use tac as well. Not only is its name opposite to cat, its result is, too. This command will dump the contents of a file to the screen, but with the last line first and the first line last. Listing 2-4. The cat Command Is Used to Display the Contents of a Text File root@RNA:/boot# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 RNA.lan RNA # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts Another very useful command is tail. If no options are used, this command will show the last ten lines of a text file. The command can also be modified to show any number of lines on the bottom of a file; for example, tail -2 /etc/passwd will display the last two lines of the configuration file in which user names are stored. Also very useful for monitoring what happens on your system is the option to keep tail open on a given log file. For example, if you use tail -f /var/log/messages, the most generic log file on your system is opened, and, when a new line is written to the bottom of that file, you will see it immediately. Use Ctrl+C to stop viewing the file that you opened using tail –f. The opposite of tail is the head command, which displays the top lines of a text file. The last two files used to view the contents of text files are less and more. The most important thing you need to remember about them is that you can do more with less. Contrary to common sense, the less command is actually the improved version of more. Both commands will open your text file in a viewer, as you can see in Listing 2-5. In this viewer you can browse down in the file by using the Page Down key or the spacebar. Only less offers the option to browse up as well. Also, both commands have a search facility. If the less utility is open and displays the content of your file, use /sometext from within the less viewer to locate sometext in the file. To quit both utilities, use the q command.

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

Listing 2-5. The less Command Can be Used as a Viewer to View File Contents 127.0.0.1 127.0.1.1

localhost RNA.lan RNA

# The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts /etc/hosts (END)

Finding Files That Contain Specific Text As a Linux administrator, you’ll sometimes need to search for a specific file by some word or phrase within the file. Because most configuration files created on your server are ASCII text files (in the /etc directory or one of its subdirectories), it is rather easy to search for text within them using the grep utility, which is one of Linux’s most useful utilities. Let’s start with a rather basic example, in which you want to get a list of all files that contain the text “linda” in /etc. You can just use grep linda /etc/* and if you want to make sure that you can search files that are readable for root only, use sudo grep linda /etc/*. Notice that the grep command is case sensitive. If you want it to be case insensitive, you should include the -i option: grep -i linda /etc/*. This command produces a list of file names, followed by the line in which the text you were looking for is shown; see Listing 2-6. Listing 2-6. The grep Utility Is Useful for Searching Files That Contain a Certain Word or Phrase sander@RNA:~$ sudo grep linda /etc/* /etc/group:linda:x:1001: /etc/gshadow:linda:!:: /etc/passwd:linda:x:1001:1001::/home/linda:/bin/sh /etc/shadow:linda:!:13671:0:99999:7::: If the output gets a little longer, you may find it confusing to see the lines that contain the text you were looking for as well. If that’s the case, use the -l (list) option. You’ll see only file names when using this option. Another disadvantage is that grep does not search subdirectories by default, but you can tell it to do so by using the -r option. If, however, you want fast results, be careful with the -r option because searching an entire directory tree structure for the occurrence of some word in a file is very labor intensive.

Using Regular Expressions When you get more experience with grep, you’ll find that it is a very powerful and versatile command. Particularly useful is the option to work with advanced regular expressions, which let you search for very specific text patterns in a file. Imagine, for example, that you want to find files that contain the text string “nds”, but only if that string occurs as a separate word.

39

40

CHAPTER 2 ■ GETTING THE MOST FROM THE COMMAND LINE

You wouldn’t want to retrieve all files that contain the word “commands”, for instance. So you would use regular expressions, and they offer a wide range of options. Four of them, which are also known as anchors, are particularly useful: • ^text searches for text at the beginning of a line. • text$ searches for text at the end of a line. • \ searches for text at the end of a word. When using any of these regular expressions, it is always a good idea to put the search pattern between single quotes. The single quotes tell the shell not to interpret anything in the regular expression, so that grep can do the interpretation. So use grep '\= 0.5) | debconf-2.0, libacl1 (>= 2.2.11-1), libattr1 (>= 2.4.4-1), libc6 (>= 2.7-1), libcomerr2 (>= 1.33-3), libcupsys2 (>= 1.3.4), libgnutls13 (>= 2.0.4-0), libkrb53 (>= 1.6.dfsg.2), libldap-2.4-2 (>= 2.4.7), libpam-modules, libpam-runtime (>= 0.76-13.1), libpam0g (>= 0.99.7.1), libpopt0 (>= 1.10), logrotate, lsb-base (>= 3.0-6), procps, samba-common (= 3.0.28a-1ubuntu4), update-inetd, zlib1g (>= 1:1.2.3.3.dfsg-1) Suggests: openbsd-inetd | inet-superserver, smbldap-tools Replaces: samba-common ( /dev/null fi endscript } This example uses some more important options. Table 3-3 provides a description of these options. Table 3-3. Options in Service-Specific logrotate Files

Option

Description

dateext

Uses the date as an extension for old versions of the log files.

maxage

Specifies the number of days after which old log files should be removed.

rotate

Used to specify the number of times a log file should be rotated before being removed or mailed to the address specified in the mail directive.

size

The size limit of a log file is specified here.

notifempty

Do not rotate the log file when it is empty.

missingok

If the log file does not exist, go on to the next one without issuing an error message.

copytruncate

Truncate the old log file in place after creating a copy, instead of moving the old file and creating a new one. This is useful for services that cannot be told to close their log files.

postrotate

Use this option to specify some commands that should be executed after performing the logrotate on the file.

endscript

This option denotes the end of the configuration file.

Like the previous example for the Apache log file, all other log files can have their own logrotate file. Some more options are available when creating such a logrotate file. Check the man pages for a complete overview.

Summary As the administrator of a Linux server, you will be doing certain tasks on a regular basis. In this chapter you read about the most important of these tasks: managing software, creating backups, scheduling services to start automatically, and configuring logging. In Chapter 4, you’ll learn how to manage your file systems on Ubuntu Server.

81

CHAPTER

4

Performing File System Management Tasks I

n Chapter 2, you learned how to perform basic management tasks related to the file system of your server. For example, you read about file copying and navigating between directories. In this chapter, you’ll learn about some more elementary file system management tasks. The concept of a mount will be discussed, and you will find out how to perform mounts automatically by using the /etc/fstab configuration file. You’ll learn about the purpose of hard and symbolic links and how you can create them. Last but not least, you’ll understand more about how the most important file systems, such as Ext3 and XFS, are organized; and how knowledge of that organization may come in handy, using advanced tools such as dd.

Mounting Disks On a Linux server such as Ubuntu Server, devices are not always mounted automatically. They are if you’re using the GUI, but otherwise you must know how to mount a device manually. Before you can access a device, you have to mount it, and in this section you’ll learn everything you need to work with this command.

Using the mount Command To mount devices manually, you use the mount command. The structure of this command is easy to understand: mount /what /where. For the “what” part, you specify a device name; for the “where” part, you provide a directory. In principle, any directory can be used, but it doesn’t make sense to mount a device (for example on /usr) because doing so will temporarily make all other files in that directory unavailable. Therefore, on Ubuntu Server, two directories are created as default mount points. These are the directories that you would typically use to mount devices. The first of these is the directory /mnt. This is typically the directory that you would use for a mount that happens only occasionally, such as if you want to test whether some device is really mountable. The second of these directories is /media, in which you would mount devices that are connected on a more regular basis. You would mount a CD-ROM or DVD in that directory with the command mount /dev/cdrom /media/cdrom. To make life easier for some of the veterans who aren’t used to a /media directory in which a CD-ROM is mounted, a third directory is available, /cdrom, which is really just a symbolic link (a kind of shortcut) to /media/cdrom. 83

84

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

The mount command lets you mount devices like CD-ROMs or DVDs, but network shares can also be mounted with this command. You just have to be more specific. If, for example, you want to mount a share named myshare that is offered by a Windows computer named lor, you would use the command mount -t cifs -o username=yourname //lor/myshare /mnt. You’ll notice in the last example that some extra information is specified. First, the file system to be used is mentioned. The mount command is perfectly capable of determining the file system for local devices by looking at the superblock that contains a short description of the file system and exists in the beginning of every file system. But if you’re using a network device, it is a good idea to avoid confusion and specify the file system type because the mount command needs to know what type of file system it is before being able to access it. The command does quite a good job guessing the right file system on the network, but you may want to avoid confusion by adding the file system type to be used as an option to mount. In the example of the share on a Windows machine, the cifs file system type is used because you want to mount on a Windows file system. You also can use this file system type to access shares on a Samba server. Next, the name of the user who performs the mount must be specified. This must be the name of a valid user account on the other system. Then the name of the share is given. In the prior example, a computer name (lor) is used, but if your system has problems working with computer names, an IP address can be used just as easily. The computer name is followed by the name of the share. Finally, the name of the directory where the mount has to be created is given. In this example, I’ve mounted it on /mnt, because this is a mount that you would perform only occasionally. If it were a mount you used on a more regular basis, you would create a subdirectory under /media (/media/lor would make sense here) and create the mount in that subdirectory. In Table 4-1, you can see a list of some of the most popular devices that you typically want to mount on a regular basis. Table 4-1. Mounting Popular Devices

Device

Address as

Remarks

Floppy disk

/dev/fd0

Because modern servers rarely have more than one floppy device drive, the floppy drive (if present) will be fd0. If more than one drive is available, use fd1, and so on.

USB drive

/dev/sdX

USB drives (including USB keys) appear on the SCSI bus. Typically, you’ll see them as “the next” SCSI disk. So, if you already have an sda, the USB device will appear as sdb.

Optical drive

/dev/sr0, /dev/hdX

If the optical drive is installed on the IDE interface, it is typically /dev/hda or /dev/hdc, depending on other IDE devices already present. On modern servers, you’ll find the optical drive more often as /dev/sr0.

Hard drive

/dev/hdX, /dev/sdX

Depending on the bus the hard drive is installed on, you will see it as /dev/hdX (IDE) or /dev/sdX (SCSI and SATA). X is replaced by “a” for the first drive, “b” for the second drive, and so on. Notice that normally you don’t mount a complete hard drive, but a file system on a partition on the hard drive. The partition on the drive is referred to by a number: /dev/sda1 for the first partition on an SCSI hard drive, and so on.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Device

Address as

Remarks

Tape drive

/dev/st0

Typically, a tape drive is installed at the SCSI bus and can be mounted as /dev/st0.

Windows Share

//server/share

Use // followed by the server name, followed by the share. Additional options are required, such as -t cifs to indicate the type of file system to be used and -o username=yourusername to specify the name of the user account that you want to use.

NFS Share

server:/share

Add -t nfs to indicate that it is an NFS server.

Options for the mount Command The mount command offers many options, and some of them are rather advanced. For example, to perform the mount using the backup of the superblock that normally sits on block 8193, you can use the command mount -o sb=8193 /dev/somefilesystem /somedirectory. Why would you want to do this? Because mounting a file system using the backup superblock may be very useful if some problem is preventing you from mounting it normally.

■Note The superblock is where all administrative data of a file system is kept. On an Ext2/Ext3 file system, a superblock is stored at the beginning of the file system, but some backup superblocks are created automatically as well. You’ll learn more about this later in the chapter.

Although these are options you would use only in an emergency, some of the more advanced options are really useful. For example, when troubleshooting your server, you may find that the root file system is automatically booted read-only. When the system is mounted read-only, you cannot change anything, so after successfully starting in read-only mode, you would want to mount read/write as soon as possible. To do that, use the command mount -o remount,rw / to make your root file system readable/writeable without disconnecting the device first. In fact, the -o remount option allows you to change any parameter of a mounted file system without unmounting it first. It’s very useful to change a parameter without losing your connection to the file system. One of the most important options for mount is the -t option, which specifies the file system type you want to use. Your server normally would detect what file system to use by itself, but sometimes you need to help it because this file system self-check isn’t working properly. Table 4-2 lists some file systems that you may encounter on your server (or other Linux systems).

85

86

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Table 4-2. Linux File System Types

Type

Description

Minix

This is the mother of all Linux file systems (it was used in the earliest Linux version). Because it has some serious limitations, like the inability to work with partitions greater than 32 MB, it isn’t used much anymore. Occasionally, it can still be seen on very small media, like boot diskettes.

Ext2

This system has been the default Linux file system for a very long time (it was first developed in the early 1990s). The Ext2 file system is a completely POSIX-compliant file system, which means it supports all the properties of a typical UNIX environment. However, it has one serious drawback: it doesn’t support journaling and therefore is being replaced by journaling file systems like Ext3 and ReiserFS.

Ext3

Basically, Ext3 is Ext2 with a journal added to it. The major advantage of Ext3 is that it is completely backward-compatible with Ext2. Its major disadvantage is that it is based on Ext2, an elderly file system that was never designed for a world in which partitions of several hundreds of gigabytes are used. For instance, it does have problems with directories that contain more than about 5,000 files. It is, however, the most stable file system we have today and therefore is used as the default file system on Ubuntu Server.

ReiserFS

ReiserFS is another journaling file system. It was developed by Hans Reiser as a completely new file system in the late 1990s. ReiserFS was used only as the default file system on SUSE Linux, but even SUSE has changed to Ext3 as its default because there just isn’t enough community support for ReiserFS.

Ext4

Ext4 is the successor to Ext3 and will fix some of the most important shortcomings of Ext3. For example, Ext4 will use a strong indexing system that helps you work with lots of files in one single directory. At the time of writing, Ext4 is still experimental, so I will not discuss it in this book.

XFS

The XFS file system was created as an open source file system by supercomputer manufacturer SGI. It has some excellent tuning options, which makes it a very good file system to store data. You’ll read some more about this file system and its options later in this chapter.

msdos

If, for example, you need to read a floppy disk with files on it that were created on a computer using MS-DOS, you can mount it with the msdos file system type. This system is something of a legacy file system that has been replaced with vfat, however.

vfat

The vfat file system is used for all Windows and DOS file systems that use a FAT file system. Use it for accessing files from a Windows-formatted diskette or optical media.

ntfs

On Windows systems, NTFS is now the default file system. However, it is possible to read from an NTFS file system. To do this, mount the medium with the ntfs file system type. Some people even trust it to write files as well, but there have been problems with that, so I wouldn’t recommend it. Anyway, there’s no reason to use it on a server.

iso9660

This is the file system that is used to mount CD-ROMs. Normally, you don’t need to specify that you want to use this file system as it will be detected automatically when you insert a CD-ROM.

cifs

When working on a network, the cifs file system is very important. This file system allows you to make a connection over the network to a share that is offered by a Windows server, as in the previous example. In the past, the smbfs file system type was used to do the same, but because cifs offers a better solution, it has replaced smbfs on modern Linux distributions. In case mounting a Samba share doesn’t work with cifs, try smbfs.

nfs

This system is used to make connections between UNIX computers.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Apart from -t, the mount command has many other options as well, which can be prefixed by using the -o option. Most of these options are file system–dependent, so no generic list of these options is provided here. You’ll find information that is specific for your file system in the man page of the mount command.

■Tip More than just partitions and external media can be mounted. For example, it’s also possible to mount an ISO file. To do this, use the command mount -t iso9660 -o loop nameofyouriso.iso /mnt. This will mount the ISO file on the directory /mnt, which allows you to work on it like you work on real optical media. Because normally you can mount devices but not files, in this command the loop kernel module is used. This module makes it possible to mount a file as if it were a device.

The last option that might interest you when using the mount command is the rather advanced -o bind option. Its purpose? It allows you to mount a directory on another directory. You can’t do that using the loop option, so if you ever need to mount a directory on another directory, use mount –o bind /somedir /someotherdir. I use it when troubleshooting a Linux server with the chroot command, for example. I first boot my server that has problems from a Linux live CD-ROM. Next, I mount my server’s root file system in a temporary directory (in /mnt, for example). Because some utilities don’t expect their configuration files to be in /mnt/etc but in /etc instead, I next use chroot /mnt to make /mnt my new fake root directory. There is one problem, though. Because the /proc and /dev directories are generated automatically, they will not be available in the chroot environment. The solution? I use mount –o bind before using chroot. The following steps show the proper command sequence to mount your Linux distribution completely using chroot and mount –o bind: 1. Boot your server from a live CD-ROM, such as Knoppix, or use the rescue system boot option that you’ll find on your Ubuntu Server installation CD-ROM. This procedure will generate the directories /proc and /dev. 2. Mount your root partition on /mnt. You’ll see that the directories /mnt/proc and /mnt/dev are almost empty. 3. Use the command mount –o bind /dev /mnt/dev and next mount –o bind /proc /mnt/proc to make sure that complete /proc and /dev directories are available in the chroot environment. 4. Use chroot /mnt to make /mnt your new root environment and do your troubleshooting. You’ll see that everything works neatly now. 5. When finished doing the troubleshooting, use exit to escape from the chroot environment and then reboot your server.

Getting an Overview of Mounted Devices Every device that is mounted is recorded in the configuration file /etc/mtab. You can browse the content of this file with a utility like cat or less. You can also use the mount command to get an overview of file systems that are currently mounted. If this command is used without

87

88

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

any other parameters, it reads the contents of /etc/mtab and displays a list of all mounted file systems that it can find, as seen in Listing 4-1. Listing 4-1. The mount Command Gives an Overview of All Devices Currently Mounted sander@ubuntu:~$ mount /dev/mapper/ubuntu-root on / type ext3 (rw,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev) /sys on /sys type sysfs (rw,noexec,nosuid,nodev) varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755) varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777) procbususb on /proc/bus/usb type usbfs (rw) udev on /dev type tmpfs (rw,mode=0755) devshm on /dev/shm type tmpfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (rw)

Unmounting Devices On a Linux system, a device not only has to be mounted, but, when you want to disconnect the device from your computer, you have to unmount it first. Unmounting devices ensures that all the data that is still in cache and has not yet been written to the device is written to the file system before it is disconnected. You’ll use the umount command to do this. The command can take two arguments: either the name of the device or the name of the directory where the device is mounted. So umount /dev/cdrom and umount /media/cdrom will both work. When using the umount command, you may get the message “Device is busy” and the dismount will fail. This is likely because a file on the device is open, and the reason you’re not allowed to disconnect the device is probably obvious: disconnecting a mounted device may lead to data loss. So first make sure that the device has no open files. The solution is sometimes simple: if you want to dismount a CD-ROM, but you are currently in the directory /media/cdrom, it is not possible to disconnect the device. Browse to another directory and try again. Sometimes, however, the situation can be more complex, and you need to first find out which processes are currently using the device. To do this, you can use the fuser command. This command displays the Process IDs (PIDs) of processes using specified files or file systems. For example, fuser -m /media/cdrom displays a list of all processes that currently have open files in /media/cdrom. The fuser command also allows you to kill the processes that have these files open automatically. For open files on /media/cdrom, use fuser -km /media/cdrom. Be careful when using the option: if you are root, it may blindly kill important processes and make your server unreadable. As an alternative to the fuser command, you can use lsof as well. This also provides a list of all processes that currently are using files on a given file system, but it provides more information about these processes. Whereas fuser just gives the PID of a process, lsof also gives information like the name of the process and the user who owns the process. After using fuser with the -k switch to kill active processes, you should always make sure that the process is really terminated by using fuser -m /var again because this will show you whether there are still processes with open files.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Another way of forcing the umount command to do its work is to use the -f option. You can force an umount with umount -f /somemount. This option is especially intended for use on an NFS or a Samba network mount that has become unreachable, and does not work on other file systems. So you will not have much success if you try it on a local file system. Another nice option, especially if you don’t like to hurry, is the -l option, which performs a “lazy umount” by detaching the file system from the file system hierarchy and cleaning up all references to the file system as soon as it is no longer busy. Using this option lets you do an umount right away, even if the file system is busy. But it may take some time to complete.

■Tip The eject command is a very easy way to dismount and eject optical media. This command will open the CD or DVD drive and eject the optical media that is currently in the drive. All you have to do is remove it. And then you can use eject -t to close the optical drive drawer.

Automating Mounts with /etc/fstab When starting your server, some mounts need to be issued automatically. For this purpose, Ubuntu Server uses the /etc/fstab file to specify how and where these file systems must be mounted. This file contains a list of all mounts that have to occur on a regular basis. In /etc/fstab, you can state per mount if it has to happen automatically when your system starts. Listing 4-2 shows the contents of the /etc/fstab file on a test server that uses LVM. In the listing, you can see that it is not only real file systems that are specified in /etc/ fstab. The /proc file system is defined here as well. This file system offers an interface to the kernel from the file system. You can read more about this in Chapter 6.

■Note The /etc/fstab file is used at system boot, but you can also use it from the command line: enter the mount -a command to mount all file systems in /etc/fstab that are currently not mounted and have the option set to mount them automatically. Also, if a device is defined in /etc/fstab with its most common mount options, you don’t need to specify all mount options on the command line. For example, if the /dev/cdrom device is in /etc/fstab, you can mount it by using a shortened mount /dev/cdrom (or mount /media/cdrom) command instead of the complete mount /dev/cdrom /media/cdrom command.

Listing 4-2. The /etc/fstab File Makes Sure That File Systems Are Mounted During System Boot sander@ubuntu:~$ cat /etc/fstab # /etc/fstab: static file system information. # # proc /proc proc defaults 0 0 /dev/mapper/ubuntu-root / ext3 defaults,errors=remount-ro 0 # /dev/sda1 UUID=62ec320f-491f-44cb-a395-1c0ee5c4afb2 /boot ext3 defaults 0 2

1

89

90

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

/dev/mapper/ubuntu-swap_1 none /dev/hda /media/cdrom0 /dev/fd0 /media/floppy0

swap sw udf,iso9660 user,noauto auto rw,user,noauto 0

0 0

0 0

0

In fstab, each file system is described on a separate line, and the fields in these lines are separated by tabs or spaces. The following fields are always present: • fs_spec: This first field describes the device or the remote file system to be mounted. Typically, you will see names like /dev/sda1 or server:/mount on this line. As you can see in the example, some /dev/mapper devicenames are used. These refer to the LVM logical volumes that have been created on this system (you’ll find more information on logical volumes later in this chapter). You can also see that the device /dev/sda1, which is mounted on the directory /boot, uses its Universal Unique ID (UUID). Every disk device has a UUID, and the advantage of using it instead of a device name is that the UUID always remains the same, whereas the device name itself may change, especially in a SAN environment. UUIDs are generated automatically. In the directory /dev/disk/ by-uuid you can see the names of all existing UUIDs. If you use the ls -l command from this directory (shown in Listing 4-3), you can see to what device a certain UUID relates. The server used in Listing 4-3 uses two LVM logical volumes as well as a normal sda1 device. Listing 4-3. In the Directory /dev/disk/by-uuid, You Can See What Device a UUID Relates To sander@ubuntu:/dev/disk/by-uuid$ ls -l total 0 lrwxrwxrwx 1 root root 26 2007-07-01 23:23 2ec482ed-2046-4e99-9a4d -583db1f31ef4 -> ../../mapper/ubuntu-swap_1 lrwxrwxrwx 1 root root 10 2007-07-01 23:23 62ec320f-491f-44cb-a395 -1c0ee5c4afb2 -> ../../sda1 lrwxrwxrwx 1 root root 24 2007-07-01 23:23 901533ec-95d5-45d7-80f2 -9f6948e227d2 -> ../../mapper/ubuntu-root

■Tip On most file systems, the device name can be replaced with a label like “ROOT”. On an Ext2 or Ext3 file system, these labels can be created with the tune2fs -L command or with xfs_admin on an XFS system. Using labels makes the system more robust and avoids the situation in which adding a SCSI disk adds all the device names. Labels are static and are not changed automatically when a disk is added. Although labels are more obvious than the UUIDs generated by the system, you should consider working with UUIDs anyway because a UUID is in the device itself, and a label is in the file system. Therefore, a UUID is more direct and used by most modern Linux distributions.

• fs_file: The second field is used to describe the mount point for the file system. This is normally a directory in which the file system must be mounted. Some file systems (such as the swap file system) don’t work with a specific directory as their mount point. In the case of swap partitions, just swap is used as the mount point instead.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Note For special devices that are mounted from /etc/fstab, the specification of the file system type and mount point aren’t really necessary. For instance, you can mount the swap device also using none or foo as a placeholder instead of the word swap.

• fs_vfstype: The third field is used to specify the file system type you can use. As seen previously, many file systems are available for use on Linux. No specific kernel configuration is needed to use them, as most file systems can be activated as a kernel module that is loaded automatically when needed. Instead of the name of a file system, you can also use ignore in this field. This is useful to show a disk partition that is currently not in use. To determine the file system type automatically, use the option auto. This is what you want to use on removable media like CD-ROMs and diskettes. Don’t use it, however, on fixed media like partitions and logical volumes because it may lead to a failure in mounting the file system when booting your server. • fs_mntops: The fourth field is used to specify the options that should be used when mounting the file system. Many options are available and of these, many are file system–specific. For most file systems, the option default is used, which makes sure the file system is mounted automatically when the server boots and normal users are not allowed to disconnect the mount. Also, the options rw, suid, dev, exec, and async are used. The following list contains some of the most-used options: • async: Does not write to the file system synchronously, but through the write cache mechanism. This ensures that file writes are performed in the most efficient way, but you risk losing data if contact with the file system is suddenly lost. • dev: Treats block and character devices on the file system as devices, not as regular files. For security reasons, it’s a good idea to avoid using this option on devices that can be mounted by ordinary users. • exec: Permits execution of binary files. • hotplug: Do not report errors for this device if it does not currently exist. This makes sense for hot-pluggable devices like USB media. • noatime: Do not update the access times on this file system every time a file is opened. This option makes your file system somewhat faster if many reads are performed on it. • noauto: The file system will not be mounted automatically when the system boots or if a user uses the mount -a command to mount everything in /etc/fstab automatically. • mode: Used to set a permission mode (see Chapter 5) for new files that are created on the file system. • remount: Remounts a file system that is already mounted. It only makes sense to use this option from the command line.

91

92

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

• user: Allows a user to mount the file system. This option is normally used only for removable devices like diskettes and CD-ROMs. • sync: Makes sure the content of the file system is synchronized with the medium before the device is dismounted. • fs_freq: This field is for use of the dump command, which is a way of making backups of your file system. The field determines which file systems need to be dumped when the dump command is called. If the value of this field is set to 0, it will not be dumped; if set to 1, it will be dumped when dump is invoked. Make sure that the value is set to 0 on all file systems that contain important data that should always be included when making backups with dump.

■Note Although you might not ever use the dump command to create backups, some backup utilities do. So if you want to make sure that your backup utilities are successful, give all file systems that contain important data the value 1 in this column.

• fs_passno: This last field in fstab determines how a file system needs to be checked with the fsck command. At boot time, the kernel will always see whether a file system has to be checked with fsck or not. If this is the case, the root file system must always be checked first, and therefore has the value 1. Other file systems should have the value 2. If the file systems have the same fsck number, they will be checked sequentially. If the files are on different drives, they can be checked in parallel. If the value is set to 0, no automatic check will occur.

Checking File System Integrity When a system crashes unexpectedly, any file systems that are open can be damaged, which might prevent you from using these file systems in a normal way. If this happens, the consistency of these file systems needs to be checked; you do this with the fsck command for most file systems. XFS has some other commands, which you’ll read about later in this chapter. You can start this command with the name of the device you want to check as its argument: for example, use fsck /dev/hda1 to check files on /dev/hda1. If you run the command without any options, fsck will check the file systems in /etc/fstab serially, according to the setting in the fs_passno field in /etc/fstab. Normally, this will always happen when booting your system. Nowadays, a system administrator does not have to regularly use fsck because most modern file systems are journaling file systems. If a journaling file system gets damaged, the journal is checked, and all incomplete transactions can easily be rolled back. To offer some protection regardless, an Ext2 or Ext3 file system is checked automatically every once in awhile.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Tip On a non-journaling file system, the fsck command can take a very long time to complete. In that case, the -C option can be used when performing a manual check. This option displays a progress bar . . . which doesn’t, of course, make it go any faster, but it at least lets you know how long you still have to wait for the process to complete. Currently, the -C option is supported only on Ext2 and Ext3 file systems.

Working with Links A very useful option—although one that is often misunderstood—is the link. A link can be compared to a shortcut: it’s basically a pointer to another file. On Linux (as on any UNIX system), two different kinds of links are supported: the hard link and the symbolic link.

Why Use Links? Basically, a link makes it easier to find files you need. Links can be created for the operating system and program files that are used on that operating system, and they can be used to make life easier for users as well. Imagine that some users belong to the group account and you want the group members to create files that are readable by all other group members in the directory /home/groups/account. To do this, you can ask the users to change to the proper directory every time they want to save a file. Or you can create a link for each user in his or her home directory. Such a link can have the name account and can be placed in the home directory of all users who need to save work in the shared directory for the group account, and it’s easy to see how this link makes it a lot easier for the users to save their files to the proper location. Another example of why links can be useful comes from the world of FHS, the Filesystem Hierarchy Standard. This standard prescribes in which directory a Linux system should store files of a particular kind. In the old days, the X Windowing System had all its binaries installed in the /usr/X11 directory. Later, the name of the directory where the X Windowing System stored its configuration files was changed to /usr/X11R6. Now imagine what would happen if an application referred to the /usr/X11 directory after this change. It would naturally fail because that directory no longer exists. A link is the solution here as well. If the administrator just creates a link with the name /usr/X11 that points to the /usr/X11R6 directory, all applications that still refer to /usr/X11 can still be used. On a Linux system, links are everywhere. After Ubuntu Server is installed, several links already exist, and as an administrator, it’s easy for you to add new ones. To do so, you should understand the difference between a symbolic link and a hard link, which is explained in the next two sections: “Working with Symbolic Links” and “Working with Hard Links.”

Working with Symbolic Links A link can refer to two different things. A symbolic link is a link that refers to the name of a file. Its most important advantage is that it can be used to refer to a file that is anywhere, even on a server on the other side of the world. The symbolic link will still work. However, the biggest disadvantage is that the symbolic link is naturally dependent on the original file. If the original file is removed, the symbolic link will no longer work.

93

94

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

To create a symbolic link, use the ln command with the option -s. When using the ln command, make sure that you first refer to the name of the original file and then to the name of the link you want to create. If, for example, you want to create a symbolic link with the name computers in your home directory that refers to the file /etc/hosts, use the command ln -s /etc/hosts ~/computers. As a result, a shortcut with the name ~/computers will be created in your home directory. This shortcut refers to /etc/hosts. Therefore, any time you open the ~/computers file, you would really be working in the /etc/hosts file.

Understanding Inodes To understand the difference between a hard link and a symbolic link, you should understand the role of inodes on a Linux file system. Every Linux file or directory (from a technical point of view, there’s no real difference between them) has an inode, and this inode contains all the file’s metadata (that is, all the administrative data needed to read a file is stored in its inode). For example, the inode contains a list of all the blocks in which a file is stored, the owner information for that file, permissions, and all other attributes that are set for the file. In a sense, you could say that a file really is the inode, and names are attached to these inodes to make it easier for humans to work with them. If you want to have a look at inodes, on an Ext2 or Ext3 file system you can use the (potentially dangerous!) command debugfs. This opens a low-level file system debugger from which you can issue advanced repair commands. You can also just check the properties of the file system and files that are used in it (which is not dangerous at all). The following procedure shows how to display the inode for a given file using this file system debugger on Ext2 or Ext3:

■Note Only the Ext2/Ext3 command debugfs offers you the possibility to show inodes. The fact that this file system has powerful utilities like this one helps to make it a very popular file system.

1. Use the command ls -il to find the inode number of the file /etc/hosts. As you can see in Listing 4-4, the inode number is the first item mentioned in the output of this command. Listing 4-4. The Command ls -il Shows the Inode Number of a File sander@ubuntu:/$ ls -il /etc/hosts 15024138 -rw-r--r-- 1 root root 253 2007-06-05 00:20 /etc/hosts 2. Using root permissions, open the file system debugger. While starting it, use as an argument the name of the Ext2 or Ext3 file system on which your file resides. For example, our example file /etc/hosts is on a logical volume with the name /dev/ubuntu/root, so the command would be sudo debugfs /dev/ubuntu/root. This opens the debugfs interactive prompt. 3. Now use the debugfs command stat to display the contents of the inode that you want to examine. For example, in this case you would type stat . The result of this command is similar to what you see in Listing 4-5.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Listing 4-5. Showing the Contents of an Inode Inode: 13 Type: regular Mode: 0644 Flags: 0x0 Generation: 5 84821287 User: 0 Group: 0 Size: 1763308 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 3460 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4664e51e -- Tue Jun 5 00:22:54 2007 atime: 0x4664e51e -- Tue Jun 5 00:22:54 2007 mtime: 0x4621e007 -- Sun Apr 15 04:19:19 2007 BLOCKS: (0-11):5716-5727, (IND):5728, (12-267):5729-5984, (DIND):5985, (IND): 5986, (268-523):5987-6242, (IND):6243, (524-779):6244-6499, (IND):650 0, (780-1035):6501-6756, (IND):6757, (1036-1291):6758-7013, (IND):701 4, (1292-1547):7015-7270, (IND):7271, (1548-1721):7272-7445 TOTAL: 1730 (END) 4. Use the quit command to close the debugfs interface.

Understanding the Differences Between Hard and Symbolic Links When comparing the symbolic link and the original file, you will notice a clear difference between them (see Listing 4-6). First, the symbolic link and the original file have different inodes: the original file is just a name that is connected directly to the inode, and the symbolic link refers to the name. The latter can be seen from the ls -il (-i displays the inode number) output: after the name of the symbolic link, an arrow is used to indicate what file you are really working on. Also you can see that the size of the symbolic link is significantly different from the size of the real file. The size of the symbolic link is the number of bytes in the name of the file it refers to because no other information is available in the symbolic link. Also, you can see that the permissions on the symbolic link are completely open because the permissions are not managed here, but on the original file instead. Finally, you can see that the file type of the symbolic link is set to l, which indicates that it is a symbolic link. Listing 4-6. Showing the Differences Between Symbolic and Hard Links root@ubuntu:~# ln -s /etc/hosts symhosts root@ubuntu:~# ln /etc/hosts hardhosts root@ubuntu:~# ls -il /etc/hosts hardhosts symhosts 15024138 -rw-r--r-- 2 root root 253 2007-06-05 00:20 /etc/hosts 15024138 -rw-r--r-- 2 root root 253 2007-06-05 00:20 hardhosts 13500422 lrwxrwxrwx 1 root root 10 2007-07-02 05:45 symhosts -> /etc/hosts You may ask what happens to the symbolic link when the original file is removed. Well, that isn’t hard to predict! The symbolic link fails. Ubuntu Server will show this when displaying file properties with the ls command; you’ll get a “File not found” error message when you try to open it.

95

96

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Working with Hard Links Every file on a Linux file system has an inode. As explained earlier, all of a file’s administrative data is kept in its inode. Your computer actually works entirely with inodes, and the file names are only a convenience for people who are not too good at remembering numbers. Every name that is connected to an inode can be considered a hard link. So when you create a hard link for a file, all you really do is add a new name to an inode. To do this, use the ln command without any options; ln /etc/hosts ~/computers will create such a hard link. The interesting thing about hard links is that there is no difference between the original file and the link—they are just two names connected to the same inode. The disadvantage of using them is that hard links must exist on the same device, which is rather limiting. If possible, you should always create a hard link instead of a symbolic link because they are faster and they don’t waste an inode. Figure 4-1 shows an overview of the relationship of hard links, inodes, and blocks.

Figure 4-1. Relationship of Inodes, Hard Links, and Symbolic Links

Configuring Storage At the beginning of this chapter, you learned how to perform some of the most common file system management tasks. However, there’s more to managing storage on your server. In Chapter 1, you saw how to install your server using LVM and how to use different file systems for your server. I will now go into more depth with regard to these subjects. First, differences between file system types will be discussed in depth. Then you will learn about the creation of file systems, which involves creating and formatting partitions and logical volumes.

Comparing File Systems An important goal of this chapter is to learn about setting up the best file storage for your server. Because everything on Linux is available as a file, setting up file storage space is one of the most important tasks on a Linux server. This goes for programs and regular text files, but also for more advanced things such as devices. Your server is probably going to host lots of

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

different applications that all create their own files: some will create a few huge files; others will require the fastest possible access to no matter what files; and others may be something like a mail server, creating thousands and thousands of small files. Ubuntu Server provides you the flexibility to choose the best file system for all these different needs because many file systems are supported right out of the box. Before diving in to the details that are needed to create the partitions hosting the different file systems, let’s first compare the most important file systems that you can use. The next subsections will cover the following file systems: • Ext2 • Ext3 • ReiserFS • FAT • XFS

Ext2 Version 2 of the Extended File System has been the de facto standard for Linux for many years. It was the first stable file system that had all elements of a POSIX file system. The only reason why it doesn’t see much use anymore is because it doesn’t offer journaling.

■Note POSIX stands for Portable Operating System Interface, and its aim is to provide a standard level of UNIX compatibility. If any element running on Linux or any other UNIX version is POSIX-compliant, it will run without problems on any flavor of UNIX. The POSIX standard is not just limited to Linux and UNIX operating systems; most versions of Windows based on the NT kernel (up to and including Vista) are POSIX-compliant.

For modern file systems, journaling is an important option. In a journal, all transactions on open files can be tracked. The advantage is that if anything goes wrong while working with a system, all you have to do to repair damage to the file system is to do a roll-back based upon the information in the journal. Ext2 doesn’t have a journal, and therefore it isn’t a good choice for very large volumes: larger volumes will always take longer to check if no journal is present. If a small (less than 500 MB) volume is created, Ext2 is still a good choice, however. The first reason is that it doesn’t make sense to create a journal on a small file system because the journal itself will occupy space (an average journal can be about 40 MB). Other good reasons to use Ext2 include the facts that it is a very mature file system, everyone knows how it works, it works on many distributions, and many utilities are available for management of an Ext2 file system. Some advanced utilities are available for tuning and troubleshooting as well. A short overview of some of them is provided next. You should notice that these commands are all available for Ext3 (Ext2’s successor) as well.

97

98

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

• e2fsck: This utility is run automatically when an administrator issues the fsck command. It has some options that are specific for an Ext2 file system, one of which is the -b option that allows you to repair the file system in case the first sectors are damaged. In an Ext2 file system, the critical information about the file system is written to the superblock. A backup superblock is always present, and its exact location depends on the block size that is used. If 1 KB blocks are used, the backup superblock is in block 8193; if 2 KB blocks are used, it is in 16384; and, if 4 KB blocks are used, you can find it in 32768. By running the e2fsck -s 8193 command, for example, you may be able to repair a file system that can’t be mounted any more by using its backup superblock. Another very interesting use is e2fsck -D, which causes e2fsck to optimize directories. It can do this by trying to index them, to compress them, or by using other optimization techniques. • tune2fs: The Ext2 file system has some tunable parameters. For example, there is the maximum mount count option (which can be set using the -C option). By using this option, you can force e2fsck to run automatically every once in awhile by forcing an integrity check. This option may sound good, but on a server in which a file system is sometimes rarely remounted, it can make more sense to use the -i option to set an interval defined as a time period. For example, tune2fs -i 2m will force an e2fsck on your Ext2 file system every two months. The options to check the consistency of your Ext2 file system automatically are not the only options you can use with tune2fs. For example, the option -l will list all information from the file system’s superblock. Another interesting option is -L label, which allows you to set a volume label. This can be very useful if device names on your system do change on a regular basis: by using volume names, you can use the name of the volume when mounting the file system in /etc/fstab instead of the name of the device where the file system was created. The last interesting option is -m, which you can use to set a percentage of reserved blocks for your Ext2 file system. By default, the last 5 percent of available disk space is always reserved for the user root to prevent users from filling up the file system by accident. Use the e2fsck -m 2 command to decrease the amount of reserved disk space. • dumpe2fs: Every file system maintains a lot of administrative information, and Ext2 stores this in the file system superblock. Also, in Ext2, the block groups that are used as groups of data files can be administered as one entity. If you need to see the information about this file system administration, use dumpe2fs followed by the name you want to dump the administrative information for. Listing 4-7 shows the result of this command.

■Note When using a tool like dumpe2fs, you will see information about available inodes. Every file on every POSIX-compliant file system needs an inode to store its administrative information. On Ext2 and Ext3, inodes are created only when you are creating the file system. Normally one inode is created for about every four data blocks. If, however, you create many very small files, you can run into a situation in which free disk blocks are still available, but there are no more available inodes. This will make it impossible to create new files. As an administrator, you can use the dumpe2fs command to get an overview of available inodes on your Ext2 file system.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Listing 4-7. The dumpe2fs Command Displays Properties of the Ext2 File System root@ubuntu:~# dumpe2fs /dev/sad1 dumpe2fs 1.40-WIP (14-Nov-2006) dumpe2fs: No such file or directory while trying to open /dev/sad1 Couldn't find valid filesystem superblock. root@ubuntu:~# dumpe2fs /dev/sda1 dumpe2fs 1.40-WIP (14-Nov-2006) Filesystem volume name:

Last mounted on:

Filesystem UUID: 62ec320f-491f-44cb-a395-1c0ee5c4afb2 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super Filesystem flags: signed directory hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 62248 Block count: 248976 Reserved block count: 12448 Free blocks: 224527 Free inodes: 62218 First block: 1 Block size: 1024 Fragment size: 1024 Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 2008 Inode blocks per group: 251 Filesystem created: Mon Jun 4 22:56:35 2007 Last mount time: Mon Jul 2 03:22:21 2007 Last write time: Mon Jul 2 03:22:21 2007 Mount count: 3 Maximum mount count: 26 Last checked: Mon Jun 4 22:56:35 2007 Check interval: 15552000 (6 months) Next check after: Sat Dec 1 21:56:35 2007 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 0f4e7f5e-c83c-491b-85ca-a83d7c06f1b5

99

100

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Journal backup: Journal size:

inode blocks 4114k

Group 0: (Blocks 1-8192) Primary superblock at 1, Group descriptors at 2-2 Reserved GDT blocks at 3-258 Block bitmap at 259 (+258), Inode bitmap at 260 (+259) Inode table at 261-511 (+260) 993 free blocks, 1993 free inodes, 2 directories Free blocks: 4640-5632 Free inodes: 16-2008 Group 1: (Blocks 8193-16384) Backup superblock at 8193, Group descriptors at 8194-8194 Reserved GDT blocks at 8195-8450 Block bitmap at 8451 (+258), Inode bitmap at 8452 (+259) Inode table at 8453-8703 (+260) 7221 free blocks, 2008 free inodes, 0 directories Free blocks: 9164-16384 Free inodes: 2009-4016 • debugfs: The debugfs utility allows you to open the Ext2 file system debugger, from which you can perform powerful tasks. These tasks are performed using the special debugfs commands that can be started from the debugfs interactive shell. One of them is the lsdel command, which lists files that were recently deleted from your file system. After finding the inodes of these recently deleted files, you can use the debugfs dump command (not to be confused with the generic Linux dump command), followed by the number of the inode. For example, use dump /somefile to dump everything the inode refers to in the file /somefile that is created automatically. However, be aware that this works only if you are acting very fast: when a file is removed on any Linux file system, the inode and blocks that were used by the file are flagged as available, and the next time data is written to the volume, the inode and blocks can be overwritten. You should also be aware of the primary disadvantage of the debugfs method: it doesn’t know anything about file or directory names. Therefore, you can see the inode number of a deleted file but not its name, and that can make recovery rather difficult. Currently, however, it is the only way to recover deleted files from an Ext2 or Ext3 file system. To summarize Ext2, it offers some advantages and disadvantages. It is the file system to use for small volumes. If the size of a volume grows up to several gigabytes, though, it’s best not to use Ext2 because it can take ages to complete a file system check.

Ext3 The Ext3 file system is just Ext2 with a journal added to it—nothing more, nothing less. Therefore, Ext3 is completely compatible with Ext2. As compared with other journaling file systems, however, Ext3 has some important disadvantages, most of which are based on the fact that Ext3 uses tables for storage of information about the files and not a B-tree database, as is the case in ReiserFS and XFS file systems. Because these tables have to be created and are

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

accessed in a linear way, Ext3 is slow when dealing with large volumes or large amounts of data. Here are the most important disadvantages of using Ext3: • It takes a relatively long time to create a large Ext3 file system. • Ext3 isn’t good in handling large numbers of small files in the same directory. • Ext3 has no option to create new inodes after the file system has been created. This leaves a possibility in which disk space is still available but cannot be used because no inodes are available to address that disk space. And, on the other hand, Ext3 has two important advantages. Most important, it is a very stable file system that has wide community support. Also, it is easy to convert an existing Ext2 file system to a journaling file system. The following procedure describes how to do this: 1. Make a complete backup of the file system you want to convert. 2. Use the tune2fs program to add a journal to a mounted or an unmounted Ext2 file system. If you want to do this on /dev/sdb1, use tune2fs -j /dev/sdb1. After creating the journal, a file with the name .journal will be created on the mounted file system. This file indicates that the conversion was successful. 3. Change the entry in /etc/fstab where your file system is mounted. Normally, it would be set to the Ext2 file system type, so change the type to Ext3. 4. Reboot your server and verify that the file system was mounted successfully. The journal is the most important item in an Ext3 file system, and this journal can be configured in different ways. These journaling options are specified when mounting the file system, so you have to put them in /etc/fstab. Before discussing the different journaling options, you need to know how data is written to a hard drive. In each file-write operation, two different kinds of information need to be written: the data blocks themselves and then the metadata of a file. This includes all administrative information about the file. You can basically think of the file metadata as the information that is displayed when using the ls -l command (but some more information is added as well). When tuning the use of an Ext3 journal, you can specify whether both metadata and blocks need to be written to the journal, or just the metadata. Two options are available to you: activate them by using mount -t ext3 -o data=xxxx /yourdevice /yourmountpoint or put the data=xxxx option in fstab: • data=journal: In this option, both the data and metadata of the file that is written are written to the journal. This is the safest option, but it’s also the slowest. • data=ordered: In this option, only the file’s metadata is journaled. However, before updating the metadata with information about the changed file, a data write is forced. This ensures consistency within the file system with minimal performance impact. This is the default option when creating an Ext3 file system on Ubuntu Server. • data=writeback: This option ensures that only metadata is written to the journal and that nothing happens to the data blocks themselves. This is a rather insecure option with a serious risk of corruption of the file system.

101

102

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

ReiserFS Hans Reiser developed the ReiserFS file system in the late 1990s as a completely new file system, no longer based on file system tables but on a balanced tree database structure. This database makes locating files very fast as compared with older file systems like Ext2 and Ext3. And ReiserFS offers other advantages as well, one of which is that it has a better disk utilization. This is because it is capable of using disk suballocation, in which it is not necessary to use a complete block when writing a very small file. More than one small file can be written to the same disk block, using one leaf node in the B-tree database. Therefore, ReiserFS is more efficient in writing many small files. ReiserFS is also more flexible because it allows for dynamic inode allocation: if a file system runs out of available inodes, new inodes can be created dynamically. Because small files are stored with their metadata in the same database record, ReiserFS is fast in handling small files. Of course, ReiserFS has some minor disadvantages as well: it’s relatively slow in heavy write environments and it also gets slow if it is more than 90 percent full. Unfortunately, ReiserFS has one hugely important disadvantage: the lack of community support. It was because of this that Novell—traditionally the most important Linux distribution that uses ReiserFS—decided to drop ReiserFS as the default file system and turned to Ext3 instead. In the long term, this doesn’t bode well for ReiserFS, nor does the fact that Hans Reiser is currently in jail, facing some serious charges. Therefore, even if ReiserFS has some nice technical features, you should not depend on it too much. Better to use the XFS file system instead. Like Ext2 and Ext3, ReiserFS also has some management utilities. Here’s a short description of these tools, including some of the most interesting options: • reiserfsck: The reiserfsck tool is used to check the consistency of a Reiser file system, and it has some powerful repair options. One of them is --rebuild-sb, which stands for rebuild superblock. Use this option when you get the error “read super_block: can’t find a reiserfs file system”. Another option is the --rebuild-tree option, which hopefully you won’t see too often. This option is required when reiserfsck wasn’t able to repair the file system automatically because it found some serious tree inconsistencies. Basically, the --rebuild-tree option will rebuild the complete B-tree database. Before using this option, always make a backup of the complete partition where you are running it and never interrupt the command; doing so will definitely leave you with an inaccessible file system. • reiserfstune: The reiserfstune command can be used for several purposes. Basically, you use it to tune options with regard to the Reiser journal, but you can also use it to set a UUID and a label that allow you to mount the file system without being dependent on the name of the device where it resides. This command has some interesting options as well. One of them is the -j option that allows you to specify the journal device you want to use. This option makes it possible to separate the Reiser journal from the actual file system; it’s a very useful option if you want to avoid a situation where a single-point failure can render your system inaccessible. Another interesting option that can be used with reiserfstune is the option -l that allows you to specify a label for the file system. This label can be used when mounting the file system and thus increases your flexibility when working on the file system.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

• resize_reiserfs: As its name suggests, the resize_reiserfs utility is used to resize a Reiser file system. You should be aware that resizing a file system involves two steps: you first need to resize the device where the file system is created, and only then can you resize the file system itself. There’ll be more on resizing devices later in this chapter. And using resize_reiserfs is rather simple: for example, use resize_reiserfs -s /dev/sdb2 to resize the Reiser file system on sdb2 so that it fills this device completely. Alternatively, you can specify the size to resize it with in kilobytes (-K), megabytes (-M), or gigabytes (-G). If you want to shrink a file system by 500 MB, use resize_reiserfs -M 500 /dev/sdb2. Be aware, however, that it is not fully supported; things could go wrong when shrinking a ReiserFS file system.

■Tip If you want to shrink a ReiserFS file system, make sure to run reiserfsck immediately after the shrinking process to limit the risk of things going wrong.

• debugreiserfs: The debugreiserfs command allows you to dive into the ReiserFS administrative information to see if all is set according to your expectations. If you run it without options, you’ll just get an overview of the superblock. Several options are available to tune its workings. For example, debugreiserfs -j /dev/sdb2 prints the contents of the journal. It’s also possible to dive into specific blocks when using the -l option, which takes as its argument the block you want to see the contents of. Using this option can be useful if you want to do a block-by-block reconstruction of a file that was destroyed by accident. Summarized, ReiserFS is a robust file system that offers many advantages. Because of its current lack of support and the unpredictable future of the company, it’s not a very good idea to depend too heavily on it.

XFS For very large environments, the XFS file system is probably the best choice. It was, after all, developed by SGI for use on supercomputers. Like ReiserFS, it is a full 64-bit file system, and its major benefit is that it works great on very large files. One of the key factors of XFS is that it uses allocation groups, which are like file systems in a file system. The advantage of using these allocation groups is that the kernel can address more than one group at the same time, and each group has its own administration of inodes and free disk space. Of course, XFS is capable of creating new inodes dynamically when this is needed. All of this makes the XFS file system very flexible and robust. The XFS file system consists of three different sections: data, log, and real-time. User data and metadata are written in the data section. The log section contains the journaling information for XFS, and the real-time section is used to store real-time files. These files are updated immediately. Each XFS file system is identified by a UUID, which is stored in the header of each allocation group and helps you distinguish one XFS file system from the other. For this reason, never use a block-copying program like dd to copy data from one XFS volume to another XFS volume; use xfsdump and xfsrestore instead.

103

104

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

A unique feature of XFS is its delayed allocation, which makes sure that a pending write is not written to hard disk immediately, but to RAM first. The decision to write is delayed to the last minute. The advantage is that, when the file doesn’t need to be written after all, it isn’t written. In this way, XFS reduces file system fragmentation and simultaneously increases write performance. Another great feature of XFS is preallocation, which makes sure that space is reserved before the data blocks are actually written. This feature increases the chances that a complete file can be written to a series of consecutive blocks and thus avoids fragmentation. When creating an XFS file system with the mkfs.xfs command, some specific options are available: • Block size in bytes: This option allows you to set the block size you want to use. By default, the block size is set to 4,096 bytes, but you can also set it to 512, 1,024, or 2,048. • Inode size: Use this option to specify the size of inodes you want to create on the file system. This option is needed only if you have to do very specific things on your file system, like working with files that have lots of extended attributes. • Percentage of inode space: If so required, you can limit the percentage of the file system that can be used for storage of inodes. By default, there is no maximum setting. • Inode aligned: Make sure this option is always set to “yes” so it will ensure that no fragmentation occurs in the inode table. Like all other file systems, XFS also has its own management tools, and, because it’s a rather complex file system, many utilities are available. Here’s an overview of these utilities with a short description: • xfs_admin: Changes the parameters of an XFS file system. • xfs_logprint: Prints the log (journal) of an XFS file system. • xfs_db: Serves as the XFS file system debugger. • xfs_growfs: Expands the XFS file system. • xfs_repair: Repairs an XFS file system. • xfs_copy: Copies the content of an XFS file system while preserving all its attributes. The main advantage of using xfs_copy instead of normal cp is that it will copy data in parallel and thus will work much faster than a normal copy command. • xfs_info: Shows generic administrative information on XFS. • xfs_rtcp: Copies files by placing them in the real-time section of the XFS file system, which makes sure that they will be updated immediately. • xfs_check: Checks the consistency of an XFS file system. • xfs_quota: Allows you to work with quotas on an XFS file system. • xfs_io: Debugs the input/output path of an XFS file system. • xfs_bmap: Prints block mapping for an XFS file system. The command allows you to see exactly what extents are used by a file.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Note An extent is a group of blocks. Working with extents speeds up administration and file handling on file systems that work with large files. Also, because every file is initially written to its own extent, working with extents prevents fragmentation of your file system.

• xfs_ncheck: Generates pathnames for inode numbers for XFS. Basically, it displays a list of all inodes on the XFS file system and the path to where the file with the given inode number is stored. • xfs_mkfile: Creates an XFS file. This command is used to create files of a fixed size to an XFS file system. By default, these files are filled with zeroes. Summarized, XFS is the most feature-rich file system of all. Because of its flexibility, it is probably the best choice to store your company data, so use it for Samba shares, for example. Because you can also tune the size of the allocation unit, XFS is also a very efficient file system when writing very large files. Upon creation, make sure that you specify a large allocation unit to be used and you will get great performance.

Creating File Systems You probably have an idea now as to what file system best fits the needs of your application. The next step is to create these file systems, which involves two steps. First, you need to create the device in which you want to store the files. This can be a partition, but it can also be an LVM logical volume. After creating the device, you can use the mkfs command to create the file system of your choice. I’ll first explain how to create the devices in which you want to store your files.

Creating Traditional Partitions You can use many utilities to create partitions on a server, but one utility can be considered the mother of all other utilities: fdisk. In this section, you’ll learn how to use fdisk to create partitions on your disk. 1. From the command line as root, use the fdisk command followed by the name of the device in which you want to create the partition. If you want to create a partition on /dev/sdb, use fdisk /dev/sdb to open the partitioning utility. 2. The fdisk utility now opens its prompt. It may complain about the number of cylinders being greater than 1,024, but this dates from when many boot loaders were not able to handle hard drives with more than 1,024 cylinders, so you can ignore this message. A good start is to press the m (menu) key to tell fdisk to show a menu with all available options (see Listing 4-8).

105

106

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Listing 4-8. Working with fdisk root@ubuntu:~# fdisk /dev/sdb The number of cylinders for this disk is set to 36483. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only) Command (m for help): 3. Press the p key to print the current partition table, which will provide an overview of the size of your disk in cylinders, the size of a cylinder, and all partitions that exist on that disk. Then fdisk asks again what you want to do. Press the n key on your keyboard to create a new partition. Listing 4-9. When Working with fdisk, Use the p Key Often So That You Can See the Current Partitioning Command (m for help): p Disk /dev/sdb: 300.0 GB, 300090728448 bytes 255 heads, 63 sectors/track, 36483 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot

Start

End

Blocks

Id

System

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

4. After pressing n to create a new partition, fdisk will ask what kind of partition you want to create. On a hard disk you can create a maximum of four primary partitions. If you need more than four, you have to create one extended partition in which logical partitions can be created. In this example, I’ll show you how to create an extended partition, so press the e key to start the interface that helps you create an extended partition. 5. The utility now asks what partition number you want to use. It will also show the numbers that are available. 6. Now you have to enter the first cylinder. By default, fdisk offers you the first cylinder that is still available. It’s often a good idea to accept this choice, so press Enter. 7. Next, fdisk asks for the last cylinder that you want to use for your partition. Instead of entering a cylinder number, you can also enter a size in megabytes or gigabytes. However, because this is an extended partition that serves only as a repository where logical partitions are created, you can press Enter to accept the default value that will use all available disk space to create the partition. Listing 4-10 is an overview of what has happened so far. Listing 4-10. Creating an Extended Partition with fdisk Command (m for help): n Command action e extended p primary partition (1-4) e Partition number (1-4): 1 First cylinder (1-36483, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-36483, default 36483): Using default value 36483 Command (m for help): 8. The extended partition is now created. By itself, an extended partition is useless; it’s just an empty box that you can use to create logical partitions. Use the n key again to create a logical partition in the extended partition. The partition number of any logical partition is always 5 or greater, regardless of whether lower partition numbers are already in use. You can follow the same guidelines for creating logical partitions as the ones you followed for creating the extended partition. When finished, press p to print the partition table you have created so far.

■Note When creating a partition, Linux fdisk will flag the partition as a Linux partition with ID 83 automatically. If you are planning on doing something else with it, you can press the t key to change the partition ID of your partition. A list of all available partition types is displayed by using the l key from the fdisk menu.

107

108

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

9. Until now, nothing has really been written to the partition table. If you want to back out, you can still do that by pressing the q key. If you are sure that you are happy with what you have done, press w to write the changes to the partition table on disk and exit the partitioning utility. 10. The partition table has now been updated, but your kernel currently does not know about it; you can see that by comparing the result of the command fdisk -l /dev/sdb (which shows the current contents of the partition table) with the file /proc/partitions (which shows the partitions that the kernel currently sees). To update the kernel with this new information, as root type partprobe /dev/sdb. Now that you have created a new partition, the next step is to create a file system on it. This isn’t too hard if you know what file system you want to create. You just have to create the file system with the proper utility, and fsck is a perfect wrapper utility to create any type of file system. Just remember to use the -t option to specify the type of file system you want to create. Before explaining how to create a file system, let’s first take a look at logical volume management.

Working with Logical Volumes There’s one major disadvantage when working with fixed-size partitions. Imagine a system with multiple partitions, and you are running out of available disk space on one partition but there’s more than enough disk space available on another partition. When using fixed-size partitions, there’s really nothing you can do.

■Note “There’s really nothing you can do?” Well, that’s not completely true. You can do something: delete and re-create the partitions from the partition table and resize the file systems that are in use on them, but this is a difficult procedure in which you risk losing all data. Alternatively, you can use utilities like Partition Magic or the parted utility (which is a common Ubuntu utility), but partitions were never meant for easy resizing, which means that things might go wrong when trying to do it. Therefore, if you want to be able to resize partitions in an easy way, use logical volumes.

If logical volumes are used, you can easily resize them and their file systems to make some more space. Another advantage of using logical volumes is that you can create snapshot volumes of them. These snapshot volumes allow you to create backups in a flexible way, and I’ll explain how it works later in this chapter. Therefore, for a flexible environment, it’s best to work with logical volumes. In a system that uses logical volumes, all available disk space is assigned to one or more volume groups (basically pools from which volumes can be created). The advantage of working with volume groups is that a volume group can include several storage devices and is therefore not limited to one physical device. Even better, if you run out of disk space in a volume group, you can just simply add a new device to it to increase the amount of usable disk space.

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Currently, two systems are available for working with logical volumes. The first is Logical Volume Manager (LVM). It’s been around for some time and can be considered mature technology. The other option is Enterprise Volume Manager System (EVMS), a volume manager that was created by IBM and then open sourced. Although LVM is the more mature volume manager, EVMS functions better in a cluster environment in which volumes have to swap over from one node to another (because volumes can be marked as shareable). Although the perfect volume manager system for such a system, EVMS is not very common on Ubuntu Server, so I’ll focus on working with LVM in the rest of this chapter.

Creating LVM Volumes Creating LVM logical volumes is a simple procedure that can be performed by using a few different commands. If you understand the way an LVM environment is organized, creating logical volumes from the command line is not difficult. The bottom layer of the LVM setup is the layer of the physical volumes (pv). These include complete hard disks, or partitions that were created with the partition type 0x8e. Based on the physical volumes, a volume (vg) group is created. From there, one or more logical volumes (lv) are created. Figure 4-2 shows how all components in an LVM setup relate to each other. To work with LVM, the lvm-binaries package must be installed, so run apt-get install lvm-binaries before you start.

Figure 4-2. Schematic of LVM structure

109

110

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Tip If you understand the role of pv, vg, and lv, the rest is peanuts. All relevant commands start with either pv, vg, or lv and are followed by the specific task you want to perform. For example, if you want to create a pv, a vg, or an lv, use pvcreate, vgcreate, or lvcreate. If you want to extend them, use vgextend or lvextend (a pv cannot be extended). You want to show the current setup? Display it with pvdisplay, vgdisplay, or lvdisplay. More commands are also involved, and a good way of getting to know them all is to type the first two letters of the command that you are interested in at the Bash prompt; for example, type vg. Then press the Tab key twice. The automatic command-completion feature displays a list of all commands that start with “vg”, which makes it rather easy to pick the right one.

1. Before you start, you have to decide what exactly you want to do. If you want to use a complete hard drive, no extra preparation is required. However, if you want only to add a partition to an LVM setup, create the partition with the partition type 0x8e and make sure not to format it. 2. Use the pvcreate command to mark the devices that you want to use in an LVM environment. For example, use pvcreate /dev/sd{b,c,d} to assign devices sdb, sdc, and sdd to be used by LVM. 3. Next, create the volume group. If you have used pvcreate before to assign the physical volumes, you can now use vgcreate to create the volume group. For example, use the command vgcreate somegroup /dev/sd{b,c,d} to create the volume group. Note that in this command, somegroup is the name of the volume group that is created. When making volumes from the volume group, you have to refer to this volume group name. 4. Now use lvcreate to create the logical volumes. For example, lvcreate -n somevol -L150M somegroup will create the volume somevol as a new volume with a size of 150 MB from logical volume group somegroup.

■Tip If you want to include a complete hard disk in a volume group, no partition table can be present on that hard disk. To wipe an existing partition table, use the command dd if=/dev/zero of=/dev/sdx bs=1k count=1, after which the hard disk will be ready for use by LVM.

Managing LVM After you’ve created logical volumes in this manner, you’ll manage them with their associated commands. For example, you can add new devices to the volume group after the device has been installed in your computer. If you have created a physical volume /dev/sde, use vgextend somegroup /dev/sde to add the /dev/sde device to the somegroup volume group. As long as the physical volume media is not in use, you can remove it from the volume group as well by using the vgreduce command. For example, vgreduce somegroup /dev/sde would remove sde from the volume group again. Be aware that you risk losing data if you issue this command on a disk that currently is in use. Another important task is to monitor the status of the volume group or the volume. Use the vgdisplay somegroup command to display the

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

properties of the volume group, and if you want to show properties of a logical volume, use lvdisplay somevol instead. It speaks for itself that somegroup is the name of the volume group you want to monitor the properties of, and somevol is the name of the volume you want to inspect. Listing 4-11 shows what the result of the vgdisplay command looks like. Listing 4-11. Use the vgdisplay Command to Show the Properties of a Volume Group root@ubuntu:~# vgdisplay --- Volume group --VG Name ubuntu System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 279.23 GB PE Size 4.00 MB Total PE 71484 Alloc PE / Size 71484 / 279.23 GB Free PE / Size 0 / 0 VG UUID kl304T-T8Qx-gQiE-b8So-1LWo-SfiP-ctS6z1 Depending on what you’re looking for, vgdisplay gives useful information. For example, if you were thinking about creating another logical volume, you would first have to be sure that some physical extents (the building blocks of both vg and lv) are available. As you can see in the example in Listing 4-11, this is not the case, so you have to add a physical volume first before you can proceed.

Using Advanced LVM Features You can easily resize existing volumes in an LVM environment. It’s also possible to create a snapshot of a volume. Let’s explore how to do that. Resizing Logical Volumes When resizing logical volumes, you should be aware that the procedure always involves two steps: you need to resize both the volume as well as the file system that is used on the volume. Of all the different file systems, ReiserFS and Ext3 support resizing with the fewest problems. The following procedure details how the volume is first brought offline and then the file system that sits on the volume is resized. It is presumed that the volume you want to shrink is called data and that it uses an Ext3 file system. It is mounted on the directory /data.

111

112

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Caution Online resizing of a file system is possible in some cases. For example, the command ext2online makes it possible to resize a live file system. However, because resizing file systems is very

labor intensive, I wouldn’t recommend doing it this way. There’s always a risk that it won’t work out simply because all of the work that has to be done. So, to stay on the safe side, umount your volume before resizing it.

1. Use umount /data to unmount the volume from the directory /data. 2. Before shrinking the volume itself, you must shrink the file system used on it. Use resize2fs /dev/system/data 2G to make it a 2 GB file system. 3. Now you have to resize the volume itself: use lvreduce -L -1G /dev/system/data. 4. Finally, you can mount the volume again. Use mount /dev/system/data /data. 5. Use the df -h command to show the current size of the file system. It should be a gigabyte smaller than it was before. In this procedure, you learned how to shrink a volume, and of course you can also increase its size. When increasing a volume, you just have to invert the order of the steps. First, you need to extend the size of the volume, and then the size of the file system can be increased as well. After dismounting the volume, this is a two-step procedure: 1. Use lvextend -L+10G /dev/system/data to add 10 GB of available disk space from the volume group to the volume. 2. Next, use resize_reiserfs -f /dev/system/data. This command will automatically increase the Reiser file system that is sitting in the volume to the maximum amount of available disk space. You now know how to resize a volume with a Reiser file system in it. Of course, you can resize Ext3 and Ext2 as well. To increase the size of an Ext3 file system, you would use resize2fs -f /dev/system/data. Creating LVM Snapshots One of the best features of LVM is the possibility to make snapshots. A snapshot is a new block device that functions as a complete copy of the original volume. This works without a complete copy being made; only changes are written to the snapshot, and therefore a snapshot can be very efficient in its use of disk space. A snapshot captures the file system metadata that is used to provide an overview of all existing files on a device and the blocks on a device that are occupied or free. So initially, the snapshot records only administrative information that is used to tell what file is at what location. Because of the close relation between the original device and its snapshot, all reads to the snapshot device are redirected to the original device. When writing anything to the original device, a backup of the old data is written to the snapshot device. Therefore, the snapshot volume will contain the original status, whereas the original volume will always include the changed status. The advantage of this technique is that

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

it requires a very limited amount of disk space. For example, if you create a snapshot of a 100 GB volume to exist only for an hour, it must be large enough to keep the file system’s metadata as well as all data that is changed within that hour. In most cases, this means that a 5 GB snapshot is more than enough. If, however, your snapshot has to exist for a longer period, the amount of disk space that is used by the snapshot will be larger. Using snapshot technology can also be very convenient for making backups of volumes that cannot be closed. Imagine, for example, the data volume of a mail server: you cannot just take the mail server down for a couple of hours to make a backup. The solution then is to make a snapshot of the original volume, back up the snapshot volume (which contains the frozen state of the logical volume at the moment the snapshot was made) and, when the backup is finished, remove the snapshot again. This is even something that you can put in a shell script and configure with cron so that it runs automatically every night. The procedure described next shows you how to create a snapshot of an existing LVM volume. 1. In the first step, you are using the lvcreate command to make a snapshot volume for the original volume /dev/system/data. The snapshot gets the name databackup. Because the original volume is in the system volume group, the snapshot will be created from that group as well. Do this by using the command lvcreate -L500M -s -n databackup /dev/system/data. Here, the option -L500M makes the snapshot 500 MB, -s makes it a snapshot volume, and -n uses the name databackup. Finally, /dev/system/ data refers to the original volume that will be captured in the snapshot.

■Tip Problems creating the snapshot volume? Make sure that the kernel module

dm_snapshot is loaded! Check this with the lsmod command; if it isn’t loaded, load it manually with modprobe dm_snapshot.

2. If next you want to create a backup of the volume, first mount it. Do this the same way that you would mount any other volume, such as with mount /dev/system/databackup /somewhere. 3. To create a backup from the snapshot volume, use your regular backup (or tar, for example). To write a backup to a rewindable tape device, you would use tar -cvf /dev/rmt0 /somewhere. 4. Finished making the backup? Then you can remove the snapshot with lvremove /dev/system/databackup. Of course, this works only after you have unmounted the snapshot device.

Doing Magic on Your File Systems with dd In your life as a system administrator, it is often necessary to copy data around. If it’s ordinary data, an ordinary command like cp works well enough. But if the data is not ordinary, cp just isn’t powerful enough. We’ll now discuss some of the options that the dd command has to offer.

113

114

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

■Caution If the file system metadata contains important information, like a UUID that makes it possible for you to uniquely identify a file system, dd can cause some unpredicted results. If you are using dd to copy the complete file system, you will copy information like the UUID as well, so you can’t differentiate the old file system from the new one.

Speaking in generic terms, the nice thing about the dd command is that it doesn’t just copy files; it can copy blocks as well. As a simple example, I’ll show you how to clone your complete hard drive. Assuming that /dev/sda is the drive that you want to clone, and /dev/sdb is an empty drive that can be used as the target, the dd command is rather easy: dd if=/dev/sda of=/dev/sdb. In this example, dd is used with two parameters only: if is used to specify an input file, and of is used to specify the output file (both of which are device files in this case). Next, wait until the command is finished, and you will end up with an exact copy of the original hard drive. In this example, the contents of one device were copied to another device. A slight variation is the way that dd is used to clone a DVD or CD-ROM and write it to an ISO file. To do that, in case your optical drive can be accessed via /dev/cdrom, you can clone the optical disk using dd if=/dev/cdrom of=/tmp/cdrom.iso. And, of course, you can mount that ISO file as well using mount -o loop /tmp/cdrom.iso /mnt. Next, you can access the files in the ISO file from the directory where the ISO is mounted. So far we have used dd only to do things that can be done with other utilities as well. It becomes really interesting if we go beyond that. What do you think of the following example, in which a backup of the master boot record (MBR) is created? Just make sure that the first 512 bytes of your hard drive, which contains the MBR, is copied to some file, as in dd if=/dev/sda of=/boot/mbr_backup bs=512 count=1. In this example, two new parameters are used. First, the parameter bs=512 specifies that the block should be 512 bytes. Next, the parameter count=1 indicates that only one such block has to be copied. Without this parameter, you would copy your entire hard drive, which I don’t recommend. The backup copy of your MBR may be useful if some day you can’t boot your server anymore because of a problem in the MBR. If this happens, just boot from a rescue disk and use the command dd if=/boot/mbr_backup of=/dev/sda bs=446 count=1. As you notice, in this restore command, only 446 bytes are written back because you may have changed the partition table since you created the backup. By writing back only the first 446 bytes of your backup file, you don’t overwrite the original partition table which is between bytes 447 and 511. Now I’ll show you how to extend your swap space by adding a swap file. This is useful if you get an alert in the middle of the night that your server is almost running completely out of memory because of a memory leak you hadn’t discovered so far. All you have to do is to create an empty file and specify that it should be added to the swap space. Creating this empty file is an excellent task for the dd command. In the following command, you are using dd to create a file that is filled with zeros completely by using the /dev/zero device: dd if=/dev/zero of=/swapfile bs=1024 count=1000000. This would write a file of 1 GB that can be added to the swap space using mkswap /swapfile and swapon /swapfile. In the next example of the marvelous things you can do with dd, let’s use it to recover the superblock on an Ext2 or Ext3 file system. To access a file system, you need the superblock, which is a 1 KB block that contains all metadata about the file system. It normally is the

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

second 1 KB block on an Ext3 file system. In Listing 4-12, you can see a part of the contents of the superblock as displayed with the debugfs utility. Listing 4-12. Partial Contents of the Superblock Filesystem volume name:

Last mounted on:

Filesystem UUID: 09979101-96e0-4533-a7f3-0a2db9b07a03 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr filetype needs_recovery sparse_super large_file Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 5248992 Block count: 10486428 Reserved block count: 524321 Free blocks: 3888202 Free inodes: 4825213 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16352 Inode blocks per group: 511 If the superblock isn’t accessible anymore because of an error, you have a serious challenge. Fortunately, some backup copies of the superblock are written on the Ext3 file system by default. Using these backup copies, you can still mount a file system that you may have otherwise considered lost. And, as you can guess, the dd command is an excellent help. The actual position on disk of the first backup of the superblock depends on the size of the file system. On modern large file systems, you will always find it at block 32768. To see if it really works, you can mount from it directly using the mount option -o sb. The issue, however, is that mount expects you to specify the position of the superblock in 1,024 byte blocks, whereas the default block size for a modern Ext3 volume or partition is often 4,096 bytes. (Use dumpe2fs if you want to be sure about that.) Therefore, to tell the mount command where it can find the superblock, you have to multiply the position of the superblock by 4, which in most cases results in a block value 131072. If, for example, your /dev/sda5 file system should have a problem, you can try mounting it with the command mount -o sb=131072 /dev/hda5 / somewhere. Did the file system mount successfully? If so, the problem really was in the superblock. So let’s fix that problem by copying the backup superblock back to the location of the old superblock. You can do this using dd if=/dev/hda5 of=/dev/hda5 bs=1024 skip=131072 count=1 seek=1. Once finished, your file system is accessible again, just as it was before.

115

116

CHAPTER 4 ■ PERFORMING FILE SYSTEM MANAGEMENT TASKS

Summary In this chapter, you learned about the most important file system management tasks. In the first part of the chapter, you read how to mount and unmount file systems. You also learned how links can make your life easier. The second part of this chapter concerned the organization of a file system. You learned how to use partitions or logical volumes to set up a file system, and you read how to use the dd command to perform some advanced file system management tasks. In Chapter 5, you’ll find out how to secure your server with user and group accounts, permissions, sudo, and many more.

CHAPTER

5

Configuring Your Server for Security N

o matter what you want to use your server for, it’ll be useless if it isn’t secure. In this chapter, you’ll learn about the different security-related items that you’ll encounter when setting up your server. First I’ll talk about the configuration of users and groups because most of the security you’ll be using on your server will be bound to users and groups. Next, I’ll cover the Linux permissions that you can use to restrict access to your server’s file system. Following that, I’ll discuss some security mechanisms that aren’t so obvious, like the sudo mechanism and the system of pluggable authentication modules (PAMs).

■Note In this section you’ll learn all about securing services on your server, but also know that none of your software skills will do you any good unless your hardware—and I mean the server itself—is physically secured. So, before you start securing your server, make sure that it is locked in a restricted-access room.

Setting Up User Accounts You have two approaches when creating users from a command-line environment: you can use the useradd command, or you can add users to the relevant configuration files by manually editing these files. Although this second approach—editing the configuration files—may be useful in an environment in which users are added from a custom-made shell script, it generally is not recommended. The reason for this is probably obvious: an error in the main user configuration files might make it impossible for every user to log in to the server. In this section, I’ll discuss how to manage users from the command line using useradd and other related commands such as usermod and userdel. You can edit related configuration files to make creating users easier.

Commands for User Management If you want to add users from the command line, useradd is just the ticket. And the other commands for user management are just as convenient: 117

118

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

• useradd: Adds users to the local authentication system • usermod: Modifies properties for users • userdel: Deletes users properly from a system • passwd: Modifies passwords for users Using useradd is simple. In its easiest form, it just takes the name of a user as its argument; thus, sudo useradd zeina creates a user called “zeina” to the system. However, you should also use the -m option because if you don’t, that user will be without a home directory. In most cases, a user should have a home directory because it allows that person to store files somewhere. Unfortunately, if you create a user without a home directory, there’s really no easy way to correct this problem later (but see the following tip for the not-so-easy way).

■Tip Did you forget to create a home directory for user zeina and want to create one now? First, use mkdir /home/zeina to make the directory itself. Then use cd /etc/skel to activate the directory that contains all files that normally need to be present in a user’s home directory. Use tar cv . | tar xvC /home/zeina to copy all files, including hidden files from this directory to the user’s home directory. Next, use chown -R zeina:users /home/zeina to set proper file ownership for all these files. You’ve now created a home directory, but wouldn’t it have been easier just to use -m?

You have a few options with the useradd command. If an option isn’t specified, useradd will read its configuration file in /etc/default/useradd, where it finds some default values such as what groups the user should become a member of and where to create the user’s home directory. But let’s take a look at the most important options, which are listed next. (For a complete list of available options, use man useradd or useradd --help for a summary.) • -c comment: Allows you to enter a comment field to the user. Information set this way can be requested with the finger command, and this comment field typically is used for the user’s name. • -e date: Sets the expiration date for the user. Use this option to automatically disable the user’s account on the specified date. This can be entered in the YYYY-MM-DD format or as the number of days since January 1, 1970. You’ll probably prefer to specify the date. • -G groups: Makes the user a member of some additional groups. By default, the user becomes a member of only those groups listed in /etc/default/useradd. • -g gid: Sets the primary group of a user (see the section called “Group Membership” later in this chapter for more details). • -m: Creates a home directory automatically. When setting up user accounts, the user is added to two configuration files: /etc/passwd and /etc/shadow. The /etc/passwd file contains generic user-related information, such as the groups the user belongs to and the unique ID assigned to the user. The /etc/shadow file

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

contains password-related information about the user. In the following subsections, you’ll find information about the properties used in these files.

UID The user ID (UID) is another major piece of information when creating a user. For your server, this is the only way to identify a user; user names are just a convenience for humans (who can’t quite handle numbers as well as a computer does). In general, all users need a unique UID. Ubuntu Server starts generating UIDs for local users at 1000, and a total of 16 bits is available for creating UIDs. This means that the highest available UID is 65535, so that’s also the maximum number of local users that your server will support. If you exceed this limit, you’ll need a directory server such as OpenLDAP. Typically, UIDs below 500 are reserved for system accounts that are needed to start services. The UID 0 is also a special one: the user with it has complete administrative permissions to the server. UID 0 is typically reserved for the user root. That said, you may want to give the same ID to more than one user in one situation: to create a backup root user. If you want to do this with useradd, use the options -o and -u 0. For example, to make user stacey a backup root user, use useradd -o -u 0 stacey.

■Tip Want to use some specific settings for all users that you are creating on your server? If so, you might be interested in the /etc/default/useradd file, which contains default settings that are used all the time when using useradd. Check the other files in this directory as well; many commands read configuration files from it.

Group Membership In any Linux environment, a user can be a member of two different kinds of groups. First, there’s the primary group, which every user has. (If a user doesn’t have a primary group setting, he won’t be able to log in.) The primary group is the group that is specified in the /etc/passwd file. By default, on an Ubuntu Linux system, all users get their own private groups as their primary groups, and this private group has the same name as the user. A user can be a member of more than just the primary group and will automatically inherit the rights granted to these other groups. The most important difference between a primary group and other groups is that the primary group will automatically become group owner of any new file that a user creates. Because every user has his or her own private group, this won’t be a great challenge for your server’s security settings (because the user is the only member). I’ll discuss file permissions and ownership in detail later in this chapter, but I’ll provide an example just to give you an idea of how it works. Imagine that user zeina is a member of the private group zeina and also of the group sales. Now user zeina wants to create a file to which only members of the group sales have access. If she does nothing to change her primary group and creates a file, the default group zeina will become group owner of the file, and all users who are members of this group (in other words, just zeina) will have access to the file. One solution is to use the newgrp command to set the primary group to sales on a temporary basis. If user zeina creates the file after using newgrp sales, the group sales will be owner

119

120

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

of that file and all other files that the user creates until she uses newgrp zeina to switch the primary group setting back to users. This is just one way of using groups, and I discuss other methods later in the chapter. As you can see, group membership in a stand-alone Linux file system environment isn’t that sophisticated. The method sounds primitive, but it hardly ever causes problems because permissions are just set at another level, such as when the user is accessing the server through a Samba share. (See Chapter 10 for more on that.) You now know some more about the relation between the primary group and the other groups that a user belongs to. In the sections about group management later in this chapter, you’ll learn how to apply this knowledge.

Shell Any user who needs to log in to your server needs a shell. (Conversely, users who don’t need to work on your server directly generally don’t need a shell; they just need a connection.) The shell will enable the user’s commands to be interpreted. The default shell in Ubuntu is /bin/ bash, a shell that offers many features. However, you should know that not every user needs a shell. A user with a shell is allowed to log in locally to your system and access any files and directories stored on that system. If you’re using your system as a mail server (and so all that your users need is to access their mail boxes with the POP protocol), it makes no sense at all to give them a login shell. Therefore, you could choose to specify an alternative command to be used as the shell. For example, use /bin/false if you don’t want to allow the user any local interaction with your system. Any other command will do as well. If, for example, you want the Midnight Commander (a clone of the once very popular Norton Commander) to be started automatically when a user logs in to your system, make sure that /usr/bin/mc is specified as the shell for that user.

■Tip Make sure that you include the complete path to the command you want to execute as the shell environment for a user. If you don’t know the complete path for your favorite command, use the which command. For example, which mc shows the exact location of the program file you’re looking for. Not installed yet? Use sudo apt-get install mc to install it now.

Managing Passwords If your user really needs to do something on your system, she needs a password. By default, login for the users you create is denied, and no password is supplied. Basically, your freshly created user does not have any permissions on your server. However, the simple passwd command will let her get to work. If the user uses the command to change her password, she will be prompted for the old password and then the new one. It’s also possible for the root user to change passwords as well. Only root can add the name of a user, for whom root wants to change a password, as an argument to the passwd command. For example, root can use the command passwd linda to change the password for user linda, which is always useful in case of forgotten user passwords.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

The passwd command can be used in three generic ways. First, you can use it for password maintenance (such as changing a password, as you have just seen). Second, it can also be used to set an expiration date for the password. Third, the passwd command can be used for account maintenance. For example, an administrator can use it to lock a user’s account so that login is temporarily disabled. In the next subsection, you’ll learn more about password management.

Performing Account Maintenance with passwd In an environment in which many users use the same server, it’s crucial that you perform some basic account maintenance. These tasks include locking accounts when they are unneeded for a longer time, unlocking an account, and reporting password status. Also, an administrator can force a user to change his password after he logs in for the first time. To perform these tasks, the passwd command has the following options: • -l: Enables an administrator to lock an account (for example, passwd -l jeroen locks the account for user jeroen) • -u: Unlocks a previously locked account • -S: Reports the status of the password for a given account • -e: Forces the user to change his or her password upon next login

Managing Password Expiration Although not many people are aware of this feature, it allows you to manage the maximum number of days that a user can use the same password. The passwd command has four options to manage expirations: • -n min: This option is applied to set the minimum number of days that a user must use his password. If this option is not used, the user can change his password any time he wants. This option can be useful if you want to apply a password policy that forces users to change their passwords on a regular basis. Using a minimum number of days prevents users from changing a password and then immediately changing it back to the old password. • -x max: With this option, you can set the maximum number of days that the user can use his password without changing it. • -c warn: Use this option to send a warning to the user when his password is about to expire. The argument of this option specifies how many days the user is warned before his password expires. • -i inact: Use this option to make an account expire automatically if it hasn’t been used for a given period. The argument of this option specifies the exact duration in days of this period.

121

122

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

■Caution By default, a password can be used for 99,999 days. So, if you do nothing, a user may use his password for 273 years without changing it. If you don’t want that, make sure you use the –x option.

Modifying and Deleting User Accounts If you know how to create a user, modifying an existing user account is no big deal, and the usermod command has many options that are exactly the same as those used with useradd. For example, use usermod -g 101 linda to set the new primary group of user linda to the group with the unique ID 101. For a complete overview of the options that usermod shares with useradd, consult the man page of usermod, but some of the useful and unique options are listed here: • -a, --append: Adds the user to some new groups. This option is followed by the group ID of the groups you want to add the user to. • -L, --lock: Disables the account temporarily. • -U, --unlock: Unlocks an account. Another command that you’ll occasionally need is userdel, which you’ll use to delete user accounts from your server. Basically, userdel is a very simple command: userdel lynette deletes user lynette from your system. However, if used this way, userdel leaves the home directory of your user untouched. This may be desired (such as to ensure that your company still has access to the work a user has done), but you may just as well wish to delete the user’s home directory. For this purpose, you can use the option -r; for example, userdel -r lynette deletes the home directory of user lynette. However, if the home directory of user lynette contains files that are not owned by user lynette, userdel can’t remove the home directory. In this case, use the option -f, which removes every file from the home directory, even those not owned by the given user. So, to make sure that user lynette and all the files in her home directory are removed, use userdel -rf lynette. You now know how to remove a user along with all the files in his home directory. But what about other files the user may have created in other directories on your system? The userdel command won’t automatically find and remove these. In such a case, the find command is very useful. You can use find to search for and remove all files owned by a given user. To locate and remove all files on your system that are created by user lynette, you can use find / -user "lynette" -exec rm {} \;. However, this may lead to problems on your server in some circumstances. Let’s say lynette was a very active user of the group sales and created many important files in the directory /home/sales that are used by other members of the group. So instead of immediately removing the files, it’d be better to copy them to a safe place instead. If no one has complained after a couple of months, you can remove them safely. To move all files owned by user lynette to a directory called /trash/lynette (that you must create beforehand), use find / -user lynette -exec mv {} /trash/lynette \;.

Behind the Commands: Configuration Files In the previous section, you learned about the commands to manage users from a console environment. All these commands put the user-related information into what are known as

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

configuration files, and a configuration file is also used for default settings that are applied when managing the user environment. The aim of this section is to give you some insight into the following configuration files: • /etc/passwd • /etc/shadow • /etc/login.defs

/etc/passwd The first and probably most important of all user-related configuration files is /etc/passwd, which is the primary database for user information: everything except the user password is stored in this file. Listing 5-1 should give you an impression of what the fields in this file look like. Listing 5-1. Contents of the User Database file /etc/passwd root@RNA:~# cat /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bin/sh list:x:38:38:Mailing List Manager:/var/list:/bin/sh irc:x:39:39:ircd:/var/run/ircd:/bin/sh gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh nobody:x:65534:65534:nobody:/nonexistent:/bin/sh dhcp:x:100:101::/nonexistent:/bin/false syslog:x:101:102::/home/syslog:/bin/false klog:x:102:103::/home/klog:/bin/false mysql:x:103:106:MySQL Server,,,:/var/lib/mysql:/bin/false bind:x:104:109::/var/cache/bind:/bin/false sander:x:1000:1000:sander,,,:/home/sander:/bin/bash messagebus:x:105:112::/var/run/dbus:/bin/false haldaemon:x:106:113:Hardware abstraction layer,,,:/home/haldaemon:/bin/false gdm:x:107:115:Gnome Display Manager:/var/lib/gdm:/bin/false sshd:x:108:65534::/var/run/sshd:/usr/sbin/nologin linda:x:1001:1001::/home/linda:/bin/sh zeina:x:1002:1002::/home/zeina:/bin/sh

123

124

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

You can see that /etc/passwd uses different fields, and they all are separated with a colon. Here’s a short explanation of these fields: • Loginname: This is the first field and it stores the user’s login name. In older UNIX versions, the field was limited to eight characters. Fortunately, Ubuntu Server does not have this limitation. • Password: In the old days of UNIX, this file stored the encrypted passwords. The only problem was that everyone—including an intruder—was allowed to read the /etc/ passwd file. This poses an obvious security risk, so passwords are now stored in the configuration file /etc/shadow, which is discussed in the next section. The “x” in the password field denotes the use of shadow passwords. • UID: As you already learned, every user has a unique user ID. Ubuntu Server starts numbering local user IDs at 1000 and typically the highest number that should be used is 65535. • GID: As discussed in the previous section, every user has a primary group, and its group ID (GID) is listed here. This is the numeric ID of the group that the user uses as his primary group. For ordinary users, the GID defaults to 100 (which belongs to the group users). • GECOS: The General Electric Comprehensive Operating System (GECOS) field is used to include some comment to make it easier for the administrator to identify the user. However, the GECOS field is optional, and it’s often not used at all. • Home directory: This is a reference to the directory that serves as the user’s home directory; it is typically the location in which a user stores files. Note that it is only a reference and has nothing to do with the real directory; just because you see a directory listed here doesn’t mean that it actually exists. • Shell: The last field in /etc/passwd refers to the program that should be started automatically when a user logs in. Most often, it’s /bin/bash, but, as discussed in the preceding section, every binary program can be referred to here as long as the complete pathname is used. As an administrator, you can manually edit /etc/passwd and the related /etc/shadow. If you intend to do this, however, don’t use any editor; use vipw instead. This tailored version of the Vi editor is specifically designed for editing these critical files. Any error can have serious consequences, such as no one being able to log in. Therefore, if you make manual changes to any of these files, you should check their integrity. Besides vipw, another way to do this is to use the pwck command, which you can run without any options to see whether there are any problems you need to fix.

/etc/shadow Encrypted user passwords are stored in the /etc/shadow file. The file also stores information about password expiration. Listing 5-2 shows an example of its contents.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

Listing 5-2. Example Contents of the /etc/shadow File root:$1$15CyWuRM$g72U2o58j67LUW1oPtDS7/:13669:0:99999:7::: daemon:*:13669:0:99999:7::: bin:*:13669:0:99999:7::: sys:*:13669:0:99999:7::: sync:*:13669:0:99999:7::: games:*:13669:0:99999:7::: man:*:13669:0:99999:7::: lp:*:13669:0:99999:7::: mail:*:13669:0:99999:7::: news:*:13669:0:99999:7::: uucp:*:13669:0:99999:7::: proxy:*:13669:0:99999:7::: www-data:*:13669:0:99999:7::: backup:*:13669:0:99999:7::: list:*:13669:0:99999:7::: irc:*:13669:0:99999:7::: gnats:*:13669:0:99999:7::: nobody:*:13669:0:99999:7::: dhcp:!:13669:0:99999:7::: syslog:!:13669:0:99999:7::: klog:!:13669:0:99999:7::: mysql:!:13669:0:99999:7::: bind:!:13669:0:99999:7::: sander:$1$Qqn0p2NN$L7W9uL3mweqBa2ggrBhTB0:13669:0:99999:7::: messagebus:!:13669:0:99999:7::: haldaemon:!:13669:0:99999:7::: gdm:!:13669:0:99999:7::: sshd:!:13669:0:99999:7::: linda:!:13671:0:99999:7::: zeina:!:13722:0:99999:7::: Just as in /etc/passwd, the lines in /etc/shadow are divided into several fields as well, but only the first two fields matter for the typical administrator. The first field stores the name of the user, and the second field stores the encrypted password. Note that, in the encrypted password field, the ! and * characters can be used as well. The ! character denotes that login is currently disabled, and * denotes a system account that can be used to start services but that is not allowed for interactive shell login. Also note that an encrypted password is stored here by default, but it’s perfectly possible to store an unencrypted password as well. The /etc/shadow file uses the following fields: • Login name • Encrypted password • Days between January 1, 1970 and the date when the password was last changed • Days before password may be changed (this is the minimum amount of time that a user must use the same password)

125

126

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

• Days after which password must be changed (this is the maximum amount of time that a user may use the same password) • Days before password expiration that user is warned • Days after password expiration that account is disabled (if this happens, administrator intervention is required to unlock the password) • Days between January 1, 1970 and the date when the account was disabled • Reserved field (this field is currently not used)

/etc/login.defs The /etc/login.defs file is a configuration file that relates to the user environment, but is used only in the background. This file defines some generic settings that determine all kinds of things relating to user login. The login.defs file is a readable configuration file that contains variables. The variable relates to logging in or to the way in which certain commands are used. This file must exist on every system because you would otherwise experience unexpected behavior. The following list contains some of the more interesting variables that can be used in the login.defs file (for a complete overview, consult man 5 login.defs): • DEFAULT_HOME: By default, a user will be allowed to log in, even if his home directory does not exist. If you don’t want that, change this parameter’s default value of 1 to the Boolean value 0. • ENV_PATH: This variable contains the default search path that’s applied for all users who do not have UID 0. • ENV_ROOTPATH: This variable works in the same manner as ENV_PATH, but for root. • FAIL_DELAY: After a login failure, it will take a few seconds before a new login prompt is generated. This variable, set to 3 by default, specifies how many seconds it takes. • GID_MAX and GID_MIN: Specify the minimum and maximum GID used by the groupadd command (see “Commands for Group Management” in the next section). • LASTLOG_ENAB: If enabled by setting the Boolean value to 1, LASTLOG_ENAB specifies that all successful logins must be logged to the file /var/log/lastlog. This works only if the lastlog file also exists. (If it doesn’t, create it by using touch /var/log/lastlog.) • PASS_MIN_LEN: This is the minimum number of characters that must be used for new passwords. • UID_MAX and UID_MIN: These are the minimum and maximum UIDs to be used when adding users with the useradd command.

Creating Groups As you’ve already learned, all users require group membership. You’ve read about the differences between the primary group and the other groups, so let’s have a look at how to create

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

these groups. We’ll discuss the commands that you can run from the shell and the related configuration files.

Commands for Group Management Basically, you manage the groups in your environment with three commands: groupadd, groupdel, and groupmod. So, as you can see, group management follows the same patterns as user management. The basic structure for the groupadd command is simple: groupadd somegroup, where somegroup is the name of the group you want to create. Also, the options are largely self-explanatory: it probably doesn’t surprise you that the option -g gid can be used to specify the unique GID number you want to use for this group.

Behind the Commands: /etc/group When a group is created with groupadd, the information entered needs to be stored somewhere, and that’s the /etc/group file. As seen in Listing 5-3, this is a rather simple file that has just two fields for each group definition. Listing 5-3. Content of /etc/group plugdev:x:46:sander,haldaemon staff:x:50: games:x:60: users:x:100: nogroup:x:65534: dhcp:x:101: syslog:x:102: klog:x:103: scanner:x:104:sander nvram:x:105: mysql:x:106: crontab:x:107: ssh:x:108: bind:x:109: sander:x:1000: lpadmin:x:110:sander admin:x:111:sander messagebus:x:112: haldaemon:x:113: powerdev:x:114:haldaemon gdm:x:115: linda:x:1001: zeina:x:1002: The first field in /etc/group is reserved for the name of the group. The second field stores the password for the group (a ! character signifies that no password is allowed for this group). You can see that most groups have an “x” in the password field, and this refers to the /etc/gshadow file, in which you can store encrypted group passwords. However, this feature

127

128

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

isn’t used very often because it is very rare to work with group passwords. The third field of /etc/group provides a unique GID, and the last field presents the names of the members of the group. These names are required only for users for whom this is not the primary group; primary group membership itself is managed from the /etc/passwd configuration file. However, if you want to make sure that a user is added to an additional group, you have to do it here.

Using Group Passwords Although not many administrators use group passwords, what can they be used for? In all cases, a user has a primary group. When the user creates a file, the primary group is assigned as the group owner automatically. If a user wants to create files that have a group owner different from the primary group, the user can use the newgrp command. For example, newgrp sales would set the primary group of a user to the group sales. Using this command would work without any questions if the user is a member of the group sales. If the user is not a member of that group, however, the shell would prompt the user to enter a password. This password is the password that needs to be assigned to that group. You can change it using the gpasswd command.

Managing the User’s Shell Environment As a system administrator of a server that users access directly, you have to do more than just create users and make them members of the appropriate groups. You also have to give them login environments. Without going into detail about specific shell commands, this section provides an overview of what is needed for that. I’ll first explain about the files that can be used as login scripts for the user, and next you’ll learn about files that are used to display messages for users logging in to your system.

■Note The task described as follows makes sense on a server in which users log in directly. If your server is used as a mail server or file server, and users don’t log in directly, it doesn’t make sense to tune it by using these commands.

Creating Shell Login Scripts When a user logs in to a system, the /etc/profile configuration file is used. This generic shell script (which can be considered a login script) defines environment settings for users. Also, commands can be included that need to be issued when the user first logs in to a server. The /etc/profile file is a generic file processed by all users logging in to the system. It also has a user-specific version (~/.profile) that can be created in the home directory of the user. The user-specific ~/.profile of the shell login script is executed last, so if there is a conflict in settings between the two files, the settings that are user-specific will always be used. In general, it isn’t a good idea to give a login file to too many individual users; instead, work it all out in /etc/profile. This makes configuring settings for your users as easy as possible.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

The /etc/profile file is not the only file that can be processed when starting a shell. If a user starts a subshell from a current environment, such as by executing a command or by using the command /bin/sh again, the administrator may choose to define additional settings for that. The name of this configuration file is /etc/bashrc, and it also has a user-specific version, ~/.bashrc (the tilde in the file name means that the file is located in the user’s home directory).

Displaying Messages to Users Logging In As an administrator, it’s sometimes necessary to display messages to users logging in to your server. Two files can be used for this: /etc/issue and /etc/motd. The first, /etc/issue, is a text file whose content is displayed to users before they log in. To process this file, the /sbin/getty program, which is responsible for creating login terminals, reads it and displays the content. You may, for example, use the file to display a message instructing users how to log in to your system, or include a message if login has been disabled on a temporary basis. Related to this file is /etc/motd, which can be used to display messages to users after they have logged in. Typically, this file can be used to display messages related to day-to-day system maintenance.

Configuring Permissions At first glance, it seems easy to manage permissions in a Linux environment: instead of the many permissions some other operating systems work with, Linux has just three. However, upon closer examination, you’ll see that the system that was invented somewhere in the 1970s is only the foundation for a system that can be pretty complex. The following subsections are overviews of all subjects relevant to permission management.

Read, Write, and Execute: The Three Basic Linux Permissions The three elementary permissions—read (r), write (w), and execute (x)—are the foundation to working with permissions in a Linux system. The use of these permissions is not hard to understand: read allows a user to read the file or the contents of the directory the permission is applied to, write allows the user to change an existing file if applied to a file and to create or remove files in a directory it is applied to, and execute is used to allow a file to execute executable code. If applied to a directory, it allows a user to access that directory with a command like cd. Therefore, the execute permission is applied as a default permission to all directories on a Linux system. Table 5-1 summarizes the workings of these three basic permissions. Table 5-1. Overview of Linux Basic Permissions

Permission

Applied to Files

Applied to Directories

read

Read contents of a file

See files existing in a directory by using the ls command

write

Modify existing files and their properties

Create or delete files from a directory

execute

Execute files that contain executable code

Activate a subdirectory with the cd command

129

130

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

Permissions and the Concept of Ownership To determine the permissions that a given user has on a file or directory, Linux works with the concept of ownership. Ownership is set on every file and on every directory, so when working with the three basic Linux permissions, there’s no such thing as “inheritance,” as there is with some other operating systems.

■Note In fact, on Linux, inheritance can be applied when working with Set Group ID (SGID) permissions and access control lists (ACLs), both of which I’ll cover later in this chapter.

Linux works with three entities that can be set as the owner of a file or directory. First is the user who is owner. By default, the user who creates a file becomes the owner of that file (but an administrator can change ownership later using the chown command). Next is the group owner. By default, the primary group of the user who is owner of a file will also become the group owner of that file.

■Note When in this section I refer to a file, I also refer to a directory, unless stated otherwise. From a filesystem point of view, a directory is just a special kind of file.

Last is the others entity. Typically, this entity refers to the rest of the world; permissions granted to the others entity apply to everyone who is able to access the given file. The ownership of files and the permissions that are set for the three different file owners can be reviewed with the ls -l command, as seen in the following line: -rw-rw-r--

1

linda users

145906 2006-03-08 09:21 somefile

In this output, the name of the file is somefile. The first character refers to the type of file. In this case, it’s just an ordinary file, therefore the - character is displayed. The next three groups of three characters refer to the permissions applied to the user, group, and others, respectively. As you can see, the user has read and write permissions, the group has read as well and write permissions, and all others just have read permission. The next important pieces of data in the preceding line are the names linda and users; they refer to user linda (who is owner) and the group users (which is group owner). Note that in the basic Linux permission scheme, just one user and just one group can be assigned as owner of a file. If you want more, you need to use ACLs. With regards to Linux rights, ownership is the only thing that really matters. An example shows why this is important; imagine the home directory of user linda. Typically, the permissions on a home directory are set, as in the following output line of the ls command: -rwxr-xr-x

1 linda users 1024

2006-03-08 09:28 linda

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

■Note In the examples shown here, the users are all members of the group user, which has GID 100. Although this is not default behavior on Ubuntu, it can be useful if you want to help users share information. If you want all new users created with useradd to automatically become members of this group 100 (which is actually the GID for the group user), make sure that you have the line GROUP=100 in the /etc/default/ useradd file.

Ownership is naturally very important when determining what a user can do to a file. Imagine that a file is created by the user root in the user linda home directory and that the permissions on this file are set as follows: -r--r-----

1

root root 1 537

2006-03-08 10:15 rootsfile

The big question is what user linda can do to this file. The answer is simple, but there is a caveat. Because user linda is not the owner of the file and is not a member of the group that owns the file, she has no permissions at all to this file. The fact that the file is in her home directory doesn’t mean much because Linux doesn’t support inheritance of permissions by default. However, user linda has the write permissions in her home directory, so she can remove the file from her home directory. This is not inheritance; write permissions in a directory apply to the things that a user can do to files in that directory. What you should remember from this example is that to determine what a user can do to a file, the most important question to ask is “Is the user also the owner of the file?” The fact that a file is in the user’s directory isn’t relevant here; it’s ownership that counts.

Changing File Ownership To change the ownership of a file, use the chown command. The structure of this command is as follows: chown {user|:group} file For example, to make user linda owner of rootsfile, the command chown linda rootsfile must be used. To change the group owner of somefile to the group sales, the chown .sales somefile command is used. Note that for changing group ownership, the chgrp command can be used as an alternative. Therefore, chown .sales somefile does the same thing as chgrp sales somefile. When using chgrp, the name of the group does not need to be preceded by a dot. By default, chown and chgrp apply only to the file or directory on which they are used. However, you can use the commands to work recursively as well: chown -R linda somedir makes user linda owner of all files in somedir and all subdirectories of it.

Group Ownership When working with group ownership, you should be aware of how group ownership is handled. By default, the primary group of the user who creates a new file becomes the group owner of that file. If, however, the user is a member of more than one group, this default setting can be manipulated. When a user issues the newgrp command, he can change the primary group setting on a temporary basis. The following steps show what happens next:

131

132

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

1. Log in as some normal user on your computer. Then use the groups command from a console window to get an overview of all groups that you are currently a member of. The primary group is listed first. If you haven’t modified anything for this user, it will have the same name as your user account. Listing 5-4 is an example of this output. Listing 5-4. The groups Command Always Shows Your Primary Group First sander@RNA:~$ groups sander adm dialout cdrom floppy audio dip video plugdev scanner lpadmin admin 2. From the console window, issue the touch newfile command to create a new file with the name newfile. Then use ls -l newfile to display the ownership information for newfile. You will see that the primary group is set as the owner of the file (see Listing 5-5). Listing 5-5. The User’s Primary Group Is Always Set As Its Owner sander@RNA:~$ ls –l newfile -rw-r--r-- 1 sander sander 0 2007-07-28

10:05 newfile

3. Use su to become root. Then use groupadd to create a new group (for example, use groupadd -g 901 sales to create a group with the name sales and group ID 901). Next, as root, use usermod -g 901 youruser to make youruser (the user you used in step 1) a member of that group. After changing this group information, use exit to close the su session and become the normal user account again. 4. As the normal user, use groups again to get an overview of all groups you are currently a member of. The new group should appear now, probably as the last group in the list. 5. As the normal user, use newgrp yournewgroup to set the primary group to your new group on a temporary basis. You can use the groups command to check this; the new group should now be listed first. You’ll also see that if you create a new file (use touch somenewfile), the new group will be group owner of the new file. This ensures that all users who are members of the same group can do the same thing to this file.

Working with Advanced Linux Permissions Until now, I’ve covered just the three basic Linux permissions. But there are more. To start with, Linux has a set of advanced permissions, and this section describes the working of these permissions. Before diving into detail, the following list provides a short overview of the advanced permissions and the way they’re used: • SUID: If this permission is applied to an executable file (also known as “Set User ID” and “setuid”), the user who executes that file will have the permissions of the owner of the file while he is executing it. You can see that SUID is a very dangerous permission that, if wrongly applied, creates serious back doors on your server. On the other hand, some applications—/usr/bin/passwd, for example—can’t function without it because these applications need the permissions of their owner root to be able to do their job.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

• SGID: This permission is also known as the Set Group ID (also commonly known as “setgid”) permission. It is applied in two ways. First, if applied to executable files, the user who executes the file will get the permissions of the group who is owner of the file upon execution. Next, the permission can be used to set the default group owner of files created in a given directory. If applied to a directory, all files and directories created in this directory, and even in its subdirectories, will get the group owner of the directory as its group owner. Imagine that all members of the group sales normally save the files they create in /data/salesfiles. In that case, you would want all files created in that directory to be owned by the group sales as well. This goal can be accomplished when setting sales as the group owner for salesfiles and next applying the SGID permission bit to this directory. • Sticky bit: If the sticky bit is used on a directory; users can remove files only if one of the following conditions is met: • The user has write permissions on the file. • The file is in a directory of which the user is owner. • The user is owner of the file. The sticky bit permission is especially useful in a shared data directory. Suppose that user linda creates a file in the directory /data/sales. She wouldn’t want her coworkers from the group sales who also have write permissions in that directory to remove her file by accident (normally they’d be able to because they have the write permission on the directory). If the sticky bit is applied, however, other users can remove the file only if one of those listed conditions has been met. Some comments on these permissions may be helpful. First, you should realize the dangers of the SUID and SGID permissions if they are applied improperly. Imagine, for example, that a given application has a security issue that allows users with the right knowledge to access a shell environment, and at the same time the user root owns this application. This would make the user misusing the security issue root and give him permissions on your entire system! So you should be extremely careful when applying SUID or SGID to files. On the other hand, you may notice that some files have SUID set by default. For example, the program file /usr/bin/passwd cannot work without it. This is because a user who changes his password needs to write information to the /etc/shadow file. Only the user root can write data to this file, and normal users cannot even read its contents. The operating system solves this problem by applying the SUID permission to /usr/bin/passwd, which temporarily grants users root permissions to change their passwords.

■Tip Someone using a back door to get access to your server may use SUID on some obscure file to get access the next time as well. As an administrator, you should regularly check your server for any occurrence of the SUID permission on unexpected files. You can do this by running find / -perm +4000, which will display all files that have the SUID permissions set.

133

134

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

The SGID permission has a dangerous side because it gives the user who runs a command that has this permission the same permissions as the group owner of the command. However, the SGID permission can be very useful. You may apply it on directories in which members of some user group need to share data with each other. The advantage of the SGID permission, if it is applied to a directory, is that all files created in that directory will get the same group owner. This allows all members of that group to read the file. Even if the group just has read rights on files that are created in this directory, the SGID permission may create a workable situation by allowing a user who is a member of the group to read the original file. Without the write permission she cannot change its contents, but she can save the file with a new name in the same directory. With the SGID permission applied to the directory, all files created in the complete tree structure under this directory will get the same group as owner, so the file will always be accessible for all members of the group. Thus, all users can work together in a shared group-data directory in an efficient way. However, in the scenario I’ve just described, there is still the problem that one user can remove the files created by another user who is a member of the same group; both have write permissions in the directory, and that’s enough to remove files. This can be prevented by applying the sticky bit as well. When this permission is set, a user can’t remove a file if he has only write permissions to the directory the file is in, without being the owner of the file.

Setting Permissions Okay, that’s enough about how the permissions can be used. It’s time to set them. You’ll use two commands to set and manipulate permissions: the chmod command to initially set permissions and the umask command to set default permissions.

Using chmod to Change Permissions The chmod command is used to set permissions on existing files. The user root or the owner of a file can use this command to change permissions of files or directories. It can be used in either an absolute or a relative mode. When using chmod in a relative way, the entity (user, group, or others) to which permissions are granted is specified, followed by the + (add), ? (remove), or = (set) operator, and then followed by the permissions you want to apply. In the absolute mode, a numeric value is used to grant the permissions. Using chmod in Relative Mode If working in relative mode, the following values have to be used for the permissions that are available: • read: r • write: w • execute: x • SUID: u+s • SGID: g+s • Sticky bit: t

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

The relative mode is useful when you want to add or remove one permission in an easy and convenient way. For example, you can easily make a script file executable by using chmod +x myscript. Because no u, g, or o is used in this command to refer to the entity the permissions are applied for, the file will be made executable for everyone. You can, however, be more specific and just remove the write permission for the other entity by using chmod o-w somefile, for example. As for the special permissions, SUID is set with u+s, and SGID is set with g+s. As the result, you will see the SUID permission at the position of the x for users and the SGID permission at the position of the x for groups. Listing 5-6 shows where the first file has SUID applied, and the second file has SGID applied. Both permissions really make sense only in combination with the execute permission, so I won’t discuss the hypothetical situation in which a file has SUID applied but not executed for the owner, or has SGID applied but not executed for the group. Listing 5-6. Displaying SUID and SIGD with ls -l -rwsr-xr-x -rwxr-sr-x

2 2

root root 48782 2006-03-09 11:47 somefile-withSUID root root 21763 2006-03-09 11:48 someotherfile-withSGID

Using chmod in Absolute Mode Although the chmod relative mode is easy to work with if you just want to set or change one permission, it can get complicated if you need to change more than that. In such a case, the absolute mode is more useful because it offers a short and convenient way to refer to the permissions that you want to set. In the absolute mode, you work with numeric values to refer to the permissions that you want to set. For example, chmod 1764 somefile can be used to change the permissions on a given file. Of these four digits, the first refers to the special permissions, the second indicates permissions for the user, the third refers to the group permissions, and the last refers to permissions for others. Of the four digits that are used in absolute mode, the first can be omitted in most instances. If you do that, no special permissions are set for this file. When working with chmod in absolute mode, you have to be aware of the values for the permissions you are working with: • Read: 4 • Write: 2 • Execute: 1 • SUID: 4 • SGID: 2 • Sticky bit: 1 For example, to set permissions to read, write, and execute for others; to read and execute for group; and to do nothing for others, you would use chmod 750 somefile. In this example, the digit 7 refers to the user permissions. Because 7 is equal to 4 + 2 + 1, the user has read, write, and execute permission. The group has 5, which equals 4 + 1. The others have no permissions.

135

136

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

As an alternative, you can use the command but with a 0 preceding the value (chmod 0750 somefile). However, it makes no sense in this case to use the initial 0 because no special permissions are used here.

Using umask to Set Default Permissions for New Files You have probably noticed that default permissions are set when creating a new file, and these permissions are determined by the umask setting. This is a shell setting that is set for all users when logging in to the system. The default umask makes sure that all users have read access to all new files. Because this isn’t very secure, it makes sense to restrict the default umask a little. A numeric value is used in the umask setting, and this value is subtracted from the maximum permissions that can be set automatically; the maximum settings are 666 for files and 777 for directories. Of course, some exceptions to this rule make it all quite hard to understand, but you can find a complete overview of umask settings in Table 5-2. Of the digits used in the umask, as with the numeric arguments for the chmod command, the first digit refers to user permissions, the second digit refers to the group permissions, and the last digit refers to the default permissions for others. The default umask setting of 022 gives 644 for all new files and 755 for all new directories that are created on your server. Table 5-2. umask Values and Their Results

Value

Applied to Files

Applied to Directories

0

read and write

everything

1

read and write

read and write

2

read

read and execute

3

read

read

4

write

write and execute

5

write

write

6

nothing

execute

7

nothing

nothing

You can change the umask setting for all users or for individual users. If you want to set the umask for all users, you must make sure the umask setting is entered in the /etc/profile configuration file. If the umask is changed in this file, it applies to all users logging in to your server. An alternative to setting the umask in /etc/profile (where it is applied to all users logging in to the system) is to change the umask settings in a file with the name .profile that is created in the home directory of an individual user. Settings applied in this file are applied for only the user who owns the home directory, so this is a nice method to create an exception for a single user. You could, for example, create a .profile in the home directory of the user root and in there apply the umask setting of 027, whereas the generic umask setting for ordinary users is set to 022 in /etc/profile.

Working with Access Control Lists Up to now, you’ve just read about the basic model to apply permissions on a Linux system. When an advanced file system like Ext3 is used, it’s possible to add some options to this

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

default model. You’d do this by using the ACL feature. In this section, you’ll learn how this technique can be applied to allow for a more flexible security mechanism. The main reason behind the development of the Linux ACL system was to compensate for the shortcomings of default Linux permissions. Basically, the system had two problems: • Default Linux permissions do not allow more than one entity to be set as user or group owner of a file. • In a default Linux permissions scheme, there is no option to set default permissions. ACLs offer an optional system that can be used to compensate for these shortcomings. In this section you’ll learn how to apply this system.

Preparing the File System for ACLs Before you can use ACLs on a file system, you must add an option to /etc/fstab for all file systems that must support ACLs (all relevant Linux file systems do). The following procedure describes how: 1. Open /etc/fstab with an editor. 2. Select the column in which the mount options are specified. Add the option acl. Repeat this procedure for all file systems in which you want to use ACLs. 3. Remount all partitions in which ACLs have been applied (or restart your server). For example, to remount the root partition so that new settings are applied, use mount -o remount /.

Using ACLs to Grant Permissions to More than One Object The idea of an ACL is that connected to a file or directory, a list of users and groups is created that has permission on a file or directory. By default, in the inode that is used for the complete administration of files and directories, there simply isn’t enough room, and you can’t easily change this because of backward compatibility. As a result, you must specify for all devices with which you want to use ACLs that ACLs have to be enabled for that device. Only then can ACLs be set. ACLs can be used on most modern Linux file systems.

■Note The /etc/fstab file on Ubuntu server uses UUIDs instead of the device names of your file system. Remember, a UUID is a unique ID that can be assigned to a file system. In the case of an Ext3 file system, for example, this is done with the tune2fs command. For better readability, I’ve chosen to omit the UUIDs from the examples in this book, and I’ll just refer to the device name of the file system.

If ACLs are enabled for a given device, you can use the setfacl command to set them. If this command isn’t available, run apt-get install acl first. The use of setfacl is not too hard to understand: for example, setfacl -m u:linda,rwx somefile can be used to add user linda as a trustee (someone who has rights to a file) on the file somefile. This command does not change file ownership, though; it just adds to the ACL a second user who also has rights to the

137

138

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

file. The normal output of the ls -l command does not show all users who have rights by means of an ACL, but the + character appears behind the permissions list on that file. To get an overview of all ACLs currently set to a given file, use the getfacl command. The following procedure gives you an opportunity to try it: 1. Make sure that you are root and then create a file somewhere in the file system. You can use the touch command to create an empty file; for example, use touch somefile to create the file somefile. 2. Now use getfacl somefile to monitor the permissions that are set for this file. You will see an overview, as shown in Listing 5-7, indicating only the default permissions that are granted to user, group, and others. Listing 5-7. Before Applying an ACL, getfacl Just Displays Normal User, Group, and Others Information myserver:~# touch somefile myserver:~# getfacl somefile # file: somefile # owner: root # group: root user::rwgroup::r-other::r-3. Now use the command setfacl -m g:account:rw somefile (you must have a group with the name account for this to work). The group will now be added as a trustee to the file, which can be seen when you use getfacl on the same command again. Listing 5-8 provides an example of this. Listing 5-8. After Adding Another Trustee, getfacl Will Show Its Name and the Rights You Have Granted to This Trustee myserver:~# touch somefile myserver:~# getfacl somefile # file: somefile # owner: root # group: root user::rwgroup::r-group:account:rwmask::rwother::r--

Working with ACL Masks In the example in Listing 5-8, you can see what happens when a simple ACL is created: not only is a new entity added as the trustee of the object but a mask setting is also added. The

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

mask is the summary of the maximum of permissions an entity can have on the file. This mask is not very important because it is modified automatically when new permissions are set with the ACL. However, the mask can be used to reduce the permissions of all trustees to a common denominator. Because it’s set automatically when working with ACLs, I recommend that you just ignore the ACL masks: it makes things very complicated if you try to modify them in a useful way.

Using Default ACLs A default ACL can be applied on a directory. When using a default ACL, you can specify the permissions that new files and directories will get when they are created in a given directory. Therefore, you can consider default ACLs as a umask setting that is applied on a directory only. If a directory has a default ACL, all files will get the permissions specified in that default ACL. Also, subdirectories will get the permissions from the default ACL, and these permissions will be set as their own permissions as well. If a default ACL exists for a directory, the umask setting is not used for that directory. To set a default ACL, the setfacl command must be used with the -d option. Otherwise, it can be used with parameters as seen earlier. The following example will apply a default ACL to somedir: setfacl -d -m group:account:rwx somedir Because this command uses the -d option, a default ACL is set for all entities that currently are listed as trustees of the directory. You can see in Listing 5-9 that the command getfacl is used to display the permissions currently applied to that directory. Listing 5-9. Displaying the Default ACL for a Directory myserver:~# getfacl somefile # file: somedir # owner: root # group: root user::rwx group::r-x other::r-x default:user::rwx default:group::r-x default:group:account:rwdefault:mask::rwx default:other::r-x The nice thing about working with default ACLs is that the rights that are granted in a default ACL are automatically added for all new files and directories created in that directory. However, you should be aware that when you apply a default ACL to a directory, files and directories that currently exist within that directory are not touched by this default ACL. If you want to change permission settings in ACLs for existing files and directories, use the setfacl command with the option -R (recursive).

139

140

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

ACL Limitations You should also be aware of the limitations of working with ACLs, such as the fact that ACLs are not cumulative (which is also the case for the normal Linux permissions). Let’s imagine the not-so-realistic situation in which user stacey is the owner of a file and has only read permission. She is also a member of the group sales, which is a trustee of the same file and has read-write permission. So, when the permissions for this user are calculated, she will not have both read and write permissions. When determining the effective permission, the operating system will check if she is the owner of the file. She is, and so the operating system will look no further and the permissions for the owner are applied. The permissions for the group are ignored. The problem of nonaccumulation becomes even more complex if a process belongs to more than one group. When determining group rights, the group from which the process will get its rights is selected randomly. Another problem when working with ACLs is that many applications still don’t support them. For example, most backup applications cannot handle ACLs, and your database probably doesn’t either. However, changes are coming, and some applications have begun supporting ACLs. One of these is the Samba file server, which uses ACLs extensively to emulate the working of Windows rights (check Chapter 10 for complete coverage of this server). Also, some of the basic Linux utilities such as cp, mv, and ls currently support ACLs. However, you should always check that the utility you want to use supports ACLs before you start using it.

Applying File Attributes When working with permissions, there’s always a combination between a user or group object and the permissions that these user or group objects have on a file or directory. An alternate but seldom-used method of securing files on a Linux system is to work with attributes, which do their work regardless of the user who accesses the file. Of course, the difference is that the owner of a file can set file attributes, whereas other users (except for the almighty root) cannot. For file attributes, an option must be provided in /etc/fstab before they can be used. This is the user_xattr option that can be seen in the fstab example in Listing 5-7. Here’s a list of the useful attributes that can be applied: • A: This attribute ensures that the file access time of the file is not modified. Normally, every time a file is opened, the file access time must be written to the file’s metadata, which slows system performance. So, on files that are accessed on a regular basis, the A attribute can be used to disable this feature.

■Tip You don’t like the access time being modified at all? In this case, use the noatime option in /etc/fstab to specify that this feature be disabled for all files on a volume.

• a: This attribute allows a file to be modified but not removed.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

• c: If you are using a file system that supports volume-level compression, this file attribute makes sure that the file is compressed the first time the compression engine is activated. This attribute is currently ignored by Ext2/Ext3 file systems. • D: This attribute makes sure that changes to files are written to disk immediately and not to cache first. This is a useful attribute on important database files to make sure that they don’t get lost between file cache and hard disk. Using this option decreases the risk of losing data because of a power failure, for instance. • d: This attribute ensures that the file is not backed up in backups when the dump utility is used. • I: This attribute enables indexing for the directory in which it is enabled. You’ll thus enjoy faster file access for primitive file systems such as Ext3 that don’t use a B-tree database for fast access to files. Users cannot set this attribute using chattr; it will be managed by file system–specific utilities.

■Note A B-tree is a tree data structure that keeps data sorted, even if changes occur. A file system can operate very quickly using a B-tree.

• j: This attribute ensures that on an Ext3 file system the file is first written to the journal; only after that it is written to the data blocks on hard disk. • s: This attribute overwrites the blocks in which the file was stored with zeros after the file has been deleted. This makes sure that recovery of the file is not possible after it has been deleted. This attribute is not currently supported by Ext2/Ext3 file systems. • u: This attribute saves undelete information. A utility can then be developed that works with that information to salvage deleted files. This attribute is not currently supported by Ext2/Ext3 file systems.

■Note Although you can use quite a few attributes, you should be aware that most of them are rather experimental and are useful only if an application is used that can work with the given attribute. For example, it doesn’t make sense to apply the u attribute if no application has been developed that can use this attribute to recover deleted files.

Use the chattr command if you want to apply attributes. For example, use chattr +s somefile to apply the attribute s to somefile. Need to remove the attribute again? Use chattr -s somefile. For an overview of all attributes that can be used, use the lsattr command.

141

142

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

Apply Quota to Allow a Maximum Amount of Files User quota is a completely different way to apply restrictions to control how users can create files and directories. By using quota, the amount of space that a user can occupy is limited. Configuring user quota is a simple five-step procedure: 1. Install the quota software. 2. Prepare the file system in which you want to use quota. 3. Initialize the quota system. 4. Apply quota to users and groups. 5. Start the quota service. Before starting to apply quota, you should first realize how it must be applied. Quotas are always user- or group-related and apply to a complete volume or partition. That is, if you have one disk in your server with one partition on it that holds your complete root file system, and you apply a quota of 100 MB for user zeina, this user can create no more than 100 MB of files anywhere on the file system. When working with quotas, you need to apply a hard limit, a soft limit, and a grace period. The soft limit is a limit that cannot be surpassed on a permanent basis. (The user can create more data than the quota allows on a temporary basis.) The grace period is the length of time that the user can temporarily exceed the soft limit. The hard limit is an absolute limit; after it’s reached (or when the grace period elapses, whichever is sooner), the user can’t create new files. Working with soft and hard limits is confusing at first glance, but it has some advantages: if a user has more data than the soft limit allows, she still can create new files and isn’t stopped in her work immediately. She will, however, get a warning to create some space before the hard limit is reached.

Installing the Quota Software To work with quotas, it makes sense that the quota software must be installed. You’ll do this with the apt-get install quota command, and you’ll notice soon enough whether you need to run it. If you try to use one of the quota management utilities (such as edquota) when the quota software has not been installed yet, you’ll see a message that it has to be installed first.

Preparing the File System for Quota Before you can use the quota software to limit the amount of disk space that a user can use on a given file system, you must add an option to /etc/fstab for all file systems that must support quota. Here’s the procedure: 1. Open /etc/fstab with an editor. 2. Select the column with options. Add the option usrquota if you want to apply quota to users and grpquota for groups. Repeat this procedure for all file systems in which you want to use quota.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

3. Remount all partitions in which quota has been applied (or restart your server). For example, to remount the root partition so that new settings are applied, use mount -o remount /.

Initializing Quota Now that you’ve finished the preliminary steps, you need to initialize the quota system. This is necessary because all file systems have to be searched for files that have already been created, and for a reason that’s probably obvious: existing files count toward each user’s quota, so a report must be created in which the quota system can see which user owns which files. The report generated by this quota initialization is saved in two files: aquota.user is created to register user quotas, and aquota.group is created for group quotas. To initialize a file system for the use of quotas, you need to use the quotacheck command. This command can be used with some options, and I’ll list only the most important ones here: • -a: This option ensures that all file systems are searched when initializing the quota system. • -u: This option ensures that user information is searched. This information will be written to the aquota.user file. • -g: This option ensures that group information is searched as well. This information is written to the aquota.group file. • -m: Use this option to make sure that no problems will occur on file systems that are currently mounted. • -v: This option ensures that the command will work in verbose mode to show exactly what it is doing. So, the best way to initialize the quota system is to use the quotacheck -augmv command, which (after awhile) creates the files aquota.user and aquota.group to list all quota information for current users.

Setting Quota for Users and Groups Now that the quota databases have been created, it’s time for the real work because you’re ready to apply quota to all users and groups on your system. You’ll do this with the edquota command, which uses the nano editor to create a temporary file. This temporary file is where you’ll enter the soft and hard limits you’ve decided upon for your users and groups. If, for example, you want to apply a soft limit of 100,000 blocks and a hard limit of 110,000 blocks for user florence, follow these steps:

■Tip The edquota command works only with blocks, not bytes, kilobytes, or anything else. So, to set quota properly, you need to know the block size that’s currently used. To find that block size, use the dumpe2fs | less command. You’ll find the block size in the second screen.

143

144

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

1. Issue the command edquota -u florence. 2. In the editor screen, six numbers specify the quota for all file systems on your computer. The first of these numbers is the number of blocks that are currently being used by the user you’re creating the quota file for. The second and third numbers are important as well: the second number is the soft limit for the number of blocks, and the third number is the hard limit on blocks in kilobytes. The fifth and sixth numbers do the same for inodes, which roughly equal the number of files you can create on your file system. The first and fourth numbers are used to record the number of blocks and inodes that are currently being used for this user. 3. Close the editor and write the changes in the quota files to disk. In this procedure, you learned that quota can be applied to the number of inodes and blocks. If quotas are used on inodes, they specify the maximum number of files that can be created. Most administrators think it doesn’t make sense to work that way, so they set the values for these to 0. A value of 0 indicates that this item currently has no limitation. After setting the quota, if the soft limit and hard limit are not set to the same value, you need to use the edquota -t command to set the grace time. This command opens another temporary file in which you can specify the grace time you want to use, either in hours or in days. The grace time is set per file system, so there’s no option to specify different grace time settings for different users. Once you have set quotas for one user, you may want to apply them to other users. Instead of following the same procedure for all users on your system, you can use the edquota -p command. For example, edquota -p florence alex copies the quotas currently applied for user florence to user alex.

■Caution To set quotas, the user you are setting quotas for must be known to the quota system. This is not done automatically. To make sure that new users are known to the quota system, you must initialize the quota system again after creating the new users. I recommend setting up a cron job (see the “Setting the System to Your Hand” section in Chapter 6 to do this automatically at a reasonable interval).

When all the quotas have been set the way you want, you can use the repquota command to monitor the current quota settings for your users. For example, the repquota -aug command shows current quota settings for all users and groups on all volumes. Now that you’ve set all the quotas you want to work with, you just have to start the quota service, and you’ll do this with the /etc/init.d/quota start command.

Understanding Pluggable Authentication Modules In the normal situation, the local user database in the Linux files /etc/passwd and /etc/shadow is checked at login to a Linux workstation. In a network environment, however, the login program must fetch the required information from somewhere else (for example, an LDAP directory service such as OpenLDAP). But how does the login program know where it has to search for authentication information? That’s where the PAMs come in, and PAMs are what

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

makes the login procedure on your workstation flexible. Using a PAM in conjunction with nsswitch.conf, you can redirect any application that has to do anything related to authentication to any service that handles authentication. A PAM is used, for example, if you want to authenticate with a private key stored on a USB stick, to enable password requirements, to prevent the root user from establishing a telnet session, and in many other situations. The only thing you need is a PAM that supports your authentication method. The main advantage of a PAM is its modularity. In a PAM infrastructure, anything can be used for authentication, provided there’s a PAM module for it. So, if you want to implement some kind of strong authentication, ask your supplier for a PAM module and it will work. PAM modules are stored in the directory /lib/security, and the configuration files specifying how these modules must be used (and by which procedures) are in /etc/pam.d. Listing 5-10 is an example of just such a configuration file, in which the login procedure learns that it first has to contact an LDAP server before trying any local login. Listing 5-10. Sample PAM Configuration File auth account password session auth auth auth #auth auth auth account password password session session

sufficient sufficient sufficient optional requisite required required required required required required required required required required

/lib/security/pam_ldap.so /lib/security/pam_ldap.so /lib/security/pam_ldap.so /lib/security/pam_ldap.so pam_unix2.so pam_securetty.so pam_nologin.so pam_homecheck.so pam_env.so pam_mail.so pam_unix2.so pam_pwcheck.so nullok pam_unix2.so nullok use_first_pass use_authok pam_unix2.so pam_limits.so

The authentication process features four different instances, and they are reflected in Listing 5-10. Authentication is handled in the first instance; these are the lines that start with the keyword auth. During the authentication phase, the user login name and password are first checked, followed by the validity of the account and other account-related parameters (such as login time restrictions). This happens in the lines that start with account. Then, all settings relating to the password are verified (the lines that start with password). Last, the settings relating to the establishment of a session with resources are defined; this happens in the lines that start with session. The procedure that will be followed upon completion of these four instances is defined by calling the different PAM modules. This occurs in the last column of the example configuration file in Listing 5-10. For example, the module pam_securetty can be used to verify that the user root is not logging in to a Linux computer via an insecure terminal. The keywords sufficient, optional, required, and requisite are used to qualify the degree of importance that the conditions in a certain module are met. Except for the first four lines (which refer to the connection a PAM has to make to an LDAP server), conditions defined in all modules must

145

146

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

be met; they are all required. Without going into detail, this means that authentication will fail if one of the conditions implied by the specified module is not met. When enabling a server for logon to an LDAP server (as in the example in Listing 5-10), four lines are added to the default PAM configuration file in /etc/pam.d/login. They are the first four lines, and they offer an alternative for valid authentication by using the pam_ldap.so module. Passing the conditions imposed by these first four modules is sufficient to authenticate successfully, but it is not required. Sufficient in this context means that if, for example, the instance auth passes all the conditions defined in pam_ldap.so, that’s enough for local authentication. The local Linux authentication mechanism will no longer be used because the user can authenticate against the LDAP server in this case. For this to work, you of course need a valid user account that has all the required Linux properties on the LDAP server.

■Note Configuring LDAP is beyond the scope of this book, but have a look at www.padl.com, for example, for more information about this subject.

A nice thing about this example PAM configuration file is that it first sees whether the LDAP server can be used to authenticate to the network. The default Linux login mechanism is used only if this procedure doesn’t work. The workings of this default mechanism are defined from the fifth line on in the example configuration file. By default, many services on Ubuntu Server are PAM-enabled. (You can see this from a simple ls command in the directory /etc/pam.d, which will show you that there is a PAM file for login, su, sudo, and many other programs. I won’t cover all of them here, but will explain a bit about some when the time is relevant.) The true flexibility of PAM is in its modules, which you can find in /lib/security. Each of these modules has a specific function. The next section provides a short description of some of the more interesting modules. But, before we dive into that, you’ll quickly learn how to set a secure default policy.

Creating a Default Security Policy In a PAM environment, every service should have its own configuration for PAM. However, the world isn’t perfect, and a given service may not have a PAM configuration. In this case, I recommend creating /etc/pam.d/other as a PAM configuration file. This file is processed by all PAM applications that don’t have their own configuration file. If you really want to know whether your system is secure, give it the contents detailed in Listing 5-11. Listing 5-11. Configuring PAM for Security in /etc/pam.d/other auth auth account account password password session session

required required required required required required required required

pam_warn.so pam_deny.so pam_warn.so pam_deny.so pam_warn.so pam_deny.so pam_warn.so pam_deny.so

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

All four phases in the authentication process call two modules: pam_warn and pam_deny. The pam_warn module generates a warning and writes that to your log environment (/var/log/ messages by default). Next, for all these instances, the module pam_deny is called. This simple module will just deny everything. The results? All modules will handle authentication properly, as defined in their own configuration file, but when that file is absent, this generic configuration will make sure that all access is denied.

■Tip Want to know if a program is PAM-enabled? Use ldd

programname. For example, use ldd /usr/bin/passwd to find the library files used by this command. If the modules libpam_misc and libpam

are listed, the module is PAM-enabled. And so it should have its own configuration file for handling user authentication.

Discovering PAM Modules The usefulness of a system like PAM is entirely determined by its modules. Some of these modules are still experimental, and others are pretty mature and can be used to configure a Linux system. I’ll discuss some of the most important modules in the following sections.

pam_deny As seen in Listing 5-11, the pam_deny module can be used to deny all access. It’s very useful if used as a default policy to deny access to the system.

pam_env The module pam_env is used to create a default environment for users when logging in. In this default environment, several system variables are set to determine what the environment a user is working in looks like. For example, there is a definition of a PATH variable in which some directories are included that must be in the search path of the user. To create these variables, pam_env uses a configuration file in /etc/security/pam_env.conf. In this file, several variables are defined, each with its own value to define essential items like the PATH environment variable.

pam_limits Some situations require an environment in which limits are set to the system resources that a user can access. Think, for example, of an environment in which a user can use no more than a given number of files at the same time. To configure these limitations, you would modify the /etc/security/limits.conf file. To make sure that the limitations that you set in /etc/ security/limits.conf are applied, use the pam_limits module. In /etc/security/limits.conf, limits can be set for individual users as well as groups. The limits can be applied to different items, some of which are listed here: • fsize: Maximum file size • nofile: Maximum number of open files

147

148

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

• cpu: Maximum CPU time in minutes • nproc: Maximum number of processes • maxlogins: Maximum number of times this user can log in simultaneously The following code presents two examples of how these limitations can be applied. In the first line, the user ftp is limited to start a maximum of one process simultaneously. Next, everyone who is a member of the group student is allowed to log in four times simultaneously. ftp @student

hard -

nproc maxlogins

1 4

When applying these limitations, you should remind yourself of the difference between hard and soft limits: a hard limit is absolute, and a user cannot exceed it. A soft limit can be exceeded, but only within the settings that the administrator has applied for these soft limits. If you want to set the hard limit to the same as the soft limit, use a – character, as seen in the previous code example for the group student.

pam_mail This useful module looks at the user’s mail directory and indicates whether there is any new mail. It is typically applied when a user logs in to the system with the following line in the relevant PAM configuration file: login

session

optional

pam_mail.conf

pam_mkhomedir If a user authenticates to a machine for the first time and doesn’t have a home directory yet, pam_mkhomedir can be applied to create this home directory automatically. This module will also make sure that the files in /etc/skel are copied to the new home directory. This module is especially useful in a network environment in which users authenticate through NIS or LDAP and do not always work on the same machine. However, it’s recommended in such situations to centralize user home directories on an NFS server so that no matter where a user logs in to a server, a home directory will always be present. Chapter 8 contains more information about configuring an NFS server. The disadvantage of pam_mkhomedir is that if the module is not applied correctly, a user may end up with home directories on many different machines in your network.

pam_nologin If an administrator needs to conduct system maintenance like installing new hardware, and the server must be brought down for a few moments, the pam_nologin module may prove useful. This module makes sure that no users can log in when the file /etc/nologin exists. So before you perform any maintenance, make sure to create this file. The user root will always be allowed to log in to the system, regardless of whether this file exists or not.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

pam_permit Pam_permit is by far the most insecure PAM service available. It does only one thing: it grants access—always—no matter who tries to log in. All security mechanisms will be completely bypassed in this case, and even users who don’t have a valid user account can use the services that are configured to use pam_permit. The only sensible use of pam_permit is to test the PAM awareness of a certain module or to disable account management completely and create a system that is wide open to everyone.

pam_rootok This module lets user root access services without entering a password. It’s used, for example, by the su utility to make sure the user root can su to any account, without having to enter a password for that user account.

pam_securetty In the old days when telnet connections were still very common, it was important for the user root never to use a telnet session for login because telnet sends passwords in clear text over the network. For this purpose, the securetty mechanism was created: a file /etc/securetty can be created to provide a list of all TTYs from which root can log in. By default, these include only local TTYs 1 through 6. On Ubuntu Server, this module is still used by default, which means that you can limit the TTYs in which root can log in by manipulating this file.

pam_tally This very useful module can be used to keep track of attempts to access the system. It also allows the administrator to deny access if too many attempts fail. The PAM module pam_tally works with an application that uses the same name pam_tally that can be used to set the maximum amount of failed logins that are allowed. All attempts are logged by default in the /var/log/faillog file. If this module is called from a configuration file, be sure to at least use the options deny=n and lock_time. The first determines the maximum number of login attempts a user can make, and the second determines how long an account will be locked after that number of login attempts has been reached. The value given to lock_time is expressed in seconds by default.

pam_time Based upon the configuration file /etc/security/time.conf, the pam_time module is used to limit the times between which users can log in to the system. You can use this module to limit the access for certain users to specific times of the day. Also, access can be further limited to services and specific TTYs that the user logs in from. The configuration file time.conf uses lines with the following form: services;ttys;users;times

149

150

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

The next line is an example of a configuration line from time.conf that denies access to all users except root (the ! character in front of the times is used to deny access). This might be a perfect solution to prevent users from breaking into a system that they shouldn’t be trying to log in to anyway. login ; tty* ; !root ; !Al0000-2400

pam_unix This is probably the most important of all modules: it is used to redirect authentication requests through the /etc/passwd and /etc/shadow files. The module can be used with several arguments, such as nullok and try_first_pass. The nullok argument allows a user with an empty password to connect to a service, and the try_first_pass argument will always try the password a user has already used (if a password is asked for again). Notice that many PAM configuration files include a line to call the common configuration file common-auth. The pam_unix file is called from here.

pam_warn The pam_warn module is particularly useful with log errors: its primary purpose is to enable logging information about proposed authentication or password modification. For example, it can be used in conjunction with the pam_deny module to log information about users trying to connect to your system.

Configuring Administrator Tasks with sudo Once upon a time, if the system administrator wanted to perform his administration tasks, he would do that as root. However, this has some security risks, the most important of which is that you might make a mistake and thus remove everything from your server by accident. Therefore, on Ubuntu Server, the root account is disabled by default. It doesn’t even have a password, so you can’t log in as root after a default installation. To perform tasks for which root privileges are required, use the sudo mechanism instead. The idea of sudo is that specific administrator tasks can be defined for specific users. If one such user wants to execute one of the sudo commands that she has been granted access to, she has to run it with sudo. For example, where normally the user root would type shutdown –h to shut a machine down, a random user with sudo privileges would type sudo shutdown –h now. Next, the user enters his password and the machine shuts down. Because sudo is the basic mechanism on Ubuntu to perform tasks that normally are reserved for root only, after a normal installation every administration tasks is performed that way. As discussed in Chapter 2, if you first run as an ordinary user the sudo passwd root command, you can then set a password for the user root and do your work as root anyway. This technique can be quite handy for administration of a server for which root privileges are required all the time. After all, you have to work in the way that you like best. To create a sudo configuration, you need to use the editor visudo. This editor is used to open a temporary file with the name /etc/sudoers. In this file, you can define all sudo tasks that must be available on your server. You should never open the /etc/sudoers file for editing directly because that involves the risk of completely locking yourself out if you make an error.

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

■Tip On Ubuntu Server, visudo uses the text editor nano by default. If you are a Linux veteran who is used to Vi, you probably won’t like this. Want to use Vi instead of nano? Use the command export VISUAL=vi. Like what you see? Put it as the last line in /etc/profile. From now on, every time that you use either visudo or edquota, Vi is started instead of nano. In this book, I’m using the Vi alternative because it automatically saves all files in the locations where they have to be saved.

As you can see in Listing 5-12, the default configuration in /etc/sudoers is rather simple. Listing 5-12. Default Configuration in /etc/sudoers root@RNA:/etc# cat sudoers # /etc/sudoers # # This file MUST be edited with the 'visudo' command as root. # # See the man page for details on how to write a sudoers file. # Host alias specification # User alias specification # Cmnd alias specification # Defaults Defaults

!lecture,tty_tickets,!fqdn

# User privilege specification root ALL=(ALL) ALL # Members of the admin group may gain root privileges %admin ALL=(ALL) ALL It’s really just two lines of configuration. The first line is root ALL=(ALL) ALL, which specifies that user root has the right to run all commands from all machines. Next, you can see that the same is true for all users who belong to the user group admin. Typically, this is only the user you have created during the installation of Ubuntu Server. If, for example, you would like to specify that user linda is allowed to run the command /sbin/shutdown no matter which host she is connecting from, add the following line: linda

ALL=/sbin/shutdown

This line consists of three parts. In the first part, the user name is entered. (Instead of the name of a specific user, you can refer to groups as well, but, if you do that, make sure to put a % sign before the group name.) The second part—ALL in this example—refers to the name of the host where the user is logged on. Here, that host name has no limitations, but you can specify the name of a specific machine to minimize the risk of abuse by outsiders. Next, the command that this user is allowed to use (/sbin/shutdown, no options) is specified. This

151

152

CHAPTER 5 ■ CONFIGURING YOUR SERVER FOR SECURITY

means that the user is allowed to run all options that can be used with this command. If you want to allow the user just one option, you need to include that option in the command line. If that’s the case, all options that do not match the pattern you have specified in sudoers are specifically denied. Now that the sudo configuration is in place, the specified user can run his commands. To do this, the complete command should be referred to because the directories that typically house the root commands (/sbin, /usr/sbin) are not in the search path for normal users. So, user linda should use the following command to shut down the machine: sudo /sbin/shutdown -h now

Summary In this chapter, you learned how to configure local security on your server by using user accounts, groups, and permissions. I introduced some advanced file-system security options: ACLs and user-extended attributes. Next, you read about some important internal mechanisms: PAM and sudo. Your server ought to be relatively secure by now, so let’s proceed to Chapter 6, in which you’ll learn how to let the system do exactly what you want it to do. I’ll cover topics like process management, the boot procedure, and kernel management.

CHAPTER

6

Setting the System to Your Hand Management of Processes, Boot Procedure, Kernel, and Hardware A

fter reading the first five chapters of this book, you should have your server up, running, and secure. Up to now, however, you haven’t really changed the way processes on your server are running. So in this chapter, you will learn how to customize and optimize your server. We’ll have a look at some important aspects of your server that can be tuned and modified to increase its efficiency. First, I’ll talk about process monitoring and management. Then, I’ll talk about cron and how you can use it to automate process execution. After that, you’ll learn about the system boot procedure, followed by kernel and hardware management.

Process Monitoring and Management Everything you do on a Linux server is handled as a process by that server, so it’s very important that you know how to manage these processes. In this section, you’ll learn how to start and stop processes, which processes can be used, and how to run and manage processes in both the foreground and background. You will also learn how to use cron and schedule processes for future execution.

Different Kinds of Processes It depends on the way you look at them, but you could say that Linux basically has two different kinds of processes: automatic and interactive. The automatic processes include services that are started automatically when you boot your server—they are known as daemons. The upstart process that is responsible for an important part of your server’s boot procedure takes care that these processes are started properly. Daemons are service processes that run in the background; in other words, they do not write their output directly to the standard output. The interactive processes are started by users from a shell. Any command started by a user and producing output on the standard output is an interactive process. 153

154

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

To start an interactive process, a user needs to type the corresponding command. The process then runs as a child process from the shell in which the user entered the command. The process will do its work and will terminate when it’s finished. While terminating, it will write its exit status to its parent (which is the shell if the process was an interactive process). Only after a child process has told its parent that it has terminated can it be closed properly. In case the parent is no longer present (which is generally considered an error condition), the child process will become a so-called zombie process, and it won’t be possible to perform any management on the process except for trying to restart the parent process. In general, zombie processes are the result of bad programming. You should try to upgrade (or maybe rewrite) your software if you see too many of them. The concepts of parent and child processes are universal on your system. The init process is started by upstart (which I’ll cover later) as the first process; from there, all other processes are started. You can get an overview of the hierarchical process structure by using the pstree command, which provides a result such as that shown in Listing 6-1. Listing 6-1. The pstree Command Shows Relations Between Parent and Child Processes init ----apache2----5*[apache2] |--atd |--cron |--dd |--dhclient3 |--events/0 |--5*[getty] |--khelper |--klogd Although interactive processes are important to users working on your machine, daemon processes are more important on a server because those daemon processes usually provide the services that are needed on a server. Daemon processes typically run in the background and normally don’t send any output to the terminal. To see what they’re doing, you must check the log files to which the daemon processes write. Generally speaking, it’s a good idea to start with /var/log/messages if you’re looking for daemon output. From a perspective of process management, it doesn’t really matter if you’re working with daemon or interactive processes; both can be handled the same way using the same commands.

Foreground and Background When working with interactive processes, it can be useful to know that processes can run in the foreground and in the background. A foreground process takes up the current terminal, which gives optimal control of the process; the background process runs without an interface and has to be interrupted with a special command before it can be stopped or parameters can be passed to it. It might be useful for processes that take some time to complete. Before talking about the way you can start and manage processes that run in the background, let’s talk about some process details so that you can understand what’s happening. A process always works with three standard file handlers that determine where the process should send its output and accept its input. They are the standard input (STDIN), the standard output (STDOUT), and the standard error (STDERR). Normally, when a process is

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

running in the foreground, the STDIN is the keyboard, the STDOUT is the terminal the process is working on, and the STDERR is also the terminal that the process is working on. As you learned in Chapter 2, you can change them all by using redirection (for example, grep –i blah * 2> /dev/null would redirect the error output of the grep command to the null device). It can be a little confusing that the three file descriptors don’t change when you decide to run a process in the background. When it starts, the STDIN, STDOUT, and STDERR for a process are set; once they are set, they stay like that no matter what you do to the process. Therefore, you can run a long command like find / -name "*" -exec grep -ls something {} \; as a background job, but you’ll still see its output and errors on your screen if you haven’t redirected STDERR as well. If you don’t want to see errors, you should use redirection to send STDOUT and STDERR somewhere else: by putting > /somewhere after the command, you are redirecting the standard output to a file called /somewhere; and by using 2> /dev/null, you can arrange for all errors to be redirected to the null device.

■Tip Want to know what’s really happening? In the /proc file system, you can see how STDIN, STDOUT, and STDERR are defined. Check the directory with the process ID (PID) of the process as its name (see the section “Managing Processes” later in this chapter for more details on process IDs). In this directory, activate the subdirectory fd (short for “file descriptors”). You’ll see a list of all files the process currently has open— these are the so-called file descriptors. Number 0 is STDIN, 1 is STDOUT, and 2 is STDERR. Use the command ls -l to check what they are linked to, and you will know how STDIN, STDOUT, and STDERR are set for this process. If the subdirectory fd is empty, you’re probably dealing with a daemon process that has no file descriptors.

Now that you know what to expect when working with processes in the background, it’s time to learn how you can tell a process that it should be a background process. Basically, you can do this in one of two ways: • Put an & after the name of the command when starting it. This makes it a background job immediately. For example, use nmap 192.168.1.10 > ~/nmap.out & to run the nmap command as a background process. What’s the advantage of this? While waiting for the command to produce its output, you can do something else. • Interrupt the process with the Ctrl+Z key sequence and then use the bg command to restart it in the background. Once the command is running as a background job, you can still control it. Use the jobs command for an overview of all current background processes. You’ll see a list of all interactive processes that have been started as a background job from the same shell environment. In front of each of these processes, you can see their current job number, and this job number can be used to manage the process with the fg command. For example, if jobs gives you a result such as RNA:~# jobs [1]- Running [2]+ Running

cat /dev/sda > /dev/null & find / -name "*" -exec grep -ls help \; > /dev/null &

155

156

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

and you want to be able to terminate the cat command with Ctrl+C, use fg 1 to bring the cat command to the foreground again.

Managing Processes As a Linux administrator, process management is a major task. If, for example, your server is reacting very slowly, you can probably find a process that’s causing the problem. If this is the case, you need to know how to terminate that process, or maybe how you can reset its priority so that it can still do its work while not stopping other processes. The following sections describe what you need to know to perform daily process management tasks.

Tuning Process Activity If something isn’t going well on your server, you want to know about it. So, before you can conduct any process management, you need to tune process activity. Linux has an excellent tool that allows you to see exactly what’s happening on your server: the top utility. From this utility you can see everything you need to know. It is very easy to start top: use the top command. When the utility starts, you’ll see something like Figure 6-1.

Figure 6-1. The top utility gives you everything you need to know about the current state of your server. Using top to Monitor System Activity The top window consists of two major parts. The first (upper) part provides a generic overview of the current state of your system. These are the first five lines in Figure 6-1. In the second (lower) part of the output, you can see a list of processes, with information about the activity of these processes.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

The first line of the top output starts with the current system time. This time is followed by the “up” time; in Figure 6-1, you can see that the system has been up for only a few minutes. Next, you see the number of users currently logged in to your server. The end of the first line contains some very useful information: the load average. This line shows three different numbers. The first is the load average for the last minute, the second is the load average for the last 5 minutes, and the third is the load average for the last 15 minutes. The load average is displayed by a number that indicates the current activity of the process queue. The value here is the number of processes that are waiting to be handled by the CPU on your system. On a system with one CPU, a load average of 1.00 indicates that the system is completely busy handling the processes in the queue, but there are no processes waiting in the queue. If the value increases past 1.00, the processes are lining up, and users may experience delays while communicating with your server. It’s hard to say what a critical value exactly is. On many systems, a value anywhere between 1 and 4 indicates that the system is just busy, but if you want your server to run as smoothly as possible, make sure that this value exceeds 1.00 only rarely. If an intensive task (such as a virus scanner) becomes active, the load average can easily rise to a value of 4. It may even happen that the load average reaches an extreme number like 254. In this case, it’s very likely that processes will wait in the queue for so long that they will die spontaneously. What exactly indicates a healthy system can be determined only by doing some proper baselining of your server. In general, 1.00 is the ideal number for a one-CPU system. If your server has hyperthreading, dual-core, or two CPUs, the value would be 2.00. And, on a 32-CPU system with hyperthreading enabled on all CPUs, the value would be 64. So the bottom line is that each (virtual) CPU counts as 1 toward the overall value. The second line of the top output shows you how many tasks currently are active on your server and also shows you the status of these tasks. A task can have four different statuses: • Running: In the last polling interval, the process has been active. You will normally see that this number is rather low. • Sleeping: The process has been active, but it was waiting for input. This is a typical status for an inactive daemon process. • Stopped: The process is stopping. Occasionally, you’ll see a process with the stopped status, but that status should disappear very soon. • Zombie: The process has stopped, but it hasn’t been able to send its exit status back to the parent process. This is a typical example of bad programming. Zombie processes will sometimes disappear after a while and will always disappear when you have rebooted your system. The third row of top provides information about current CPU activity. This activity is separated into different statistics: • us: CPU activity in user space. Typically, these are commands that have been started by normal users. • sy: CPU activity in system space. Typically, these are kernel routines that are doing their work. Although the kernel is the operating system, kernel routines are still often conducting work on behalf of user processes or daemons.

157

158

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• id: CPU inactivity, also known as the idle loop. A high value here just indicates that your system is doing nothing. • wa: For “waiting,” this is the percentage of time that the CPU has been waiting for new input. This should normally be a very low value; if not, it’s time to make sure that your hard disk can still match up with the other system activity. If your CPU utilization is high, this should be the first parameter to check because a high workload on the CPU might be caused by a slow I/O channel. • hi: For “hardware interrupt,” this is the time the CPU has spent communicating with hardware. It will be rather high if, for example, you’re reading large amounts of data from an optical drive. • si: For “software interrupt,” this is the time your CPU has spent communicating with software programs. It should be rather low on all occasions. • st: This parameter indicates the time that is stolen by the virtualization hypervisor (see Chapter 13 for more details about virtualization and the hypervisor) from a virtual machine. On a server that doesn’t use any virtualization, this parameter should be set to 0 at all times. If there is a virtual machine that is very active on your server, this parameter will increase from time to time because it measures activity in the host operating system; the host operating system has to share CPU cycles with the virtual machine. The fourth and fifth lines of the top output display the memory statistics. These lines show you information about the current use of physical RAM (memory) and swap space. (Similar information can also be displayed using the free utility.) An important thing that you should see here is that not much swap space is in use. Swapping is bad because the disk space used to compensate for the lack of physical memory is approximately 1,000 times slower than real RAM. If all memory is in use, you should take a look at the balance between buffers and cache. Cache is memory that is used to store files recently read from the server’s hard drive. When files are stored in the cache, the request can be handled very quickly the next time a user requests the same file, thus improving the general speed of the server. Cache is good, and having a lot of it isn’t bad at all. A buffer is a region of memory reserved for data that still has to be written to the server’s hard drive. After the data has been written to the buffers, the process that owns the data gets a signal that the data has been written. This means that the process can go on doing what it was doing and has to wait no longer. Once the disk controller has time to flush the buffers, it will flush them. Although using buffers is helpful, there is a disadvantage: everything that was stored in the server’s buffers will be lost in case of a power outage if it hasn’t yet been written to disk. And that’s where the journal becomes useful: when the server reboots, the journal is read to recover the damaged files as fast as possible (see Chapter 4 for more information about journaling). The bottom line of monitoring the cache and buffers parameter is that it is occupied memory that can be freed as soon as it is needed for something else. If a process has a memory request, the server can clear these memory areas immediately to give the memory that becomes available to the process in need.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

The lower part of the top window provides details about the process that’s most active in terms of CPU usage. It will be the first process listed, and the line also displays some usage statistics: • PID: Every process has a unique process ID. Many tools such as kill need this PID for process management. • User: This is the name of the user ID the process is using. Many processes run as root, so you will see the user name root rather often.

■Note For well-programmed processes, it’s generally not a problem that they’re running as root. It’s a different story, though, for logging in as the user root.

• PRI: This is the priority indication for the process. This number is an indication of when the process will get some CPU cycles again. A lower value indicates a higher priority so that the process will have its share of CPU cycles sooner. The value RT indicates that it is a real-time process and is therefore given top priority by the scheduler. • NI: The nice value of the process. See “Setting Process Priority” later in this chapter for more details on nicing processes. • VIRT: The total amount of memory that is claimed by the process. • RES: The resident memory size is the amount of memory that is actually mapped to physical memory. In other words, it represents the amount of process memory that is not swapped. • SHR: The amount of shared memory is what the process shares with other processes. You’ll see this quite often because processes often share libraries with other processes. • S: This is the status of the process; they’re the same status indications as the ones in the second line of the top screen. • %CPU: This is the amount of CPU activity that the process has caused in the last polling cycle (which is typically every five seconds). • %MEM: This is the percentage of memory that the process used in the last polling cycle. • TIME+: This indicates the total amount of CPU time that the process has used since it was first started. You can display this same value by using the time command, followed by the command that you want to measure the CPU time for. • Command: This is the command that started the process. As you have seen, the top command really provides a lot of information about current system activity. Based upon this information, you can tune your system so that it works in the most optimal way.

159

160

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Other Tools to Monitor System Activity Although top is not the only tool that you can use for process monitoring, it’s the most important. Its major benefit is that it shows you almost all you need to know about your system’s current activity, but you should be aware that top itself takes up system resources as well, thus skewing the parameters that it shows. Some other good performance-monitoring tools are available as well: • ps: This tool gives a list of processes. • uptime: This tool shows how long the server is up and gives details about the load average as well. Basically, it displays the same output as the first line of output of the top command. • free: Use this tool to show information about memory usage. Use the –m option to display the result in megabytes.

Terminating Processes In your work as an administrator, you’ll need to terminate misbehaving processes on a regular basis. When terminating a process, you’ll send it a predefined signal. In general, the three important ones are SIGHUP, SIGKILL, and SIGTERM. If you send the SIGHUP signal to a process, it doesn’t really terminate the process, but just forces it to reread its configuration files. This is very useful to make sure that changes you made to configuration files are applied properly. Next is SIGKILL, which is sent to a process when someone uses the infamous kill -9 command to terminate a process. In this command, the -9 is a numerical representation for the SIGKILL signal. (Check the signal(7) man page for more details about signals and their numerical representations.) The SIGKILL signal doesn’t terminate a process nicely: it just cuts it off, and the results can be severe because the process doesn’t have an opportunity to save open files. Therefore, SIGKILL will definitely damage any open files and possibly even lead to system instability. So use it only as a last resort. SIGTERM is the third signal that a process will always listen to. When a process receives this signal, it shuts down gracefully. It closes all open files and also tells its parent that it’s gone. Using SIGTERM is the best way to terminate processes you don’t need anymore.

Commands for Process Termination You can use different commands to terminate a process. The following list provides a short description of the most important ones: • kill: This is one of the most commonly used commands to terminate processes. It works with the PID of the process you need to kill. If a special signal needs to be sent to a process, the signal is commonly referred to with its numeric argument (for example, kill -9 1498), but you can use kill --sigkill instead. If no signal is referred to, the default SIGTERM signal (15) is sent to the process.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• killall: The major disadvantage of kill is that it works with one or more PIDs and thus isn’t the best solution to kill more than one process at once. If you need to kill more than one process, you’d better use killall, which works with the name of the process. For example, killall httpd kills all instances of the Apache web server that are currently active on your server. By default, killall sends SIGTERM to the processes you want to kill. If you need it to do something else, add the name or number of the signal you want to send to the process. For example, use killall -SIGKILL httpd to kill all instances of the Apache web server.

■Tip The killall command works with processes that have exactly the name you’ve specified. It doesn’t work for processes that have that name somewhere in the command line that was started to invoke them. Thus, killall won’t kill these processes. To kill these processes anyway, you’ll need to issue a somewhat more complex command. The following command kills all processes that have the text “evolution” somewhere in the command line: kill `ps aux | grep evolution | grep –v grep | awk '{ print $2 }'`. The technique used in this line is referred to as command substitution, which means that the result of a command is used as input for another command. In this example, the command ps aux | grep evolution | grep –v grep | awk '{ print $2 }' results in a list of PIDs of all processes that have the text “evolution” somewhere in the command line. This list is next used by the kill command to terminate all these processes. You do want to be very sure about what you type before you run this command because a typo will kill processes that you maybe don’t want to.

• top: Killing a process from top is easy. From the top interface, press the k key. You’ll first be asked for the PID of the process you want to kill. Enter it, and then you’ll be asked what signal to send to the process. Specify the numeric value of the signal and press Enter. This terminates the process. • pkill: The pkill command is useful if you want to kill a process based on any information about the process. This command is related to the pgrep command, which allows you to find process details easily. For example, you can use pkill to kill all processes owned by a certain user: pkill -U 501 kills all processes owned by the user with UID 501. Because it knows many ways to refer to processes that you want to terminate, pkill is a very convenient command.

Using ps to Get Details About Processes Before killing a process, you most likely want some more information about it. Of course, you can use top to do this, but the utility has the disadvantage that it shows only the most active processes in its standard output. You can run it in batch mode, though, to get a complete list of all processes. Use top –b to obtain this result. If you need to manage a process that isn’t among the most active processes, the ps utility is very useful. By using the right parameters, this command will show all processes that are currently active on your server and, combined with grep, it offers a flexible way to find exactly what you were looking for.

161

162

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

If you don’t use any options with ps, it will just show you the processes that are interactive and that you own. Normally, this will be a rather short list. As a system administrator, you probably want to see a complete list of all processes. Now there is something funny with the ps command: you can use it in the BSD-style UNIX syntax, but also with the System V-style syntax.

■Note In the history of UNIX, two different flavors of UNIX developed: the BSD style and the System V style. Both flavors had different ways of doing things. In some Linux commands, Linux tries to make both users happy by implementing both flavors in one command. Therefore, the ps command has two different styles you can use.

You probably don’t care what kind of syntax you’re using, and you just want to see a list of active processes. This can be done by using the ps -ef command. Alternatively, ps -aux does this as well; check Listing 6-2 for an example of the output of this command. Both commands provide a complete list of all processes that are running on your system, and it’s just a matter of taste as to which you prefer. Although ps has some options to do sorting, instead of remembering what these options do, you can use grep to do some filtering. For example, ps -ef | grep httpd shows detailed information, but only about the output line where the httpd string occurs. Listing 6-2. The ps –aux Command Displays a Complete List of All Processes on the Server root@ubuntu:~# ps aux USER PID %CPU %MEM root 1 0.0 0.3 root 2 0.0 0.0 root 3 0.0 0.0 root 4 0.0 0.0 root 5 0.0 0.0 root 6 0.0 0.0 root 7 0.0 0.0 root 30 0.0 0.0 root 31 0.0 0.0 root 32 0.0 0.0 root 90 0.0 0.0 root 115 0.0 0.0 root 116 0.0 0.0 ... root 4202 0.0 0.1 root 4213 0.0 0.4 root 4215 0.0 0.3 root 4237 0.0 0.1

VSZ 2908 0 0 0 0 0 0 0 0 0 0 0 0

RSS 1844 0 0 0 0 0 0 0 0 0 0 0 0

TTY ? ? ? ? ? ? ? ? ? ? ? ? ?

STAT Ss S SN S S< S< S< S< S< S< S< S S

START 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43 11:43

TIME 0:01 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00

COMMAND /sbin/init [migration/0] [ksoftirqd/0] [watchdog/0] [events/0] [khelper] [kthread] [kblockd/0] [kacpid] [kacpi_notify] [kseriod] [pdflush] [pdflush]

5084 7864 4048 2564

968 2472 1780 996

? ? pts/0 pts/0

Ss Ss Ss R+

11:44 11:44 11:44 12:02

0:00 0:00 0:00 0:00

/usr/sbin/sshd sshd: root@pts/ -bash ps aux

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Setting Process Priority Killing a process may improve the performance of your server, but what if you still need that process? In this case, resetting its priority (renicing) may be an option. To understand what the commands nice and renice are doing, we first need to have a look at how the process scheduler works. Every system uses a process queue. All processes sit in this queue to wait for some CPU cycles. So, if three processes are named A, B, and C, they will each get an equal number of CPU cycles. If a process still needs more cycles after the process has been handled, it reenters the queue. Because it was the last process that was handled, it rejoins the process queue at the end. This all sounds pretty fair to all processes, but it just doesn’t work in some cases. Imagine that process A is the company database that causes 90 percent of all the workload on your server, and processes B and C are less important. In this case, you’d want to give process A a higher priority, and give a slightly lower priority to the other two processes. This is exactly what the nice and renice commands do. Both commands work with a numeric argument from –20 up to 19. If a process has the nice value –20 (which means that it is not nice at all to other processes), it gets the most favorable scheduling (highest priority), and if it gets 19, it gets the least favorable scheduling. Giving a nice value of –20 to a very important process may look like a good solution. But you should never do this. A very busy process that gets a nice value of –20 will exclude all other processes. Because, for example, kernel processes such as writing to disk also need to enter the process queue, you could give your database the highest priority, but then the database wouldn’t be able to write its data to disk. So that wouldn’t work. Let’s just say that –20 is a nice value you should never use. If you want to renice a process, do it carefully, such as by increasing or decreasing the nice value of a process in increments of 5. Several methods are available to renice processes: • nice: The nice command can be used to start a process with a given nice value. For example, nice 10 find / -name "*" -exec grep -ld help {} \; starts the find command with a lower priority. The disadvantage of this command is that you have to know beforehand that you are going to want to adjust the nice value of a process. • renice: The renice command is used to change the priority of a running command. This command normally works with the PID of the process you want to renice. For example, renice -10 1234 increases the priority of process 1234. • top: A very convenient way to renice a process is to use the top interface. From this interface, press the n key. You’ll then be asked to enter the PID of the process you want to renice. After entering this PID, enter the new nice value you want this process to have.

Executing Processes Automatically On a server system, it’s often important that processes are executed at a regular predefined time, and Linux offers the cron facility to do this. It works with two parts: a daemon called crond and some configuration files that the administrator can use to specify when the

163

164

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

processes should be started. Both parts ensure that the command is executed at regular times. Apart from cron, the at command can be used to run a command just once.

Configuring cron The cron service is activated by default. It checks its configuration files every minute to see if something needs to be done. The cron process looks for configuration data in different places: • The generic file /etc/crontab can contain lines that tell cron when to execute a given command. • In the directory /etc/cron.d, an administrator can put a file that defines what should happen, and when. When installing software packages, you’ll see that some files are automatically created in here as well. • Every user can have its own cron configuration file, telling the system when to execute certain tasks. A user can enter the command crontab –e to create a personal crontab configuration file. • The directories /etc/cron.hourly, cron.daily, cron.weekly, and cron.monthly are used to activate jobs once per hour, per day, per week, or per month, respectively. These directories contain files that are activated every hour, day, week, or month. Working with the cron time activation mechanisms makes it very easy for an administrator to start jobs on a regular basis. The scripts in these directories are just regular shell scripts that make sure the job gets done. So all you have to do is include a shell script that activates the job you want to start. These scripts can be very simple: just a line that starts the service is enough.

■Note Some daemons need to be restarted to make sure that changes are activated, but this isn’t true for cron, which rereads its configuration every minute to see whether any new jobs have been scheduled.

cron User Jobs You can set up your system to allow individual users to start their cron jobs. Such a configuration starts with the files /etc/cron.allow and /etc/cron.deny. A user who is listed in /etc/ cron.allow or who isn’t listed in /etc/cron.deny is capable of adding cron jobs. If /etc/cron. allow exists, only this file is evaluated and settings in /etc/cron.deny are ignored. If both files don’t exist, only root can create cron jobs. The cron configuration files for individual users are stored in the directory /var/spool/cron/crontabs. The crontab command can be used to edit these files. Next we’ll look at some examples of the crontab command in action.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

■Note By default, the crontab command uses the nano editor to do its work. The problem with nano, however, is that it doesn’t write its configuration files to the right location by default. To fix this, as root add the following line to the end of /etc/profile: export VISUAL=vim. After logging in again, vim will be used as the default editor for crontab and some other editor-related commands as well. The advantage? The right file will be created at the right location automatically.

• crontab -e: This creates or edits cron jobs for the user who executes the command. Again, nano is used as the editor to modify these files. • crontab -l: This command displays a list of all jobs that are scheduled for the current user. • crontab -r: This command deletes all jobs for the current user. In the cron files, you use lines to define what should happen, and each line specifies one command. The lines consist of six fields each: the first five specify when the command should be activated, and the last field specifies what command should be activated. The following code is an example of such a line: */5 8-18 * * 1-6 fetchmail mailserver The easiest part to understand in this line is the actual command: fetchmail mailserver. This command makes sure that incoming mail is fetched from mailserver. Then, in the first five fields, you can see an indication of the times that it should happen. These fields have the following meanings: • Minutes: This field specifies the minute when the command should be executed. It has a range from 0 to 59. Always specify something for this field; if you don’t, the command will run every minute. In the example, the construct */5 is used to specify that the command should run every 5 minutes. • Hours: This field specifies the hour that the command should run. Possible values are between 0 and 23. In the example, you can see that the command will run every hour between 8 and 18. • Day of the Month: Use this field to execute a command only on given days of the month. This field is often not specified. • Month: Use this field to specify in which month of the year the command should run. • Day of Week: This field specifies on which day of the week the command should run. The range is 0 to 7, and both of the values 0 and 7 should be used to specify that the command should run on Sunday.

165

166

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

■Note You normally will not use it, but, if you ever want to work with the /etc/crontab file, be aware that, between the time setting and the command you want to execute, the name of the user whose account should be used to execute the command is entered. For example, 0 17 * * * root /sbin/shutdown -h now would make sure the system shuts down automatically every day at 5 p.m. by using the permissions of the user root.

Executing Once with at The cron mechanism is used to execute commands automatically on a regular basis. If you want to execute a command just once, at is the solution you need. The at mechanism comprises different parts: • The service atd: Make sure it is started if you want to schedule commands to run once. • The files /etc/at.allow and /etc/at.deny: Used to specify which users can and cannot schedule commands with at. • The at command: Used to schedule a command. • The atq command: Used to display an overview of all commands that currently have been scheduled. • The atrm command: Used to delete jobs from the at execution queue. Scheduling a job with the at command is not hard; just use the at command followed by the time when you want to run the command, for example at 17:00. This command opens the interactive at prompt, where you’ll enter the commands you want to schedule for execution at the specific time. When you’ve finished entering the names of commands, use the Ctrl+D key sequence to close the interactive at prompt, and the commands will be scheduled. The at command has different options to specify when exactly a command should be executed. Some of the most useful options are listed here: • HH:MM: In its most elementary form, time is indicated in an HH:MM format; for example an entry of 17:00 will execute the command the next time it is 17:00 hours. • am/pm: If you don’t like the HH:MM notation, use am/pm instead; for example, at 5 pm. • DDMMYY HH:MM: To run a command at a specific time on a specific day, you can use a full day specification as well. Other options are available. For example, you can use words to tell at when to run a command. For example, at teatime tomorrow would run the command at 4 p.m. the next day. Check the at man page for more details.

Tuning the Boot Procedure It’s important that you know what happens during the boot procedure of your computer for two reasons. First, you need to know how it works to be able to perform troubleshooting.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Second, you need to be aware of what happens if you want to make sure that a service is activated automatically. In the Linux boot procedure, the following phases can be distinguished: • The boot loader GRUB is executed. • The upstart process is started. • The initial boot phase is executed. • The runlevel is activated. In the next subsections, you’ll learn in detail how these phrases work and how you can modify the boot procedure.

Managing the GRUB Boot Loader The BIOS of every computer has a setting for the device that should be used for booting by default. Often, the server will try to initiate the boot procedure from its hard drive. It reads the very first sector of 512 bytes (the master boot record, or MBR), in which it finds the GRUB primary boot loader in the first 446 bytes. After that are the 64 bytes in which the partition table is stored; and to finish, in the last 2 bytes are where a magic code is written. Upon installing your server, the installation program writes the GRUB boot code onto the hard drive. This code makes sure that your server is started automatically. However, you can also interrupt the automatic startup by pressing the Esc key. Figure 6-2 shows the menu that you’ll see in this case.

Figure 6-2. The GRUB boot menu allows you to interrupt the boot procedure.

The GRUB Configuration File GRUB has a text configuration file—/boot/grub/menu.lst—that defines all options from the boot menu. Here, you can specify the different boot options on your server. Listing 6-3 shows

167

168

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

the data that is normally in the GRUB configuration file just after installation of Ubuntu Server. For better readability, I removed all the comment sections from this file. Listing 6-3. Default GRUB menu.lst File default timeout hiddenmenu

0 3

title Ubuntu 8.04, kernel 2.6.24-16-server root (hd0,0) kernel /boot/vmlinuz-2.6.24-16-server\ root=UUID=1aa61aba-4b23-4e9d-9718-289f1c84a3a ro quiet splash initrd /boot/initrd.img-2.6.24-16-server quiet

title Ubuntu 8.04, kernel 2.6.24-16-server (recovery mode) root (hd0,0) kernel /boot/vmlinuz-2.6.24-16-server\ root=UUID=1aa61aba-4b23-4e9d-9718-e289f1c84a3a ro single initrd /boot/initrd.img-2.6.24-16-server title root kernel

Ubuntu, memtest86+ (hd0,0) /boot/memtest86+.bin

This file consists of several parts. The first is the general section, which defines some options that determine how the menu is used. Next are three sections, each devoted to one of the three different boot menu options. The first parts of the GRUB boot menu are the generic options. The example file shown in Listing 6-3 has three of them. The option default 0 specifies that the first section in menu.lst should be executed as the default section. Next, timeout 3 is used to give the user 3 seconds to interrupt the startup procedure. If the user doesn’t do anything during these 3 seconds, the server will continue with the boot process. The last generic boot option is hiddenmenu. As you can guess, this option causes the boot menu to be hidden by default. If the user presses the Esc key at this moment, the menu in Figure 6-2 will be displayed. In the second part, the first item in the boot menu is specified. This item has the title Ubuntu, kernel 2.6.20-15 server, which is defined with the title option. Next, everything that is needed to start the server is defined. First is the name of the root device that should be read. This line is needed for GRUB to know where it can find the kernel that it should load. In this example, this is the device root (hd0,0), which corresponds to /dev/sda1 or /dev/hda1. However, because the device names are not known at this stage in the boot procedure, it’s not possible to refer to these device names, so that’s why (hd0,0) is used. Check the file /boot/grub/device.map to see how these device mappings are currently defined.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

After specifying the root device, the kernel itself is referred to in the line that starts with kernel /boot/vmlinuz. This line also specifies all the options that are required to load the kernel properly. Some of the more common options are as follows: • root: This option refers to the device where the root file system is found. It’s possible to refer to common device names such as /dev/sda1 here. To add increased flexibility, however, file system UUIDs are used. In case your root is on a logical volume, you’ll see the logical volume device name here. Check Chapter 4 for more details, or use the dumpe2fs command to see parameters that are set for your Ext2/Ext3 file systems. • ro: Use this option to make sure that the root device is mounted read-only at this stage. This is necessary so that you’ll be able to perform a file system check later during the system boot. • quiet: This option suppresses most messages that are generated while booting. If you want to see exactly what happens, remove this option from menu.lst. • splash: Use this option to show a splash screen. In general, this is a graphical screen that is shown to the user during the boot process. You don’t want this option on a server; you should disable it so that you can see what happens when the server comes up. • vga: Use this option to specify the VGA mode as a hexadecimal argument when booting. This line determines the number of columns and lines used when starting your system. As an alternative to a value such as 0x314, you can use the option ask. In that case, you can enter the mode you want to use when booting. • ide: You can use this option to specify the mode that should be used for starting the IDE device. Use ide=nodma if you suspect that your server might have problems initializing IDE in DMA mode. • acpi: The advanced configuration and power interface (ACPI) option allows you to specify what to do with this sometimes problematic technique. By default, ACPI is on. Use acpi=off if you suspect that it’s causing some problems. • noresume: If your system was suspended, this option will just ignore that fact and start a new system. While starting this new system, the suspended system is terminated. Because normally you wouldn’t suspend a server, you probably don’t need this option, either. • nosmp: Use this option if symmetric multiprocessing (SMP) is causing you any trouble. But be aware that you’ll be using only one CPU if this option is used. • noapic: The advanced programmable interrupt controller (APIC) allows you to use interrupts with many more outputs and options than when using normal interrupts. However, this option can cause problems; so use noapic if you think that your system can’t properly handle APICs. • maxcpus: This option tells your kernel how many CPUs to work with. Use maxcpus=0 to force all off except the primary processor.

169

170

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• edd: This option specifies whether enhanced disk drive (EDD) support should be used. If you suspect that it’s causing problems, switch it off here. • single: This option is used only in recovery mode. It starts single-user mode, in which a minimal amount of services is started so that the administrator can perform troubleshooting. The following line specifies what to load as the initial RAM drive (initrd). The use of an initrd is very important on modern Linux systems because it’s used to load the kernel modules that are needed to boot the system. The other menu items that are defined in this boot menu work in more or less the same way: each starts with a specification of the root device and then a referral to the kernel that should be loaded. One of the nice features of GRUB is that it reads its configuration dynamically, which means that if you made any modifications to the options used in menu.lst, you don’t have to recompile or reinstall GRUB. This is a huge advantage as compared with LILO, the boot loader from the early days of Linux, in which you had to run the lilo command after all changes or modifications to the configuration. Any changes that you make to menu.lst will show immediately the next time you restart your server.

Installing GRUB Installing GRUB is not something that you’ll do very frequently because it’s installed by default. However, if your GRUB ever causes a problem when booting your server, you may need to reinstall it. Before you do this, however, you’ll have to boot your server from the installation CD-ROM. From the boot menu on the CD-ROM, select the option to rescue a broken system. After answering some generic questions about your server, a menu will offer different options (see Figure 6-3). From this menu, you can choose to reinstall the GRUB boot loader to reinstall GRUB directly. Or you may choose to select a shell in either your root device or your installer’s environment. When choosing this latter option, you can use the command grub-install /dev/sda to reinstall GRUB.

Figure 6-3. It’s relatively easy to reinstall GRUB from the rescue environment.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Working with the GRUB Boot Menu When GRUB runs, it displays a boot menu. (Remember to press the Esc key when booting your server in silent mode.) From the boot menu, you will normally have a choice between the three different sections that are defined in /boot/grub/menu.lst. Normally you will select the first option to boot the server. If you want to use your server for XEN virtualization, select the XEN option from the boot menu. (This option is available only if you selected the XEN software when installing your server. See Chapter 13 for more on XEN.) The failsafe option is the one you need if you run into trouble, and finally, you can select the Memory Check option if you suspect that you have problems with your server’s RAM.

■Note The failsafe option is more than just a single-user mode. A minimal number of services are loaded in single-user mode, but the kernel is loaded in the normal way. Selecting the failsafe option from the boot menu starts the single-user mode, but the kernel is also started with minimal options to increase chances that you can boot successfully.

If the default startup option from the GRUB menu is not good enough, select the item that you want to start and press the e key. You’ll next see a window like the one in Figure 6-4.

Figure 6-4. After pressing the e key, you can choose from more details when booting your server. You’ll now see the selected boot item in more detail. Every line in the selected item can be edited from this interface. From this interface, you can perform the following tasks:

171

172

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• Press b to boot: Use this option if you want to boot your computer with the selected settings. • Select a line and press the e key to edit the selected line. This option is very convenient if you know that you have made an error in a certain line and you want to fix it. • Press c to open the GRUB command line. This not-so-intuitive interface allows you to type GRUB-specific commands to tell your server what you want to do. If GRUB still is capable of showing you some boot options, you probably won’t use this option much. • Press o or O to open a new line. On this line, you can add new options that you want to pass to GRUB while starting your machine. For example, if you want to start your server in troubleshooting mode instead of its normal startup mode, type single to start singleuser mode. • Press d to remove a line from the menu. • Press Esc to return to the main menu. From there, you can press Enter to continue booting.

Upstart After GRUB, the kernel is loaded. In the old days, the kernel loaded the init process that read its configuration file /etc/inittab. Since Ubuntu 7.04, however, a new program is used instead. The Upstart program is responsible for the remainder of the boot procedure. To start your computer, it still uses a boot method that looks a lot like the one that was used in the old days. The most important part of Upstart is found in the /etc/event.d directory. It’s here that Upstart looks for a definition of all the jobs it has to start, and there’s a file for every job. Next, you’ll find a description of all the available jobs: • control-alt-delete: This job defines what should happen when a user presses the Ctrl+Alt+Del key sequence. The default behavior is that the system will reboot. If you don’t like that, open the /etc/event.d/control-alt-delete file and replace this shutdown command with something else. • logd: This job makes sure that the log process /sbin/logd is started. This process ensures that all log messages generated by daemons on your server can be logged. • rc0-rc6: A Linux computer uses the concept of runlevels, which are used to define what services have to be started when booting your server. The scripts with the names rc0 up to rc6 define what should happen in the corresponding runlevels. Typically, Ubuntu Server is started in runlevel 2. This master script makes sure that all services normally required in that runlevel are launched. Later in this chapter you’ll learn how to define what happens in a given runlevel. • rc-default: This script determines the default runlevel, which is normally runlevel 2 on Ubuntu Server. If you want to use something else for the default runlevel, you should create a file with the name /etc/inittab that contains the following line: id:N:initdefault:. In this line, N refers to the number of the runlevel that you want to activate.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• rcS: This script is used to ensure compatibility with System V startup scripts. Ubuntu Server still uses these old scripts to start services, and you can read in more detail how to configure them in the section “Runlevels” later in this chapter. • rcS-sulogin: Normally, single-user mode is used for troubleshooting, and no administrator password is asked for. Of course, this is a serious security issue, and some measures have to be taken. The rcS-sulogin service makes sure that the root password has to be provided every time the single-user mode is entered. • sulogin: In this script, the administrator can specify the message that a user should see when entering single-user mode. • tty1-tty6: On Ubuntu Server, virtual terminals are used. To activate a virtual terminal, the key sequences Ctrl+Alt+F1 up to Ctrl+Alt+F6 have to be used. The services files in /etc/event.d specify what needs to be done when activating one of these virtual terminals. If you want to have more than six virtual terminals, copy one of these files to (for example) a file with the name tty8 (never use tty7 because it is by default used for the graphical environment). Next, change the last line of this file to reflect the name of the TTY it is related to. See Listing 6-4 for an example. Listing 6-4. The TTY Files Specify What Should Happen on a Virtual Console root@ubuntu:~# cat /etc/event.d/tty1 # tty1 - getty # # This service maintains a getty on tty1 from the point the system is # started until it is shut down again. start start start start

on on on on

runlevel runlevel runlevel runlevel

2 3 4 5

stop on runlevel 0 stop on runlevel 1 stop on runlevel 6 respawn exec /sbin/getty 38400 tty1 As you have seen, the Upstart service activates services as specified in the different files in /etc/event.d. This is pretty much the same as what happened on older versions of Ubuntu Server that still used the init process. One of the most important tasks of Upstart is that it’s also responsible for starting all the services that are needed on your server. To do this, it uses the concept of runlevels.

173

174

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Runlevels The default runlevel on Ubuntu Server is runlevel 2, in which all the services that have to be started are referred to. Before entering runlevel 2, Ubuntu Server passes through runlevel S. In this runlevel, all the essential services that are always required on your server are started. The configuration of both works in more or less the same way. To understand the working of a runlevel, you need to understand two components: the service scripts and the scripts that execute these service scripts. All the service scripts are found in the /etc/init.d directory, and they are used to start fundamental services such as the mounting of file systems as well as network services like your Apache server. To specify which of these scripts have to be executed when starting your server, two runlevel-related directories are used. The first of these directories is /etc/rcS.d, and on a system that follows a default installation, the second of them is /etc/rc2.d. In the /etc/rcS.d directory, services are started that are always needed, whereas in the /etc/rc2.d directory, services are started that are specific to a given runlevel. To make sure that a service starts automatically during system initialization, a symbolic link is created in the /etc/rcS.d directory. The name of this link starts with an S, followed by a two-digit number, followed by the name of the script in /etc/init.d that the link refers to. All these links are processed when booting your server, and they are processed in alphabetical order. So S01blah is processed before S99blah. The same thing happens for the runlevel directories, except that when working with runlevels, there is an option to change the current runlevel. When changing a runlevel, some scripts may have to be started as well. To do this, more symbolic links are created. The name of these links starts with K, followed by a two-digit number. Listing 6-5 shows an example of the default runlevel 2. Listing 6-5. To Determine What Is Started and What Is Stopped in a Runlevel, Some Symbolic Links Are Processed root@ubuntu:/etc/rc2.d# ls -l total 4 -rw-r--r-- 1 root root 556 2007-04-10 17:46 README lrwxrwxrwx 1 root root 18 2007-07-29 07:34 S10sysklogd -> ../init.d/sysklogd lrwxrwxrwx 1 root root 15 2007-07-29 07:34 S11klogd -> ../init.d/klogd lrwxrwxrwx 1 root root 15 2007-07-29 07:36 S15bind9 -> ../init.d/bind9 lrwxrwxrwx 1 root root 23 2007-07-29 07:36\ S17mysql-ndb-mgm -> ../init.d/mysql-ndb-mgm lrwxrwxrwx 1 root root 19 2007-07-29 07:36 S18mysql-ndb -> ../init.d/mysql-ndb lrwxrwxrwx 1 root root 15 2007-07-29 07:36 S19mysql -> ../init.d/mysql lrwxrwxrwx 1 root root 17 2007-07-29 07:32 S20makedev -> ../init.d/makedev lrwxrwxrwx 1 root root 15 2007-07-29 07:36 S20rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 13 2007-07-29 11:44 S20ssh -> ../init.d/ssh lrwxrwxrwx 1 root root 13 2007-07-29 07:36 S89atd -> ../init.d/atd lrwxrwxrwx 1 root root 14 2007-07-29 07:36 S89cron -> ../init.d/cron lrwxrwxrwx 1 root root 17 2007-07-29 07:36 S91apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 18 2007-07-29 07:33 S99rc.local -> ../init.d/rc.local lrwxrwxrwx 1 root root 19 2007-07-29 07:33 S99rmnologin -> ../init.d/rmnologin

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

If you want to make sure that a given service is started automatically, it follows that you first need to make sure that it has a service script in /etc/init.d. If it does, you next need to make a symbolic link for this service. If it is a service that has to be started when your server is booting, you just need a start link in /etc/rcS.d. If it is a service that you want to be included in your server’s runlevels, you need to create a start link as well as a stop link in the directory of the default runlevel, which would be /etc/rc2.d in most cases. So let’s see how this works for the imaginary service blahd. 1. To include blahd in system startup, make sure that it has a start script in /etc/init.d. If blahd was developed to be used on either Debian or Ubuntu Linux, it will have such a script. Let’s say that the name of this script is /etc/init.d/blah. If you can write a decent Bash shell script, open the example script /etc/init.d/skeleton and change it to start the blah service instead of the default foo service. 2. If blahd is a nonessential service, you should include it in the default runlevel. Therefore, you’re going to create two symbolic links in /etc/rc2.d, and to put the service in the right place, you should first analyze its dependencies. If it depends on some other service to be started first, give it a relatively high number after the S, such as S50. If it doesn’t depend on anything, you can give it a relatively low number. The inverse is true for the kill scripts that make sure that the service is stopped once you quit the runlevel: scripts that depend on many other services but don’t have dependencies themselves get a low number; scripts that don’t depend on other services get a high number. 3. Now create the links. To create the start link, first use cd /etc/rc2.d and then ln -s ../init.d/blah S10blahd. Next, to create the kill link, use ln -s ../init.d/blah K90blahd. When restarting your server, the service will now be executed automatically.

■Tip When determining the proper load number for a script, on Ubuntu Server you can always assume that all device drivers are initialized, local file systems have been mounted, and networking is available after the S40 scripts have been processed. So in case of doubt, use S41 or higher.

Making Service Management Easier When reading the information about starting services in the preceding section, maybe you began to suspect that it’s not really easy. And you know what? You’re right. Even with the modern Upstart system, Ubuntu Server is still compatible with the old way of starting services (System V); you can use one of the many tools available for System V service management to make service configuration easier. One of these tools is sysv-rc-conf. Use apt-get install sysv-rc-conf to install it. Once installed, you can start it with the command sysv-rc-conf, and what follows is an interface similar to that shown in Figure 6-5. From this interface, you’ll see all the services available on your server. To make sure that a given service is started, move the arrow key to the right location in the runlevel columns for the runlevel in which you want the service started; then press the spacebar to select it. All required symbolic links will be automatically created for you.

175

176

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Figure 6-5. The sysv-rc-conf tool makes service management a lot easier.

■Tip Don’t worry about service management too much. You will find that after installing a package with apt-get, most service management tasks are accomplished automatically. So you just have to see whether

the required link is really created as well. Everything usually works just fine.

Managing Hardware One of the hardest challenges when working with Linux is making your hardware do what you want it to do. Many times, the real cause for not being able to get your hardware to work is the lack of the right drivers. Many hardware vendors still think that Windows is the only operating system on Earth, so they don’t offer any Linux drivers for their devices. This means that it’s up to the open source community to do the work. Often this goes very well, especially if the specifications of the hardware are clear. However, in some cases, hardware vendors think that their product is unique and therefore are unwilling to share their code specifications with the rest of the world, which makes it sometimes nearly (sometimes completely) impossible to produce the right drivers. In this subsection you’ll learn what you can do to get your hardware working. To begin with, you’ll have a look at the kernel and its capability to add load modules to get your hardware working. Related to the kernel, I’ll also talk about the initial RAM drive (initrd) and how you can configure it to load additional modules while booting. Next, I’ll talk about udev and the way it has changed hardware management on modern Linux distributions. At the end of this chapter, you’ll learn how the lspci and lsusb commands can be useful for finding out

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

what kind of hardware you are using. You’ll also learn how the sysfs file system can help you manage hardware.

Kernel Management The kernel is the heart of the operating system; it is the software layer that sits directly on top of the hardware and makes it possible to do something useful with that hardware. With Ubuntu Server, you are working with a default kernel in which certain functionality is enabled and other functionality is not. I’ll now explain how the kernel is organized and what you can do to tune it to your needs.

Working with Modules On all modern Linux distributions, kernels are modular—which means that the core of the operating system is in the kernel file itself, but lots of drivers that aren’t needed by default are dynamically loaded as modules. The idea of this modularity is an increased efficiency—if a driver is needed, its module is loaded; if it isn’t needed, the module isn’t loaded either. It’s really as simple as that. As an administrator, you’ll find that module management is an important task. With Ubuntu Server, modules are installed in the directory /lib/modules/`uname –r`. As you can see, command substitution is used in this directory name: the command uname –r gives the correct version of the current kernel; by using this command in the directory path, you can be sure always to refer to the right path in which kernel modules can be found. Under this directory, you can find a directory structure in which all modules are stored in an organized way, according to the type of module. You can recognize the kernel modules in this directory structure by their file name: all kernel modules have the extension .ko. You should be aware how modules are loaded. The good thing is that on a default installation, most of your hardware is detected automatically, and the required modules are loaded automatically as well. So in most cases there is no need to do anything. Sometimes, however, some tuning of the load process of modules is required. In the next subsection you can read how.

Loading Modules You can load modules in one of three methods: manually, from initrd, or by udev. Let’s see how this process works. Tuning initramfs As soon as your system boots, it immediately needs some modules, such as the modules necessary to access the root device on your server. These modules are loaded by the initial RAM file system (initrd), which is loaded from GRUB. Normally, this initial RAM drive is created automatically, and you don’t have to worry about it. However, you may need to tune your own initrd in some situations; in this case, the mkinitramfs command can be used.

177

178

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

■Note The correct way to refer to initrd is to call it the initial RAM file system (initramfs). In the old days, the initramfs was referred to as the initrd. It basically is the same thing, and you can use either term to refer to it.

To create your own initramfs, the first thing you should tune is the /etc/initramfstools/initramfs.conf file. This file is used to specify generic options that should be used on your initramfs, such as a time-out parameter that specifies how long you’ll have to interrupt the boot procedure. Normally, it’s not necessary to change anything in this file. Also, there is the /etc/ initramfs-tools/modules file, in which you refer to the modules that you want to be loaded automatically. Next, in the /etc/initramfs-tools/scripts directory you can create scripts that allow the mkinitramfs command to find the proper modules. Done all that? Then it’s time to run the mkinitramfs command to create your own initramfs. When using mkinitramfs, the command needs the option -o to specify the name of the output file it needs to create; for example, mkinitramfs -o newinitrd. Once created, it is a good idea to copy newinitrd to the /boot directory; everything your kernel needs to boot the system must be in this directory. Next, tune /boot/grub/menu.lst to make sure the new initramfs is included in one of the GRUB sections (be aware that in the menu.lst file it is referred to as initrd when doing so): 1. Open /boot/grub/menu.lst with your favorite editor. 2. Copy the default section that is used to boot your system in the file. This will result in this section occurring twice in the menu.lst file. 3. Change the title of the default boot section to something else (“test with new initrd” would be a decent name while you are still testing) and make sure the initrd line refers to the new initrd that you just created. This would result in something like the following lines: title root kernel initrd

Test with new initrd (hd0,0) /boot/vmlinuz-2.6.24-16-server root=/dev/sda1 ro quiet splash /boot/newinitrd

4. Reboot your server, and while rebooting, select the new GRUB menu item to test if the new initrd is working properly. If it does, change /boot/grub/menu.lst to make the test section permanent.

Loading Modules on Boot Normally, the kernel ensures that all modules you need when booting your server are loaded when the hardware that requires them is detected. In some situations, though, this doesn’t work out. If not, you can make sure that the module is loaded anyway by including it in the /etc/modules configuration file. The structure of this file is not complicated; just specify the names of all the modules that you want to load, one per line, and they will be loaded when you reboot your server. The following listing shows an example of the contents of this file:

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

root@RNA:/etc# cat modules # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. loop lp sbp2 fuse Need to add a new module? Make sure that it is listed in this file and it will be loaded when your server restarts. Loading Modules Manually Modules can be managed manually as well, which can be useful when testing new hardware devices. Here are the commands: • lsmod: This command displays a list of all currently loaded modules. In this list, you’ll also see the current status of the module. The output of lsmod is given in four columns (as can be seen in Listing 6-6). The first column provides the name of the module. The second column shows its size. In the third column, a 0 indicates that the module currently is not used. Everything greater than 0 indicates that the module is in use. The last column shows the name of other modules that require this module to be loaded. Listing 6-6. Output of lsmod root@ubuntu:/etc/init.d# lsmod Module Size Used by ipv6 273344 20 lp 12324 0 af_packet 23688 2 snd_ens1371 27552 0 gameport 16520 1 snd_ens1371 snd_ac97_codec 97952 1 snd_ens1371 ac97_bus 3200 1 snd_ac97_codec snd_pcm_oss 44416 0 snd_mixer_oss 17408 1 snd_pcm_oss snd_pcm 79876 3 snd_ens1371,snd_ac97_codec,snd_pcm_oss snd_seq_dummy 4740 0 snd_seq_oss 32896 0 snd_seq_midi 9600 0 snd_rawmidi 25472 2 snd_ens1371,snd_seq_midi snd_seq_midi_event 8448 2 snd_seq_oss,snd_seq_midi snd_seq 52464 6\ snd_seq_dummy,snd_seq_oss,snd_seq_midi,snd_seq_midi_event snd_timer 23684 2 snd_pcm,snd_seq

179

180

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

... capability commoncap

8192

5896 0 1 capability

• modprobe: If you want to load a module by hand, the modprobe command is the way to do it. The importance of this command is that it will do a dependency check. Some modules need another module to be present before they can do their job, and modprobe makes sure that these dependencies are fulfilled. To load the dependent modules, it looks in the configuration file modules.dep, which is created automatically by the depmod command (see later in this section). Loading a module with modprobe isn’t hard; if, for example, you want to load the module vfat by hand, just use the modprobe vfat command. In the early days, modprobe had an alternative: the insmod command. But insmod has the disadvantage that it doesn’t check for dependencies, so you probably shouldn’t use it anymore. • rmmod: An unused module still consumes system resources. It usually won’t be much more than 50 KB of system memory, but some heavy modules (such as the XFS module that offers support for the XFS file system) can consume up to 500 KB. On a system that is short on memory, this is a waste, and you can use rmmod followed by the name of the module you want to remove (for example, rmmod ext3). This command will remove the module from memory and free up all the system resources it was using. A more modern alternative for rmmod is the modprobe -r command. The major difference is that modprobe -r takes dependencies into consideration as well. • modinfo: Have you ever had the feeling that a module was using up precious system resources without knowing exactly what it was doing? Then modinfo is your friend. This command will show some information that is compiled in the module itself. As an example, you can see how it works on the pcnet32 network board driver in Listing 6-7. Especially for network boards, the modinfo command can be very useful because it shows you all the parameters the network board is started with (for instance, its duplex settings), which can be handy for troubleshooting. Listing 6-7. The modinfo Command Shows What a Module Is Used For myserver # modinfo pcnet32 root@ubuntu:/etc/init.d# modinfo pcnet32 filename: /lib/modules/2.6.20-15-server/kernel/drivers/net/pcnet32.ko license: GPL description: Driver for PCnet32 and PCnetPCI based etherboards author: Thomas Bogendoerfer srcversion: 8C4DDF304B5E88C9AD31856 alias: pci:v00001023d00002000sv*sd*bc02sc00i* alias: pci:v00001022d00002000sv*sd*bc*sc*i* alias: pci:v00001022d00002001sv*sd*bc*sc*i* depends: mii vermagic: 2.6.20-15-server SMP mod_unload 686 parm: debug:pcnet32 debug level (int) parm: max_interrupt_work:pcnet32 maximum events\ handled per interrupt (int)

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

parm: rx_copybreak:pcnet32 copy breakpoint for\ copy-only-tiny-frames (int) parm: tx_start_pt:pcnet32 transmit start point (0-3) (int) parm: pcnet32vlb:pcnet32 Vesa local bus (VLB) support (0/1) (int) parm: options:pcnet32 initial option setting(s) (0-15) (array of int) parm: full_duplex:pcnet32 full duplex setting(s) (1) (array of int) parm: homepna:pcnet32 mode for 79C978 boards (1\ for HomePNA, 0 for Ethernet, default Ethernet (array of int) root@ubuntu:/etc/init.d# • depmod: The depmod command is used to generate the modules dependency file in /lib/ modules/`uname -r`. The name of this file is modules.dep, and it simply contains a list of all dependencies that exist for modules on your system. As can be seen in Listing 6-7, modules normally know what dependencies they have (indicated by the depends field). The depmod command just analyzes this data and makes sure that the dependency file is up to date. There’s normally no need to run this command manually because it’s started automatically when your system boots. If, however, you’ve installed new kernel modules and you want to make sure the dependency file is up to date, run depmod manually.

The Role of the sysfs File System When trying to understand what happens with your hardware, you need to take the sysfs file system into consideration. This file system is created dynamically by the kernel, and it is mounted in the /sys directory. In this file system you can find configuration information that relates to the way your system loads modules. Consider the /sys/devices/pci0000:00 directory shown in Listing 6-8. You’ll find a subdirectory for each device that was found on the PCI bus. In the names of these subdirectories, the PCI ID is used. Listing 6-8. The /sysfs Keeps Information About Drivers that Are Used root@mel:/sys/devices/pci0000:00# ls 0000:00:00.0 0000:00:1a.1 0000:00:1c.2 0000:00:01.0 0000:00:1a.7 0000:00:1c.3 0000:00:03.0 0000:00:1b.0 0000:00:1c.4 0000:00:19.0 0000:00:1c.0 0000:00:1d.0 0000:00:1a.0 0000:00:1c.1 0000:00:1d.1

0000:00:1d.2 0000:00:1d.7 0000:00:1e.0 0000:00:1f.0 0000:00:1f.2

0000:00:1f.3 0000:00:1f.5 power uevent

It’s not hard to find out which device is using which PCI ID. To do this, you can use the lspci command. An example of usage of this command is shown in Listing 6-9. Listing 6-9. To Find Out What Device Is On What PCI ID, Use lspci root@mel:/sys/devices/pci0000:00# lspci 00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02) 00:01.0 PCI bridge: Intel Corporation 82P965/G965 PCI Express Root Port (rev 02) 00:03.0 Communication controller: Intel Corporation\ 82P965/G965 HECI Controller (rev 02) 00:19.0 Ethernet controller: Intel Corporation 82566DC\

181

182

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Gigabit Network Connection (rev 02) 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB UHCI Controller #5 (rev 02) 00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB2 EHCI Controller #2 (rev 02) 00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family)\ HD Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family)\ PCI Express Port 1 (rev 02) 00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family)\ PCI Express Port 2 (rev 02) 00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family)\ PCI Express Port 3 (rev 02) 00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family)\ PCI Express Port 4 (rev 02) 00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family)\ PCI Express Port 5 (rev 02) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family)\ USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2) 00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R)\ LPC Interface Controller (rev 02) 00:1f.2 IDE interface: Intel Corporation 82801H (ICH8 Family)\ 4 port SATA IDE Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02) 00:1f.5 IDE interface: Intel Corporation 82801H (ICH8 Family)\ 2 port SATA IDE Controller (rev 02) 01:00.0 VGA compatible controller: nVidia Corporation GeForce 8600 GTS (rev a1) 03:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6101\ single-port PATA133 interface (rev b1) 07:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd.\ RTL-8169 Gigabit Ethernet (rev 10) 07:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A\ IEEE-1394a-2000 Controller (PHY/Link) Suppose that you want to know more about the network card that uses the Intel chip set. At the top of the lspci output, you can find its PCI ID, which is 00:19.0. The complete configuration relating to this PCI ID is located in the sysfs file system inside the /sys/devices/ pci0000:00/0000:00:19.0 directory. You can now change to this directory to get an overview of the configuration that is stored for this device. You’ll find a set of configuration files that will

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

tell you exactly what the device is all about. For instance, you can find the driver file, which is a symbolic link and in this example refers to /sys/bus/pci/drivers/e1000 (see Listing 6-10). So this network board is using the e1000 driver! If you are having problems with the network board, this information might help you. For instance, you now know what driver to update to try solving its current problems. Listing 6-10. From the sys File System, You Get Information About the Drivers Your Hardware Uses root@mel:/sys/devices/pci0000:00/0000:00:19.0# total 0 -rw-r--r-- 1 root root 4096 2008-05-19 04:06 -r--r--r-- 1 root root 4096 2008-05-19 04:03 -rw-r--r-- 1 root root 256 2008-05-19 04:03 -r--r--r-- 1 root root 4096 2008-05-19 04:03 lrwxrwxrwx 1 root root 0 2008-05-19 02:24 -> ../../../bus/pci/drivers/e1000 -rw------- 1 root root 4096 2008-05-19 04:06 -r--r--r-- 1 root root 4096 2008-05-19 04:03 -r--r--r-- 1 root root 4096 2008-05-19 04:06 -r--r--r-- 1 root root 4096 2008-05-19 04:06 -rw-r--r-- 1 root root 4096 2008-05-19 04:06 drwxr-xr-x 3 root root 0 2008-05-19 02:24 drwxr-xr-x 2 root root 0 2008-05-19 04:06 -r--r--r-- 1 root root 4096 2008-05-19 04:03 -rw------- 1 root root 131072 2008-05-19 04:06 -rw------- 1 root root 4096 2008-05-19 04:06 -rw------- 1 root root 32 2008-05-19 04:06 lrwxrwxrwx 1 root root 0 2008-05-19 02:24 -r--r--r-- 1 root root 4096 2008-05-19 04:06 -r--r--r-- 1 root root 4096 2008-05-19 04:06 -rw-r--r-- 1 root root 4096 2008-05-19 02:24 -r--r--r-- 1 root root 4096 2008-05-19 04:03

ls -l broken_parity_status class config device driver\ enable irq local_cpus modalias msi_bus net power resource resource0 resource1 resource2 subsystem -> ../../../bus/pci subsystem_device subsystem_vendor uevent vendor

Installing Your Own Custom Kernel In the early days of Linux, it was often necessary to recompile your own kernel. Today this process is almost never necessary. Here’s why: • In the old days, drivers were included in the kernel. To add a driver, you had to recompile the kernel. Today’s drivers are modular; you just have to load them. In some situations, however, you need to recompile the driver, not the kernel itself. • In early Linux versions, to tune the Linux kernel, you had to recompile it. Now you can write parameters to tune the kernel directly to the /proc file system. For instance, to tell your server that it has to do packet forwarding between its network boards, you have to write the value 1 to the /proc/sys/net/ipv4/ip_forward file. You can do that by using the echo "1" > /proc/sys/net/ipv4/ip_forward command.

183

184

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

• Some Linux servers are supported by external parties, which don’t like you to change the kernel (you could even lose support when doing it). The kernels that are used on Ubuntu Linux (no matter what version of the operating system you’re using) are pretty close to the official Linux kernels as found on ftp.kernel.org. Ubuntu distinguishes between the Linux kernel (the so-called vanilla kernel) and the kernel that Ubuntu uses, which is installed automatically for the hardware platform that you are using. When using special software, such as XEN (see Chapter 13), an additional kernel may be installed as well. To use them, the GRUB boot menu will be automatically modified as necessary. You can create your own binary kernel from the kernel sources. This is referred to as compiling the kernel, and it’s a four-step procedure: 1. Install the kernel sources for your platform. 2. Configure the kernel using a command such as make menuconfig. 3. Build the kernel and all its modules. 4. Install the new kernel.

Installing the Kernel Source Files In some situations—if you want to build a custom kernel, for example—the kernel sources may have to be present. This is mainly the case if you need to be able to add new functionality to the kernel, such as to compile a module for a certain piece of hardware for which you have only the source code and no compiled version. To install the kernel sources, use the following procedure: 1. Use the command apt-cache search linux-source to see a list of all kernel sources that are suitable for your server. See Listing 6-11 for an example. 2. Now, as root, use the apt-get install command to install the sources that you want to use; for instance, use apt-get install linux-source-2.6.20. If this command causes an error message, run apt-get update before you start. 3. After downloading the kernel sources, an archive file is placed in /usr/src/. Now you need to extract this file by using tar -jxvf linux-source-2.6.20.tar.bz2. This command will create a subdirectory and put the new kernel sources in it. 4. To make sure that you don’t run into problems later, you now need to create a symbolic link to the directory that was just created in /usr/src. The name of this link is linux: ln -sf linux-source-2.6.20 linux. The kernel sources are now ready and waiting for you to do anything you want.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Listing 6-11. Before Installing Kernel Sources, Use apt-cache search to Find Out What Sources Are to Be Used on Your Server root@ubuntu:/# apt-cache search linux-source xen-source-2.6.16 - Linux kernel source for version 2.6.17 with Ubuntu patches linux-source - Linux kernel source with Ubuntu patches linux-source-2.6.20 - Linux kernel source for version 2.6.20 with Ubuntu patches

Configuring the Kernel You should compile your own kernel if some modifications to the default kernel are required or if some new functionality has to be included in your default kernel. This latter scenario is the more realistic because the default kernel on Ubuntu Server is flexible enough to meet most situations. Because of its modularity, it will do the right things in almost any circumstances. But, in general, you would want to recompile a kernel for four reasons: • You need access to a new type of hardware that isn’t yet supported by the kernel. This option is pretty rare now that most hardware is supported by loading kernel modules dynamically. • You need some specific feature that isn’t available in the default binary kernel yet. • You really need to strip the kernel to its essential components, removing everything that isn’t absolutely necessary. • You are running Ubuntu Server on old hardware that the default binary kernel doesn’t support. To tune a kernel, you need to create a new configuration for it, and you can choose among several methods of tuning what you do and don’t need in your kernel: • Run make config if you want to create the .config file that is needed to compile a new kernel completely by hand. The one drawback is that if you realize that you’ve made a mistake in the previous line after entering the next line, there’s no going back. • Use make oldconfig after installing patches to your kernel. This command makes sure that only the settings for new items are prompted for. • Use make menuconfig if you want to set all kernel options from a menu interface. • Use make xconfig to tune kernel options from an X-windows interface. If you are configuring kernel settings with make menuconfig, you’re working from a menu interface in which kernel functionality is divided into different sections. Each of these sections handles different categories of kernel options. For example, the File Systems section handles everything related to the available file systems, and Networking is the place to activate some obscure networking protocol. After opening the selection of your choice, you’ll get access to the individual parameters. For many of these parameters, you won’t necessarily see immediately what it’s used for. If so, use the Tab key to navigate to the Help button while the parameter is selected and then press Enter. You’ll be provided with a description of the selected option that in most cases will be very informative as to whether this is a reasonable option to use. Most options have three

185

186

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

possibilities. First, it can be selected with an * character, which means that the selected functionality is hard-coded in the kernel. Second, it can be selected with an M (not available for all options), which means that the selected component will be available as a kernel module. Third, you can choose not to select it at all.

Build the New Kernel After specifying what you need and what you don’t in the new kernel, you must build the new kernel. This involves running the gcc compiler to write all the changed kernel source files to one new kernel program file. To do this, you’ll use the make-kpkg kernel-image command. This reads all the changes that you made to your kernel and writes the new kernel to a Debian package with the name kernel-image-.deb, which is then placed in /usr/src.

Install the New Kernel After creating the Debian package with make-kpkg kernel-image, you have to install it. Use the command dpkg -i kernel-image-.deb, which not only installs the new kernel but also updates your GRUB configuration. Next, reboot your server, and the new kernel will be used.

Hardware Management with udev When earlier kernel modules were loaded by specifying them in /etc/modules.conf and later /etc/modprobe.conf, on a recent Ubuntu Server the udev system is the most common way of loading kernel modules; in fact, udev is the central service to handle hardware initialization on your server. It’s implemented as the daemon udevd, which is started at a very early stage in the boot process. When the kernel detects a device by some event that occurs on one of the hardware busses, it tells udev about the device. After receiving a signal from the kernel that a device has been added or removed, udev initializes this device. Then it creates the proper device files in the /dev directory. This all is a major improvement in the way devices are handled on a Linux system. In older versions of Linux, a device file existed for all devices that could possibly exist. Now, a device file is created only for devices that are really present. This is the task of udev. After initializing the device, udev informs all applications about the new device through the hardware abstraction layer (HAL). One problem with udev is that it loads at a stage when some devices have already been initialized. Think, for example, about the hard disk your system is working from. To initialize these devices properly, udev parses the sysfs file system, which is created in the directory /sys when the kernel is starting. This file system contains configuration parameters and other information about devices that have already been initialized. As an administrator, it is useful to know that udev can be monitored using the udevmonitor tool. Listing 6-12 shows what happens in the udevmonitor when a USB stick is plugged in the system.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Listing 6-12. In udevmonitor You Can See Exactly What Happens When a Device Is Connected to Your System SFO:/ # udevmonitor udevmonitor prints the received event from the kernel [UEVENT] and the event which udev sends out after rule processing [UDEV ] UEVENT[1158665885.090105] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6 UEVENT[1158665885.090506] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6/4-6:1.0 UEVENT[1158665885.193049] add@/class/usb_device/usbdev4.5 UDEV [1158665885.216195] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6 UDEV [1158665885.276188] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6/4-➥ 6:1.0 UDEV [1158665885.414101] add@/class/usb_device/usbdev4.5 UEVENT[1158665885.500944] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6/4-6.1 UEVENT[1158665885.500968] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6/4-➥ 6.1/4-6.1:1.0 UEVENT[1158665885.500978] add@/class/usb_device/usbdev4.6 UDEV [1158665885.604908] add@/devices/pci0000:00/0000:00:1d.7 /usb4/4-6/4-6.1 UEVENT[1158665885.651928] add@/module/scsi_mod UDEV [1158665885.652919] add@/module/scsi_mod UEVENT[1158665885.671182] add@/module/usb_storage UDEV [1158665885.672085] add@/module/usb_storage UEVENT[1158665885.672652] add@/bus/usb/drivers/usb-storage UDEV [1158665885.673200] add@/bus/usb/drivers/usb-storage UEVENT[1158665885.673655] add@/class/scsi_host/host0 UDEV [1158665885.678711] add@/devices/pci0000:00/0000:00:1d.7\ /usb4/4-➥ 6/4-6.1/4-6.1:1.0 UDEV [1158665885.854067] add@/class/usb_device/usbdev4.6 UDEV [1158665885.984639] add@/class/scsi_host/host0 UEVENT[1158665890.682084] add@/devices/pci0000:00/0000:00:1d.7/usb4/46/4-➥ 6.1/4-6.1:1.0/host0/target0:0:0/0:0:0:0 UEVENT[1158665890.682108] add@/class/scsi_device/0:0:0:0 UDEV [1158665890.858630] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-6/4 -6.1/4-➥ 6.1:1.0/host0/target0:0:0/0:0:0:0 UEVENT[1158665890.863245] add@/module/sd_mod UEVENT[1158665890.863971] add@/bus/scsi/drivers/sd UDEV [1158665890.864828] add@/module/sd_mod UDEV [1158665890.865941] add@/bus/scsi/drivers/sd UEVENT[1158665890.875674] add@/block/sda UEVENT[1158665890.875949] add@/block/sda/sda1

187

188

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

UEVENT[1158665890.880180] UDEV [1158665890.880180] UEVENT[1158665890.880207] UDEV [1158665890.906347] UDEV [1158665890.986931] UDEV [1158665891.084224] UDEV [1158665891.187120] UEVENT[1158665891.413225] UDEV [1158665891.413937] UEVENT[1158665891.427428] UDEV [1158665891.436849] UEVENT[1158665891.449836] UDEV [1158665891.451155] UEVENT[1158665891.467257] UDEV [1158665891.467795] UEVENT[1158665891.489400] UDEV [1158665891.491809]

add@/module/sg add@/class/scsi_device/0:0:0:0 add@/class/scsi_generic/sg0 add@/module/sg add@/class/scsi_generic/sg0 add@/block/sda add@/block/sda/sda1 add@/module/fat add@/module/fat add@/module/vfat add@/module/vfat add@/module/nls_cp437 add@/module/nls_cp437 add@/module/nls_iso8859_1 add@/module/nls_iso8859_1 mount@/block/sda/sda1 mount@/block/sda/sda1

The interesting part of this rather lengthy listing is that you can see exactly how udev interacts with the /sys file system that contains information about devices. First, the kernel detects the new device. At that moment, almost nothing is known about the nature of the device; udev sees only the PCI ID for the device (you can reveal these IDs with the lspci command as well). Based on this PCI information, udev can communicate with the device and it finds out what kernel modules need to be loaded to communicate with the device. You can see this in the lines where the scsi_mod and usb_storage modules are added. Based on that information, udev finds out that an sda and sda1 are present on the device. After finding that out, it can read the file system signature and load the proper modules for that as well; in this case, these are the fat and vfat modules. Once the proper file system drivers are loaded, some support modules can be used to read the files that are on the stick. Finally, the file system on the device is mounted automatically. As you can see, working with udev makes “automagic” loading of modules a lot less magical than it was. When working with udev, you can do some tweaking. For instance, imagine a server that has two network boards. One of those network boards would be known as eth0, whereas the other one would be available as eth1. You can use the configuration files that you’ll find in the /etc/udev/rules.d directory if you want to change the device names that are associated with the network cards. In this directory, you can find files that define rules for all hardware devices that support working with rules on your server. For instance, there is the file with the name 70-persistent-net.rules. You can see the contents of this file in Listing 6-13. Listing 6-13. The 70-persistent-net.rules File Determines How Your Network Boards Are Initialized root@mel:/etc/udev/rules.d# cat 70-persistent-net.rules # This file was automatically generated by the /lib/udev/write_net_rules # program run by the persistent-net-generator.rules rules file. # # You can modify it, as long as you keep each rule on a single line.

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

# PCI device 0x8086:0x104b (e1000) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",\ ATTR{address}=="00:19:d1:ed:82:07", ATTR{type}=="1",\ KERNEL=="eth*", NAME="eth0"

# PCI device 0x10ec:0x8169 (r8169) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",\ ATTR{address}=="00:0c:f6:3f:5d:bb", ATTR{type}=="1",\ KERNEL=="eth*", NAME="eth1" As you can see, this file matches the MAC addresses of your network boards with their device names. Imagine that this configuration is wrong, and you want to make sure that MAC address 00:0c:f6:3f:5d:bb is available as eth0 in the future. Just change the eth name of the device, and udev will do the rest for you. Likewise, configuration scripts are available for almost all other devices. Some of them tend to be pretty complex, though, and you shouldn’t touch them if you don’t have a deep understanding of shell scripting. Working with udev has one other major advantage as well: it ensures that your storage devices can be referred to by unique names. Unique names are names that relate to the devices, and they don’t ever change—no matter how and where the device is loaded. Normally a storage device gets its device name (/dev/sda and so on) based on the order that it is plugged into the system: the first storage device gets /dev/sda, the second storage device gets /dev/sdb, and so on. When activating a device, udev generates more than just the device name /dev/sda, and so on. For storage devices, some links are created in the directory /dev/disk. These links are in the following subdirectories and all contain a way to refer to the disk device: • /dev/disk/by-id: This subdirectory contains information about the device based on the vendor ID and the name of the device. Because this name never changes during the life of a device, you can use these device names as an alternative to the /dev/sda devices that may change in an uncontrolled way. The only disadvantage is that the /dev/disk/ by-id names are rather long. • /dev/disk/by-path: This subdirectory contains links with a name that is based on the bus position of the device. • /dev/disk/by-uuid: In this subdirectory, you can find links with a name that is based on the serial number (the UUID) of the device. Because the information in /dev/disk won’t change for a device the next time it is plugged in, you can create udev rules that work with that information and make sure that the same device name is always generated. The udev rules for storage devices are in /etc/udev/rules.d/ 60-persistent-storage.rules, in which you can create a persistent link that makes sure that a device is always initialized with the same device name. This solution can be used for disk devices and other devices as well. Just have a look at the files in /etc/udev/rules.d to see how the different device types are handled.

189

190

CHAPTER 6 ■ SETTING THE SYSTEM TO YOUR HAND

Summary In this chapter, you learned how to manage and customize your server. In the first part of this chapter, you learned how to manage processes with utilities such as top, ps, and kill. After that, you learned how to schedule processes to run in the future. Next, you read about the boot procedure, which may help you when troubleshooting or optimizing your server’s boot procedure. In the last part of this chapter, you read about the kernel and hardware management. In Chapter 7, you’ll learn how to create shell scripts on Ubuntu Server.

CHAPTER

7

Running It Any Way You Like An Introduction to Bash Shell Scripting K

nowing your way around commands on Linux is one thing. But if you really want to understand what is happening on Ubuntu Server, you must at least be able to read shell scripts. On Ubuntu Server, many things are automated with shell scripts. For example, the entire startup procedure consists of ingenious shell scripts that are tied together. As an administrator, it’s very useful to know how to do some shell scripting yourself. For these reasons, this chapter will give you an introduction to Bash shell scripting. After a short introduction, you’ll learn about the most important components that you’ll see in most shell scripts, such as iterations, functions, and some basic calculations. Notice that this chapter is meant to give a basic overview of the way that a shell script is organized and should help you write a simple shell script. It’s not meant to be a complete tutorial that discusses all elements that can be used in a script.

Before You Even Start If you know how to handle your Linux commands properly, you can perform magic. But imagine the magic when combining multiple Linux commands in a shell script. Shell scripting is a fine art, and you aren’t going to learn it by just studying this chapter: you need to do it yourself, again and again. Expect to spend a few frustrating hours trying it without success. You should know that that is only part of the fun because you’ll get better bit by bit. It takes practice, though. I hope you’ll find the examples in this chapter inspiring enough to start elaborating on them, improving on them, and converting them to your own purposes.

To Script or Not to Script? Before you start writing shell scripts, you should ask whether a shell script is really the best solution. In many cases, other approaches are available and maybe even preferable. Instead of using the Bash shell as your scripting language, you can use Perl or write a complete program in C. Each of these solutions has advantages and disadvantages. 191

192

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

That said, Bash offers some important advantages as a scripting language: • Bash scripts are relatively easy to understand and to write. • You don’t have to compile a Bash script before you can start using it. The only thing you need on the computer where you will run your shell script is the shell for which the script is written. The Bash shell is omnipresent, so it’s no problem performing tasks with your script on other Linux computers as well. • A shell script is platform independent. You can run the same script on a Solaris machine, a Power PC, or an Intel machine. It just doesn’t matter. • Although Bash is almost always present, you can get even greater portability by using the Bourne shell (/bin/sh) instead of Bash. Bourne shell scripts run on any flavor of Linux and even different brands of UNIX, without the need to install anything else on your server. The most important disadvantage of using Bash to create your script, however, is that it is relatively slow. It’s slow because the shell always has to interpret the commands that it finds in the script before it can do its job. For contrast, a C program is compiled code, optimized to do its work on the hardware of your computer directly and therefore is much faster. Of course, it’s also much more difficult to create such a program.

What Shell? Many shells are available for Linux. When writing a shell script, you should be aware of that and choose the best shell for your script. A script written for one shell will not necessarily run on another. Bash is the best choice to write shell scripts on Linux. Bash is compatible with the UNIX Bourne shell /bin/sh, which has been used on UNIX since the 1970s. The good thing about that compatibility is that a script that was written to run on /bin/sh will work on Bash as well. However, the opposite is not necessarily true because many new features have been added to Bash that don’t exist in the traditional UNIX Bourne shell. On Ubuntu, dash is used as the default shell. You can see this in most system scripts that call #!/bin/sh, which is a link to /bin/dash, on the first line of the script. However, because I have had problems with dash and scripting on several occasions, I recommend using Bash for scripting. You can ensure that Bash is used by including #!/bin/bash on the first line of each script you create. You’ll likely occasionally encounter Linux shells other than Bash or dash. The most important of these includes the Korn shell (/bin/ksh), which is the default shell on Sun Microsystems’ Solaris. An open source derivative of that shell is available as the Public Domain Korn Shell /bin/pdksh. Another popular shell is the C shell, which on Linux exists as /bin/tcsh. The C shell is especially popular among C programmers because it has a scripting language that closely resembles the C programming language. You’ll sometimes encounter C shell users in a Linux environment. The Korn shell, however, is not used often in Linux environments because almost all its important features are also offered by Bash. Both the Korn shell and the C shell are incompatible with Bash, and this incompatibility could prevent you from running a C shell script in a Bash environment. However, there is a solution, and that’s to include the so-called shebang in your shell script. This is an indicator of the program that must be used when executing the script. The shebang consists of the pound sign (#), followed by an exclamation mark, followed by the name of the required command

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

interpreter. If the program that is referred to by the shebang is present on your system, the script will run, no matter what shell environment you’re currently in as a user. Listing 7-1 shows a script that starts with a shebang. Listing 7-1. Shell Scripts Should Always Start with the Shebang #!/bin/bash # # myscript [filename] # # Use this script to....

Basic Elements of a Shell Script Some elements should occur in all shell scripts. First, as you’ve just read, every shell script should start with the shebang. After this, it’s a good idea to add some comment lines to explain what the script is for. A comment line starts with a pound sign, which ensures that the line is not interpreted by the shell. Of course, how you create your scripts is entirely up to you, but starting every script with some comment that explains how to use the script will make your scripts much easier to use. It’s really a matter of perspective: you know exactly what you’re doing at the moment you’re writing the script, but months or years later you might have forgotten what your shell script is all about. The first line in a good comment shows the exact syntax to be used for launching it. After the syntax line, it’s normally a good idea to explain in two or three lines what exactly your script is doing. Such an explanation makes it much easier for others to use your script the right way. If the script starts growing, you might even add comments at other places in the script. It’s not a bad idea to start every new chunk of code with a short comment, just to explain what it’s doing. One day you’ll be glad that you took the few extra seconds to add a comment or two. Apart from the comments, your script naturally includes some commands. All legal commands can be used, and you can invoke Bash internal commands as well as work with external commands. An internal command is loaded into memory with Bash and therefore can execute very fast. An external command is a command that is somewhere on disk, and this is its main disadvantage: it needs to be loaded first and that takes time. Listing 7-2 is a shell script that although rather simple, still includes all the basic elements. Listing 7-2. Example of a Simple Shell Script That Contains All Basic Elements #!/bin/bash # # This is just a friendly script. It echoes "Hello World" to the person # who runs it. # # Usage: hello # echo 'Hello World!' exit 0

193

194

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

In the example script, you can see some things happening. After the comment, the echo command is used to greet the user who runs the script. Notice that the text to be echoed to the screen is placed between single quotes. These are also called “strong quotes,” and they make sure that the shell does not actually interpret anything that appears between them. In this example, it’s a good idea to use them because the exclamation mark has a special meaning for the shell. Also note that after the successful termination of the script, the exit 0 command is used. This command generates the so-called exit status of the script; it tells the parent shell whether the script executed successfully. Normally, the exit status 0 is used to indicate that everything went well. If some problems were encountered executing the script, an exit status value of 1 can be used. Any other exit status can be used at the discretion of the programmer. Using more than just 0 and 1 as values for exit status can make troubleshooting much easier. Using an exit status is important in more complex shell scripts because you can decide that something else needs to happen based on the success or failure of your script.

■Tip Did you know that you can request the exit status of the last command executed by the shell? Typing echo $? displays the exit status of that command as a numerical value.

Making It Executable Now that you created your shell script, it’s time to do something with it. You can execute it with several different options: • Activate it as an argument of your shell. • “Source” the script. • Make it executable and run it. If you just need to check that the script works, the easiest way to test it is as a shell argument. To do this, you have to start a new shell that starts the script for you. If the name of your script is hello, you can start the script with the bash hello command. This method starts a subshell and executes the script from there. If a variable is set in this subshell, it’s available within that subshell only, not in the parent shell. If you want to set a variable from a shell script and make sure that that variable is available in the parent shell as well, use the source command to run the shell script. You’ll learn more about variables later in “Changing Variables.”

■Tip Want a variable that’s set in a script to be available in all subshells? Use the export command when defining the variable. However, there’s no way to define a variable in a subshell that will also be set in the parent shell.

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

The second way to execute a script is with the source command. This command is referred to by entering a dot, followed by a space and the name of the script. For example, the script with the name hello can be started with . hello. The important difference with the source command is that no subshell is started, and the script runs directly from the current shell. The result is that all variables that are defined when running the script are also available after running the script. This can be both useful and confusing. The source method is often used to include another script in a generic script. In this other script, for example, some system variables are set. Listing 7-3 shows how this works in the script that starts networking: /etc/init.d/networking. As you can see in about the approximate middle of the listing, the . /lib/lsb/init-functions command is included to set some generic functions that should be used in this script. Listing 7-3. The Sourcing Method Is Used to Include Scripts Containing Variables in Many Startup Scripts #!/bin/sh -e ### BEGIN INIT INFO # Provides: # Required-Start: # Required-Stop: # Default-Start: # Default-Stop: ### END INIT INFO

networking mountkernfs ifupdown $local_fs ifupdown $local_fs S 0 6

PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin" [ -x /sbin/ifup ] || exit 0 . /lib/lsb/init-functions

case "$1" in start) log_action_begin_msg "Configuring network interfaces" type usplash_write >/dev/null 2>/dev/null && usplash_write "TIMEOUT 120" || true if [ "$VERBOSE" != no ]; then if ifup -a; then networking The last and possibly the most frequently used method to run a script is to make it an executable first. Do this by adding the execute permission to your script, as in the following command: chmod +x hello Next, you can simply run the script: ./hello

195

196

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

Notice that in this example the script is executed as ./hello, not just hello (assuming that the script is in the current directory). This is because you need to indicate that the script must run from the current directory. As a default security feature, no Linux shell looks for executable code in the current directory. Excluding the current directory from the search path ensures that a user or an administrator who runs a command always runs the proper command from the search path, not some rogue command that was installed in the current directory. Without the ./, Bash would search for hello in its current PATH setting and would probably not find it.

■Note The shell PATH variable is used to specify a list of directories that should always be searched for executable files. You can see its contents by using the echo $PATH command.

One last remark before diving into real scripting: always be careful about what you name your script and try to avoid names that already exist. For example, you might be tempted to use test as the name of a test script. This would, however, conflict with the test command that’s installed on all Linux systems by default. Want to see whether the name of your script is already used by some other command? Use the which command to search all directories set in the PATH variable for a binary with the name you’ve entered. Listing 7-4 shows the result of this command issued for the test command. Listing 7-4. Using which to Check Whether a Command with a Given Name Already Exists root@RNA:/# which test /usr/bin/test

Making a Script Interactive It’s cool if your script can execute a list of commands, but it’ll be much better if you can make it interactive. This way, the script can ask a user for input, and the user decides how the script should be run. To make a script interactive, use the read command followed by the name of a variable. This variable is used as a label to the input of the user, but the great thing is that you can use it later in the script to check exactly what the user entered. Listing 7-5 is an example of an interactive script. You’ll also learn a new method to display script output on the screen. Listing 7-5. Making Your Script Interactive #!/bin/bash # # Send a message to the world # # Usage: ./hello cat echo $BLAH 2. 3. sander@linux %> echo ${BLAH:-variable is empty} 4 variable is empty 5. sander@linux %> echo $BLAH 6. 7. sander@linux %> echo ${BLAH=value} 8. value 9. sander@linux %> echo $BLAH 10. value 11. sander@linux %> BLAH= 12. sander@linux %> echo ${BLAH=value} 13. 14. sander@linux %> echo ${BLAH:=value} 15. value 16. sander@linux %> echo $BLAH 17. value 18. sander@linux %> echo ${BLAH:+sometext} 19. sometext The example of Listing 7-13 starts with the echo $BLAH command, which reads the variable BLAH and shows its current value. Because BLAH doesn’t have a value yet, nothing is shown in line 2. Next, a message is defined in line 3 that should be displayed if BLAH is empty. As you can see, the message is displayed in line 4. However, this doesn’t assign a value to BLAH, which you see in lines 5 and 6 where the current value of BLAH is asked again. In line 7, BLAH finally gets a value, which is displayed in line 8. The shell remembers the new value of BLAH, which you can see in lines 9 and 10 where the value of BLAH is referred to and displayed. In line 11, BLAH is redefined but it gets a null value. The variable still exists; it just has no value here. This is demonstrated when echo ${BLAH=value} is used in line 12; because BLAH has a null value at that moment, no new value is assigned. Next, the construction echo ${BLAH:=value} is used to assign a new value to BLAH. The fact that BLAH really gets a value from this is shown in lines 16

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

and 17. Finally, the construction in line 18 is used to display sometext if BLAH currently does have a value. Notice that this doesn’t change anything to the value that is assigned to BLAH at that moment; sometext just indicates that it has a value and that’s all.

Pattern-Matching Operators You’ve just seen how substitution operators can be used to do something if a variable does not have a value. You can consider them a rather primitive way of handling errors in your script. A pattern-matching operator can be used to search for a pattern in a variable and modify the variable if that pattern is found. This can be very useful because it allows you to define a variable exactly the way you want. For example, think of the situation in which a user enters a complete pathname of a file, but only the name of the file itself (without the path) is needed in your script. The pattern-matching operator is the way to change this. Pattern-matching operators allow you to remove part of a variable automatically. Listing 7-14 is an example of a script that works with pattern-matching operators. Listing 7-14. Working with Pattern-Matching Operators #!/bin/bash # # script that extracts the file name from a file name that includes the complete path # usage: stripit filename=${1##*/} echo "The name of the file is $filename" When executed, the script shows the following result: sander@linux %> ./stripit /bin/bash the name of the file is bash Pattern-matching operators always try to locate a given string. In this case, the string is */. In other words, the pattern-matching operator searches for a /, preceded by another character. In this pattern-matching operator, ## is used to search for the longest match of the provided string, starting from the beginning of the string. So, the pattern-matching operator searches for the last / that occurs in the string and removes it and everything that precedes the / as well. You might ask how the script comes to remove everything in front of the /. It’s because the pattern-matching operator refers to */ and not to /. You can confirm this by running the script with /bin/bash/ as an argument. In this case, the pattern that’s searched for is in the last position of the string, and the pattern-matching operator removes everything. This example explains the use of the pattern-matching operator that looks for the longest match. By using a single #, you can let the pattern-matching operator look for the shortest match, again starting from the beginning of the string. If, for example, the script in Listing 7-14 used filename=${1#*/}, the pattern-matching operator would look for the first / in the complete file name and remove that and everything before it. You should realize that in these examples the * is important. The pattern-matching operator ${1#*/} removes the first / found and anything in front of it. The pattern-matching

203

204

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

operator ${1#/} removes the first / in $1 only if the value of $1 starts with a /. However, if there’s anything before the /, the operator will not know what to do. These examples showed how a pattern-matching operator is used to start searching from the beginning of a string. You can start searching from the end of the string as well. To do so, a % is used instead of a #. This % refers to the shortest match of the pattern, and %% refers to its longest match. The script in Listing 7-15 shows how this works. Listing 7-15. Using Pattern-Matching Operators to Start Searching at the End of a String #!/bin/bash # # script that isolates the directory name from a complete file name # usage: stripdir dirname=${1%%/*} echo "The directory name is $dirname" While executing, you’ll see that this script has a problem: sander@linux %> ./stripdir /bin/bash The directory name is As you can see, the script does its work somewhat too enthusiastically and removes everything. Fortunately, this problem can be solved by first using a pattern-matching operator that removes the / from the start of the complete file name (but only if that / is provided) and then removing everything following the first / in the complete file name. The example in Listing 7-16 shows how this is done. Listing 7-16. Fixing the Example from Listing 7-15 #!/bin/bash # # script that isolates the directory name from a complete file name # usage: stripdir dirname=${1#/} dirname=${1%%/*} echo "The directory name is $dirname" As you can see, the problem is solved by using ${1#/}. This construction starts searching from the beginning of the file name to a /. Because no * is used here, it looks for a / only at the very first position of the file name and does nothing if the string starts with anything else. If it finds a /, it removes it. So, if a user enters usr/bin/passwd instead of /usr/bin/passwd, the ${1#/} construction does nothing at all. In the line after that, the variable dirname is defined again to do its work on the result of its first definition in the preceding line. This line does the real work and looks for the pattern /*, starting at the end of the file name. This makes sure that everything after the first / in the file name is removed and that only the name of the top-level directory is echoed. Of course, you can easily edit this script to display the complete path of the file: just use dirname=${dirname%/*} instead.

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

So, to make sure that you are comfortable with pattern-matching operators, the script in Listing 7-17 gives another example. This time, though, the example does not work with a file name, but with a random text string. Listing 7-17. Another Example with Pattern Matching #!/bin/bash # # script that extracts the file name from a file name that includes the complete path # usage: stripit BLAH=babarabaraba echo BLAH is $BLAH echo 'The result of echo 'The result of echo 'The result of echo 'The result of

##ba is '${BLAH##*ba} #ba is '${BLAH#*ba} %%ba is '${BLAH%ba*} %ba is '${BLAH%%ba*}

When running it, the script gives the result shown in Listing 7-18. Listing 7-18. The Result of the Script in Listing 7-17 root@RNA:~/scripts# ./pmex BLAH is babarabaraba The result of ##ba is The result of #ba is barabaraba The result of %%ba is babarabara The result of %ba is root@RNA:~/scripts#

Performing Calculations in Scripts Bash offers some options that allow you to perform calculations from scripts. Of course, you’re not likely to use them as a replacement for your spreadsheet program, but performing simple calculations from Bash can be useful. For example, you can use calculation options to execute a command a number of times or to make sure that a counter is incremented when a command executes successfully. The script in Listing 7-19 provides an example of how counters can be used. Listing 7-19. Using a Counter in a Script #!/bin/bash counter=1 while true do counter=$((counter + 1)) echo counter is set to $counter done

205

206

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

As you can see, this script uses a construction with while (which is covered in more detail in the “Using while” section). The construction is used to execute a command as long as a given condition is met. In this example, the condition is simple: you must be able to execute the true command successfully. This won’t be a problem: the name of the command is true because it always executes successfully. That is, true always gives an exit status of 0, which tells the shell that it has executed with success, just like the false command always gives the exit status of 1. What has to happen if the condition is met is specified between the do and done. First, the line counter=$((counter + 1)) takes the current value of the variable counter (which is set in the beginning of the script) and increments that by 1. Next, the value of the variable counter is displayed with the line echo counter is set to $counter. Once that’s happened, the condition is checked again and the command is executed again as well. The result of this script will be that you see a list of numbers on your screen that’s updated very quickly. Does it go too fast? Just add a line with the command sleep 1 in the loop. Now the calculation of the new value of counter is performed only once per second. Although the previous example explains how a simple calculation can be performed from a script, it isn’t very useful. Listing 7-20 provides a more useful one. I once had to test the durability of USB sticks for a computer magazine. As you have probably heard, some people think that the life of flash storage is limited. After a given number of writes, according to some people, the stick dies. Such a claim called for a test, which can be performed through the following shell script with the name killstick: Listing 7-20. Script to Test USB Sticks #!/bin/bash # # Script to test USB sticks # # usage: killstick # counter=0 while cp /1MBfile $1 do sync rm -rf $1/1MBfile sync counter=$((counter + 1)) echo Counter is now set to $counter done echo Your stick is now non-functional The script again starts with a little explanation of how it works. To run this script, you first need to mount the stick on a certain mount point, and this mount point needs to be declared by specifying it as an argument to the script. Next, a while loop is started. The command that needs to execute successfully is cp /1MBfile $1. You can use this script on a stick of any size, but before starting it, make sure that all the available disk space on the stick is used—with the exception of 1 MB. Next, create a file with a size of 1 MB (or just a little smaller). This way you’ll

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

make sure that the controller of the stick isn’t playing any tricks on you, and the write always occurs at the same spot. As long as the file copy is successful, the commands in the do loop are executed. First, the file is synchronized to the stick using the sync command to make sure that it isn’t just kept somewhere in memory. Next, it’s immediately removed again, and this removal is synchronized to the physical storage media. Finally, the calculation is used again to increment the counter variable by 1. This continues as long as the file can be copied successfully. When copying fails, the while loop is terminated and the echo Your stick is now non-functional command is displayed, thus allowing you to know exactly how often the file could be copied to the stick.

■Note Would you like to know how many writes it takes to kill a stick completely? As it turns out, flash memory has improved enormously over the last few years, and you can expect the memory chip to support at least 100,000 writes. In many cases, however, more than 1,000,000 writes can be performed without any problem.

So far, we’ve dealt with only one method to do script calculations, but you have other options as well. First, you can use the external expr command to perform any kind of calculation. For example, the following line produces the result of 1 + 2: sum=`expr 1 + 2`; echo $sum As you can see, a variable with the name sum is defined, and this variable gets the result of the command expr 1 + 2 by using command substitution. A semicolon is then used to indicate that what follows is a new command. After the semicolon, the command echo $sum shows the result of the calculation. The expr command can work with addition, and other types of calculation are supported as well. Table 7-2 summarizes the options. Table 7-2. expr Operators

Operator

Meaning

+

Addition ( 1 + 1 = 2 )

-

Subtraction ( 10 – 2 = 8 )

/

Division ( 10 / 2 = 5 )

*

Multiplication ( 3 * 3 = 9 )

%

Modulus—calculates the remainder after division; this works because expr can handle integers only ( 11 % 3 = 2 )

When working with these options, you’ll see that they all work fine with the exception of the multiplication operator *. Using this operator results in a syntax error: linux: ~> expr 2 * 2 expr: syntax error

207

208

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

This seems curious, but can be easily explained. The * has a special meaning for the shell, as in ls -l *. When the shell parses the command line, it interprets the * (you don’t want it to do that here). To indicate that the shell shouldn’t touch it, you have to escape it. Therefore, change the command as follows: expr 2 \* 2 Alternatively, you could have escaped the * with single quotes by using the following command: expr 2 '*' 2 Another way to perform some calculations is to use the internal command let. Just the fact that let is internal makes it a better solution than the external command expr: it can be loaded from memory directly and doesn’t have to come all the way from your computer’s hard drive. Using let, you can make your calculation and apply the result directly to a variable, as in the following example: let x="1 + 2" The result of the calculation in this example is stored in the variable x. The disadvantage of working this way is that let has no option to display the result directly as can be done when using expr. For use in a script, however, it offers excellent capabilities. Listing 7-21 shows a script in which let is used to perform calculations. Listing 7-21. Performing Calculations with let #!/bin/bash # # usage: calc $1 $2 $3 # $1 is the first number # $2 is the operator # $3 is the second number let x="$1 $2 $3" echo $x If you think that we’ve now covered all methods to perform calculations in a shell script, you’re wrong. Listing 7-22 shows another method that you can use. Listing 7-22. Another Way to Calculate in a Bash Shell Script #!/bin/bash # # usage: calc $1 $2 $3 # $1 is the first number # $2 is the operator # $3 is the second number x=$(($1 $2 $3)) echo $x

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

You saw this construction when you read about the script that increases the value of the variable counter. Note that the double pair of parentheses can be replaced by one pair of square brackets instead, assuming that the preceding $ is present.

Using Flow Control Up until now, you haven’t read much about the way in which the execution of commands can be made conditional. The technique for enabling this in shell scripts is known as flow control. Bash offers many options to use flow control in scripts: • if: Use if to execute commands only if certain conditions were met. To customize the working of if some more, you can use else to indicate what should happen if the condition isn’t met. • case: Use case to work with options. This allows the user to further specify the working of the command when he runs it. • for: This construction is used to run a command for a given number of items. For example, you can use for to do something for every file in a specified directory. • while: Use while as long as the specified condition is met. For example, this construction can be very useful to check whether a certain host is reachable or to monitor the activity of a process. • until: This is the opposite of while. Use until to run a command until a certain condition has been met. The following subsections cover flow control in more detail. Before going into these details, however, you can first read about the test command. This command is used to perform many checks to see, for example, whether a file exists or whether a variable has a value. Table 7-3 shows some of the more common test options. For a complete overview, consult its man page. Table 7-3. Common Options for the test Command

Option

Use

test -e $1

Checks if $1 is a file, without looking at what particular kind of file it is.

test -f $1

Checks if $1 is a regular file and not (for example) a device file, a directory, or an executable file.

test -d $1

Checks if $1 is a directory.

test -x $1

Checks if $1 is an executable file. Note that you can test for other permissions as well. For example, -g would check to see if the SGID permission (see Chapter 5) is set.

test $1 -nt $2

Controls if $1 is newer than $2.

test $1 -ot $2

Controls if $1 is older than $2.

test $1 -ef $2

Checks if $1 and $2 both refer to the same inode. This is the case if one is a hard link to the other (see Chapter 4 for more on inodes).

test $1 -eq $2

Sees if the integers $1 and $2 are equal. Continued

209

210

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

Table 7-3. Continued

Option

Use

test $1 -ne $2

Checks if the integers $1 and $2 are not equal.

test $1 -gt $2

Gives true if integer $1 is greater than integer $2.

test S1 -lt $2

Gives true if integer $1 is less than integer $2.

test $1 -ge $2

Sees if integer $1 is greater than or equal to integer $2.

test $1 -le $2

Checks if integer $1 is less than or equal to integer $2.

test -z $1

Checks if $1 is empty. This is a very useful construction to find out if a variable has been defined.

test $1

Gives the exit status 0 if $1 is defined.

test $1=$2

Checks if the strings $1 and $2 are the same. This is most useful to compare the value of two variables.

test $1 != $2

Sees if the strings $1 and $2 are not equal to each other. You can use ! with all other tests to check for the negation of the statement.

You can use the test command in two ways. First, you can write the complete command, as in test -f $1. This command, however, can be rewritten as [ -f $1 ]. Most of the time you’ll see the latter option only because people who write shell scripts like to work as efficiently as possible.

Using if...then...else Possibly the classic example of flow control consists of constructions that use if...then... else. Especially if used in conjunction with the test command, this construction offers various interesting possibilities. You can use it to find out whether a file exists, if a variable currently has a value, and much more. Listing 7-23 provides an example of a construction with if...then...else that can be used in a shell script. Listing 7-23. Using if to Perform a Basic Check #!/bin/bash if [ -z $1 ] then echo You have to provide an argument with this command exit 1 fi echo the argument is $1 The simple check from the Listing 7-23 example is used to see if the user who started your script provided an argument. If he or she didn’t, the code in the if loop becomes active, in which case it displays the message that the user needs to provide an argument and then terminates the script. If an argument has been provided, the commands within the loop aren’t executed, and the script will run the line echo the argument is $1, and in this case echo the argument to the user’s screen.

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

Also notice how the syntax of the if construction is organized. First, you have to open it with if. Then, separated on a new line (or with a semicolon), then is used. Finally, the if loop is closed with an fi statement. Make sure that all those ingredients are used all the time or your loop won’t work.

■Note You can use a semicolon as a separator between two commands. So ls;

who would first execute

the command ls and then the command who.

The example in Listing 7-23 is rather simple, and it’s also possible to make if loops more complex and have them test for more than one condition. To do this, use else or elif. By using else within the loop, you can make sure that something happens if the condition is met, but it allows you to check another condition if the condition is not met. You can even use else in conjunction with if (elif) to open a new loop if the first condition isn’t met. Listing 7-24 is an example of the latter construction. Listing 7-24. Nesting if Loops if [ -f $1 ] then echo "$1 is a file" elif [ -d $1 ] echo "$1 is a directory" else echo "I don't know what \$1 is" fi In this example, the argument that was entered when running the script is checked. If it is a file (if [ -f $1 ]), the script tells the user that. If it isn’t a file, the part under elif is executed, which basically opens a second loop. In this second loop, the first test performed is to see if $1 is perhaps a directory. Notice that this second part of the loop becomes active only if $1 is not a file. If $1 isn’t a directory either, the part after else is run, and the script reports that it has no idea what $1 is. Notice that for this entire construction, only one fi is needed to close the loop. You should know that if .. then ... else constructions are used in two different ways. You can write out the complete construction as in the previous examples, or you can use constructions that use && and ||. These so-called separators are used to separate two commands and establish a conditional relationship between them. If && is used, the second command is executed only if the first command is executed successfully (in other words, if the first command is true). If || is used, the second command is executed only if the first command isn’t true. So, with one line of code, you can find out if $1 is a file and echo a message if it is: [ -f $1 ] && echo $1 is a file Note that this can be rewritten differently as well: [ ! -f $1 ] || echo $1 is a file

211

212

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

■Note This example only works as a part of a complete shell script. Listing 7-25 shows how the example from Listing 7-24 is rewritten if you want to use this syntax.

In case you don’t quite follow what is happening in the second example: it performs a test to see if $1 is not a file. (The ! is used to test if something is not the case.) Only if the test fails (which is the case if $1 is indeed a file), it executes the part after the || and echoes that $1 is a file. Let’s have a look (see Listing 7-25) at how you can rewrite the script from Listing 7-24 with the && and || tests. Listing 7-25. The Example from Listing 7-24 Rewritten with && and || ([ -z $1 ] && echo please provide an argument; exit 1) || (([ -f $1 ] && echo $1 is\ a file) || ([ -d $1 ] && echo $1 is a directory || echo I have no idea what $1 is))

■Note You’ll notice in Listing 7-25 that I used a \ at the end of the line. This slash makes sure that the carriage return sign at the end of the line is not interpreted and is used only to make sure that you don’t type two separated lines. I’ve used the \ for typographical reasons only. In a real script, you’d just put all code on one line (which wouldn’t fit on these pages without breaking it, as I’ve had to do). I’ll use this convention in some later scripts as well.

If you understand what the example script from Listing 7-24 does, it is not really hard to understand the script in Listing 7-25 because it does the same thing. However, you should be aware of a few things. First, I added a [ -z $1 ] test to give an error if $1 is not defined. Next, the example in Listing 7-25 is all on one line. This makes the script more compact, but it also makes it a little harder to understand what is going on. I used brackets to increase the readability a little bit and also to keep the different parts of the script together. The parts between brackets are the main tests, and within these main tests some smaller tests are used as well. Let’s have a look at some other examples with if ... then ... else. Consider the following line, for example: rsync -vaze ssh --delete /srv/ftp 10.0.0.20:/srv/ftp || echo "rsync failed" | mail [email protected] Here, the rsync command tries to synchronize the content of the directory /srv/ftp with the content of the same directory on some other machine. If this succeeds, no further evaluation of this line is attempted. If something happens, however, the part after the || becomes active and makes sure that user [email protected] gets a message. Another more complex example could be the following script that checks whether available disk space has dropped below a certain threshold. The complex part lies in the sequence of pipes used in the command substitution:

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

if [ `df -m /var | tail -n1 | awk '{print $4} '` -lt 120 ] then logger running out of disk space fi The important part of this piece of code is in the first line, where the result of a command is used in the if loop by using backquoting, and that result is compared with the value 120. If the result is less than 120, the following section becomes active. If the result is greater than 120, nothing happens. As for the command itself, it uses the df command to check available disk space on the volume where /var is mounted, filters out the last line of that result, and from that last line filters out the fourth column only, which in turn is compared with the value 120. And if the condition is true, the logger command writes a message to the system log file. This example isn’t really well organized; the following rewrite does exactly the same, but makes it somewhat more readable: [ `df -m /var | tail -n1 | awk '{print $4}'` -lt $1 ] && logger running out of disk space This shows why it’s fun to write shell scripts: you can almost always make them better.

Case Let’s start with an example this time (see Listing 7-26). Create the script, run it, and then try to explain what it’s done. Listing 7-26. Example Script with Case #!/bin/bash # Your personal soccer expert # usage: soccer cat /dev/null do sleep 5 done logger HELP, the IP address $1 is gone.

Using until Although while does its work as long as a certain condition is met, until is used for the opposite: it runs until the condition is met. This can be seen in Listing 7-29, in which the script monitors whether the user, whose name is entered as the argument, is logged in. Listing 7-29. Monitoring User Login #!/bin/bash # # script that alerts when a user logs in # usage: ishere until who | grep $1 >> /dev/null do echo $1 is not logged in yet

215

216

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

sleep 5 done echo $1 has just logged in In this example, the who | grep $1 command is executed repeatedly. In this command, the result of the who command that lists users currently logged in to the system is grepped for the occurrence of $1. As long as that command is not true (which is the case if the user is not logged in), the commands in the loop will be executed. As soon as the user logs in, the loop is broken and a message is displayed to say that the user has just logged in. Notice the use of redirection to the null device in the test, ensuring that the result of the who command is not echoed on the screen.

Using for Sometimes it’s necessary to execute a series of commands, whether for a limited or an unlimited number of times. In such cases, for loops offer an excellent solution. Listing 7-30 shows how you can use for to create a counter. Listing 7-30. Using for to Create a Counter #!/bin/bash # # counter that counts from 1 to 9 for (( counter=1; counter languages2.txt

217

218

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

Next, you can copy the new output file over the old file.

■Caution Never redirect the output to the file that you are analyzing, as in sed

"s/English/French/g" languages.txt > languages.txt. It will completely erase the content of the file!

Another useful task that can be accomplished with sed is to remove text from a file. In that case, just add empty replacement text, as seen in the following example: sed "s/something//g" list.txt Of course you then have to make sure that the result is then written to some temporary file. Also very useful is the option to remove any lines that match a certain pattern from a file. For example, the following command would remove user sander from the /etc/passwd file: sed "/sander/d" /etc/passwd Notice that no substitution is used in this example; instead, the d (delete) command removes the line. You can even make it somewhat more complicated by removing an empty line. In this case, you need to use a regular expression. The next example shows how: sed "/^$/d" /myfile The special construction used here is a regular expression that searches for the beginning of the line, indicated by a ^, which is followed immediately by the end of the line, which is indicated by a $. Because there’s nothing between the two of them, this construction helps you to find empty lines.

Working with Functions The function is an element that can be very useful in longer shell scripts. A function is a subroutine in a script that is labeled with a name. Using functions makes it easier to refer to certain command sequences that have to be used more than once in a script. For instance, you could create a generic error message if something went wrong. You can define functions in two ways. You can use something like this: function functionname { command1 command2 commandn } Or you could do it this way:

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

functionname () { command1 command2 commandn } To increase the readability of a script, it’s a good idea to use functions if certain code sequences are needed more than once. Listing 7-33 is an example of a script in which a function is used to display an error message. This script is a replacement for the file command, with the difference that the script displays a more elegant error message. Listing 7-33. Displaying Error Codes Using Functions #!/bin/bash # This script shows the file type # # usage: filetype $1 function noarg { echo "You have made an error" echo "When running this script, you need to specify the name of the file" echo "that you want to check" exit 1 } if [ -z $1 ]; then noarg else file $1 fi exit 0 In Listing 7-33, the function has the name noarg. In it, some text is specified that has to be echoed to the screen when the function is called. The function basically defines an error message, so it makes sure that the script terminates with an exit status of 1. As you can see, the function is called just once in this script, when a user forgets to enter the required argument.

A Complex Scripting Example Let’s discuss one more script—one that provides a rather complex example in which process activity is monitored (see Listing 7-34). To do this, the script will periodically check the most active process, and if this script’s activity rises above a threshold, it will send an e-mail to the user root.

219

220

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

Listing 7-34. Complex Scripting Example #!/bin/bash # Script that monitors the top-active process. The script sends an email to the user # root if utilization of the top active process goes beyond 80%. Of course, this # script can be tuned to do anything else in such a case. # # Start the script, and it will run forever. while true do # Check if we have a process causing high CPU load every 60 seconds sleep 10 BLAH=`ps -eo pcpu,pid -o comm= | sort -k1 -n -r | head -1` USAGE=`echo $BLAH | awk '{ print $1 }'` USAGE=${USAGE%.*} PID=`echo $BLAH | awk '{print $2 }'` PNAME=`echo $BLAH | awk '{print $3 }'` # Only if we have a high CPU load on one process, run a check within 7 sec. # In this check, we should monitor if the process is still that active # If that's the case, root gets a message if [ $USAGE -gt 70 ] then USAGE1=$USAGE PID1=$PID PNAME1=$PNAME sleep 7 # FIX MIJ BLAH2=`ps -eo pcpi,pid -o comm= | sort -k1 -n -r | head -1` USAGE2=`echo $BLAH2 | awk '{ print $1 } '` USAGE2=${USAGE2%.*} PID2=`echo $BLAH2 | awk '{print $2 }'` PNAME2=`echo $BLAH2 | awk '{print $3 }'` # Now we have variables with the old process information and # with the new information [ $USAGE2 -gt 70 ] && [ $PID1 = $PID2 ] && mail -s "CPU load of\ $PNAME is above 70%" root < . fi done Again, you can see that this script comprises several parts. The first thing that happens is that a variable with the name BLAH is defined. In this variable, three values are stored for the most active process: the first value indicates CPU usage, the second value indicates the PID of that process, and the third value refers to the name of the process. These three values are

CHAPTER 7 ■ RUNNING IT ANY WAY YOU LIKE

stored in the variables USAGE, PID, and PNAME. Notice that the script performs a patternmatching operation on the variable USAGE. This is to make the value of this variable a whole number, such as 81 instead of a fractional number like 81.9. (This is necessary because Bash cannot perform calculations on fractional numbers.) Also notice the use of the awk command, which plays an essential role in the script, and that’s to strip the value of the different fields that are stored in the variables BLAH and BLAH2. In the second part, the script looks to see whether the CPU utilization of the most active process is higher than 70 percent. If it is, a second check is made to get the usage, name, and PID of that process at that time. These values are stored in the variables USAGE2, PID2, and PNAME2. In the third part, the script determines whether the script that has a CPU utilization greater than 70 percent in the first run is the same as the script with the highest CPU utilization in the second run. If so, a message is sent to the user root.

Summary In this chapter, you learned about some basic ingredients of shell scripts. This chapter is in no way a complete overview of everything that can be used in a shell script; it’s just meant to familiarize you with the basic ingredients of a script, so that you can analyze scripts that are used on your server or write simple scripts yourself to simplify tasks that you have to perform repeatedly. In Chapter 8, you’ll learn how to set up networking on Ubuntu Server.

221

CHAPTER

8

Making a Connection Configuring the Network Interface Card and SSH W

hat is a server without a network connection? Of no use whatsoever. We’ve already explored the possibilities of the Linux operating system itself, and now we need to talk about the network. In this chapter, you’ll learn how to configure the network card. Also, I’ll talk about setting up remote connections using SSH. And you’ll learn some basic steps to troubleshoot a network connection.

Configuring the Network Card When installing your server, the installer automatically configures your server’s network board to get its IP configuration from a DHCP server. As you read in Chapter 1, you can configure it to use a static IP address instead. Also, after installing a server, it’s possible to change the IP address assignment of your server’s network card. Let’s see how this works. When your server boots, it starts the networking script from /etc/init.d. The script reads the configuration that is stored in the /etc/network directory, paying particular attention to the /etc/network/interfaces file. This configuration file stores the entire configuration of the network board. Listing 8-1 shows an example configuration for a server that has two Ethernet network cards. Listing 8-1. Example Configuration for a Network Board root@ZNA:~# cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback

223

224

CHAPTER 8 ■ MAKING A CONNECTION

# The primary network interface auto eth0 iface eth0 inet static address 192.168.1.33 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 gateway 192.168.1.254 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 193.79.237.39 dns-search lan #The second network board auto eth1 iface eth1 inet static address 10.0.0.10 netmask 255.255.255.0 network 10.0.0.0 broadcast 10.0.0.255 As you can see from the configuration file, the server has activated three network devices. The first is lo, which is the loopback interface. It’s required for many services to function, even if your server has no network connection at all. Typically, it uses the IP address 127.0.0.1. In most cases, an Ethernet network board is used to connect with the rest of the world. This network board is represented by the name eth0 if it’s the first, and names such as eth1 and so on for the next cards. The definition of each of the network cards starts with auto ethn, in which n is the number of the network interface. This line is used to start the network board automatically when your server boots. Although you can omit this line, you need to use the ifup or ifconfig commands (as described in a bit) to start the network board by hand. In most situations, you don’t want to do that, so make sure that the line that starts with auto is used at all times. Following the auto line, there is a definition of the interface itself. In this example, a server is configured with two static IP addresses (which is what you typically want for a server). If you need DHCP on an interface, make sure that the iface line reads iface ethn inet dynamic. Following is the rest of the configuration for the network board. You’ll need address, netmask, network, and broadcast in all cases. The other options are optional. To show the current network configuration of your server, the ifconfig command is the easiest command to use. For more versatile management of your network board, you should use the ip command, which is discussed later in this chapter. Listing 8-2 is an example of the output of ifconfig. Especially notice the address given by the inet addr parameter, which is the IP address your server uses to connect to the rest of the world. Listing 8-2. The ifconfig Command Shows the Current Network Configuration root@RNA:/etc/network# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:03:C4:1C inet addr:192.168.1.70 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe03:c41c/64 Scope:Link

CHAPTER 8 ■ MAKING A CONNECTION

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2627 errors:0 dropped:0 overruns:0 frame:0 TX packets:335 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:282945 (276.3 KiB) TX bytes:35050 (34.2 KiB) Interrupt:16 Base address:0x2000 lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:14 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:700 (700.0 b) TX bytes:700 (700.0 b)

Now that you know where the server stores its network configuration, you can also change it directly in this file. This is useful if you want to change the IP address of your network card quickly, for example. Next, you can restart the network card with the ifdown and ifup commands, after which the new configuration is activated. Or you can use /etc/init.d/ networking restart if you want to reread the configuration for all your network interfaces.

Using ifup, ifdown, and Related Tools The ifup and ifdown commands make managing a network board easy, and these tools are simple: call the tool followed by the name of the network board that you want to manage. For example, ifup eth0 starts network card eth0 and ifdown eth0 stops it again. Another useful tool to manage the network card is ifplugstatus, which shows the state of a network interface. As seen in Listing 8-3, this utility shows if a link is detected on a network interface. (If the ifplugstatus utility hasn’t been installed yet, use apt-get install ifplugd.) Listing 8-3. The ifplugstatus Tool Shows the Current Connection State of a Network Boar root@RNA:/# ifplugstatus lo: link beat detected eth0: link beat detected

Using ifconfig The ifconfig command is used to manage a network interface card. The command has been around for years, so it’s not the most flexible command, but it’ll still do the job. And the biggest advantage is that it’s a relatively easy command to use. If you just use the ifconfig command without any parameters, you’ll see information about the current configuration of the network cards in your server. You saw an example of this in Listing 8-2.

225

226

CHAPTER 8 ■ MAKING A CONNECTION

Displaying Information with ifconfig The ifconfig command provides different kinds of information about a network card. It starts with the name of the protocol used on the network card. The protocol is indicated by (for example) Link encap: Ethernet, which states that it is an Ethernet network board. Almost all modern LAN interfaces will show you Ethernet as the link encapsulation type. Then, if applicable, the MAC address is given as the HWaddr (hardware address). This address is followed by first the IPv4-related address information and then the IPv6 address information if IPv6 hasn’t been disabled. Then several statistics about the network board are given. Pay special attention to the RX packets (received packets) and TX packets (transmitted packets) because you can see from these statistics what the network board is doing and whether any errors have occurred. You typically shouldn’t see any errors at all. Apart from the information about the physical network cards that are present in your server, you’ll also always see information about the loopback device (lo), which is the network interface that’s used for internal purposes on your server. You need this loopback device because some services depend on it; for example, the graphical environment that’s used on Linux is written on top of the IP stack offered by the loopback interface.

Configuring a Network Card with ifconfig Although the server is provided with an IP address upon installation, it’s important for you to be able to manage IP address configuration on the fly, using the ifconfig command. Fortunately, it’s relatively easy to configure a network board in this way: just add the name of the network board you want to configure, followed by the IP address you want to use on that network board (for example, ifconfig eth0 192.168.1.125). This command will configure eth0 with a default class C subnet mask of 255.255.255.0, which indicates that the first three bytes of the IP address are a part of the network address and that the last byte is the unique host identifier within that network.

■Tip Not sure which eth device number is used? You can manage this via the udev mechanism. In the /etc/udev/rules.d/70-persistent-net.rules file, a mapping is made between the MAC address and interface number of your network cards; see the section about udev in Chapter 6 for more details. So if you want your eth1 device to be presented as eth20, this is the place where you can change the device

number.

If you need something other than a default subnet mask, add an extra parameter. An example of this is the ifconfig eth0 172.16.18.18 netmask 255.255.255.0 broadcast 172.16.18.255 command, which configures the eth0 device with the given IP address and a 24-bit subnet mask. Note that this example uses a nondefault subnet mask. If this happens, you have to specify the broadcast address that’s used to address all nodes in the same network as well; the ifconfig command just isn’t smart enough to realize that you’re using a nondefault IP address and to calculate the right broadcast address accordingly.

CHAPTER 8 ■ MAKING A CONNECTION

Bringing Interfaces Up and Down with ifconfig Apart from adding an IP address to a network board, you can use the ifconfig command to bring a specific network board up or down. For example, ifconfig eth0 down shuts down the interface, and ifconfig eth0 up brings it up again with its default settings as specified in the /etc/network/interfaces configuration file. This is useful if you want to test a new configuration, but you’re not sure whether it will really work properly. Instead of using ifconfig to manipulate your network card, you can also use ifup and ifdown. These commands allow you to bring a network card up or down easily, and without changing the configuration of a given network board. For example, to bring a network board down, use ifdown eth0; to bring it up again, use ifup eth0. In both cases, the default configuration for the network board as specified in /etc/network/interfaces is applied.

Using Virtual IP Addresses with ifconfig Another rather useful way of using ifconfig is to add virtual IP addresses, which are just secondary IP addresses that are added to a network card. A network board with virtual IP addresses can listen to two different IP addresses, which is useful if you are configuring services on your server that all need their own IP addresses. Think of different virtual Apache web servers, for example. You can use the virtual IP address within the same address range or on a different one. To add a virtual IP address, add :n where n is a number after the name of the network interface. For example, ifconfig eth0:0 10.0.0.10 adds the address 10.0.0.10 as a virtual IP address to eth0. The number after the colon must be unique, so you can add a second virtual IP address with ifconfig eth0:1 10.0.0.20, and so on. When you use the ifconfig tool to display the current configuration of your server, you’ll see all virtual IP addresses that are configured, as shown in Listing 8-4. Listing 8-4. The ifconfig Tool Shows Virtual IP Addresses root@ZNA:~# ifconfig eth0:0 10.0.0.10 root@ZNA:~# ifconfig eth0:1 10.0.0.20 root@ZNA:~# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:A0:A5:80 inet addr:192.168.1.33 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fea0:a580/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3035 errors:0 dropped:0 overruns:0 frame:0 TX packets:199 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:240695 (235.0 KiB) TX bytes:19035 (18.5 KiB) Interrupt:18 Base address:0x1400 eth0:0

Link encap:Ethernet HWaddr 00:0C:29:A0:A5:80 inet addr:10.0.0.10 Bcast:10.255.255.255 Mask:255.0.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0x1400

227

228

CHAPTER 8 ■ MAKING A CONNECTION

eth0:1

Link encap:Ethernet HWaddr 00:0C:29:A0:A5:80 inet addr:10.0.0.20 Bcast:10.255.255.255 Mask:255.0.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:18 Base address:0x1400

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Using the ip Tool Although the ifconfig tool can still be used to display information about the configuration of a network card, it’s not the only tool available. A more flexible (but also more difficult) tool is ip. The ip tool has many options that allow you to manage virtually all aspects of the network connection. For example, you can use it to configure an IP address, but it manages routing as well, which is something that ifconfig can’t do.

■Note The ip tool offers more than ifconfig. Its syntax is also more difficult, so many people stick to using ifconfig instead. To be honest, it doesn’t really matter because they can both be used for the same purposes in almost all cases. If you really want to be sure not to run into trouble, you should use ip, however.

The first option you use after the ip command determines exactly what you want to do with the tool. This first option is a reference to the so-called object, and each object has different possibilities. The following objects are available: • link: Used to manage or display properties of a network device • addr: Used to manage or display IPv4 or IPv6 network addresses on a device • route: Used to manage or display entries in the routing table • rule: Used to manage or display rules in the routing policy database • neigh: Used to manage or display entries in the ARP cache • tunnel: Used to manage or display IP tunnels • maddr: Used to manage or display multicast addresses for interfaces • mroute: Used to manage or display multicast routing cache entries • monitor: Used to monitor what happens on a given device

CHAPTER 8 ■ MAKING A CONNECTION

Each of these objects has options of its own. The easiest way to learn about these options is to use the ip command, followed by the object, and then followed by the keyword help. For example, ip address help provides information on how to use the ip address command, as shown in Listing 8-5. Listing 8-5. The ip address help Command Gives Help on Configuring IP Addresses with the ip Tool root@ZNA:~# ip address help Usage: ip addr {add|del} IFADDR dev STRING ip addr {show|flush} [ dev STRING ] [ scope SCOPE-ID ] [ to PREFIX ] [ FLAG-LIST ] [ label PATTERN ] IFADDR := PREFIX | ADDR peer PREFIX [ broadcast ADDR ] [ anycast ADDR ] [ label STRING ] [ scope SCOPE-ID ] SCOPE-ID := [ host | link | global | NUMBER ] FLAG-LIST := [ FLAG-LIST ] FLAG FLAG := [ permanent | dynamic | secondary | primary | tentative | deprecated ] It can be quite a challenge to find out how the help for the ip tool works, so I’ll give you some help with this help feature. To understand what you need to do, you must first analyze the Usage: lines. In this output, you see two of them: a usage line that starts with ip addr {add|del} and another that starts with ip addr {show|flush}. Let’s have a look at the first one. The complete usage line is ip addr {add|del} IFADDR dev STRING. You can add or delete an IP address that is referred to by IFADDR from a device (dev) that is referred to by STRING. Now a string is just a string, and that can be anything, but that’s not the case for the IFADDR part. Therefore, you can find an explanation of that part in the next section of the help output: IFADDR := PREFIX | ADDR peer PREFIX [ broadcast ADDR ] [ anycast ADDR ] [ label STRING ] [ scope SCOPE-ID ]. In this line, the help explains that you have to use a PREFIX or an ADDR statement, which might be followed by several options such as the broadcast address, the anycast address, a label, or a SCOPE-ID. Now that you understand how help works, let’s have a look at some examples.

Displaying IP Address Setup Information with the ip Tool A common use of the ip tool is to display information about the use of IP addresses for a given interface. The command to use is ip address show. Note that if it is clear exactly what you want and there can be no confusion between options, you can specify the options used with the ip command in short form, such as ip a s, which accomplishes the same thing as ip address show. This command displays the information shown in Listing 8-6. Listing 8-6. Showing ip Address Configuration with ip address show root@ZNA:~# ip address show 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host

229

230

CHAPTER 8 ■ MAKING A CONNECTION

valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:a0:a5:80 brd ff:ff:ff:ff:ff:ff inet 192.168.1.33/24 brd 192.168.1.255 scope global eth0 inet 10.0.0.10/8 brd 10.255.255.255 scope global eth0:0 inet 10.0.0.20/8 brd 10.255.255.255 scope global secondary eth0:1 inet6 fe80::20c:29ff:fea0:a580/64 scope link valid_lft forever preferred_lft forever If you look hard enough, you can see that the result of ip address show is almost the same as the result of ifconfig. It’s just presented differently.

Monitoring Device Attributes Another simple use of the ip tool is to show device attributes, which you can do with the ip link show command. This command shows usage statistics for the device you specified, but no address information. Listing 8-7 provides an example of its output. Listing 8-7. Use the ip link show Command for an Overview of Link Attributes root@ZNA:~# ip link show 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0c:29:a0:a5:80 brd ff:ff:ff:ff:ff:ff The information displayed by ip link show is related to the activity on the network board. Of particular interest are the device attributes returned for each of the devices (they’re displayed in brackets right after the name of the device). For example, in most cases you can see the attributes BROADCAST,MULTICAST,UP for a normal network interface card. The BROADCAST attribute indicates that the device is capable of sending broadcasts to other nodes in the network, the MULTICAST attribute indicates that the device can also send multicast packets, and UP indicates that the device is working. The command also shows all IP protocol attributes, such as the maximum transmission unit (mtu) that is used on the interface.

Setting the IP Address Just as with the ifconfig tool, you can use the ip tool to assign an IP address to a device. To do this, you could use a command like ip address add 10.0.0.10/16 brd + dev eth0. This command sets the IP address to 10.0.0.10 for eth0. With this IP address, a 16-bit subnet mask is used, which is indicated by the /16 directly behind the IP address. The broadcast address is calculated automatically, which is indicated with the brd + construction. Once you have set the IP address with the ip tool, you can use the following command to check whether it’s set correctly: ip address show dev eth0. As with the ifconfig command, you can add more than one IP address to a network interface when using the ip tool as well. And it isn’t hard: just use ip address add 10.0.0.20/16 brd + dev eth0 and 10.0.0.20 with its specified properties is added as a second IP address to eth0. You should, however, note the difference between the secondary IP addresses that are added with ifconfig and the IP addresses that are added with the ip tool. An address added

CHAPTER 8 ■ MAKING A CONNECTION

with ip won’t show up when you use ifconfig. So, when using secondary IP addresses, make sure that you use the right tool to check their properties.

Managing IPv6 Currently, IPv4 is the default protocol on most servers. However, because it has some serious shortcomings, a new version of the IP protocol began development a few years ago. Because this draft for IP version 5 just didn’t make it, the new generation of Internet protocol is referred to as IPv6, and it’s this version that’s installed by default on Ubuntu Server, so you can use it as an alternative to IPv4. In this section, you’ll learn about the properties of IPv6 and how to configure it on your server. This section isn’t meant as an in-depth coverage of IPv6 and all its properties. Instead, it aims to help you configure IPv6 on a server and see whether it’s useful in your environment.

IPv6 Addressing Before you start the actual implementation of IPv6, you should know about its peculiarities, of which the first and most important is the address length. Although IPv4 has to work with 32-bit addresses that are grouped in 4 groups of 8 bits (such as 192.168.1.13) and that theoretically allow for approximately 4,000,000,000 unique addresses, IPv6 offers a 128-bit address space, which yields more than enough IP addresses to assign one to every square meter of the Earth’s surface, including the oceans. Opposite to the decimal-written IPv4 addresses, the IPv6 addresses are written in hexadecimal and split into 16-bit groups. An example of such an address is 2bad:babe:5655:8812:0BFC:1234:0:1234. Not really something you would care to memorize. If an IPv6 address has more than one group of 16 bits with the value of 0, you can abbreviate this using the double colon (::). For example, the IPv6 address 2bad:0:0:0:0:1234:5678:90ab can also be written as 2bad::1234:5678:90ab, and 0:0:0:0:0:0:0:1 is just ::1. This clever shortcut makes working with IPv6 addresses much easier. Another nice feature of IPv6 is that you can share an IP address among different NICs so that several network cards listen to the same IP address. This easy-to-implement method is for load balancing. Because more than enough bits are available in an IPv6 address, there’s a default division between the network part of the address and the node part of the address. This is an important advantage of IPv4, in which you must use a subnet mask to specify which part of the address refers to the network and which part refers to the node on that network. So with IPv6 you don’t need to struggle with subnet masks any more. The last 64 bits of an IPv6 address are normally used as the node ID. This node ID is a socalled IEEE EIA-64 ID, which on Ethernet consists of the 48-bit MAC address with FFFE added between the vendor identifier and the node identifier. If a network interface doesn’t have a MAC address, the node ID is randomly generated. Because the IPv6 address includes the MAC address, something important follows: the node in an IPv6 network can determine its own IPv6 address. All that a node must do is listen on the network to check for the address that’s in use. Next, it can add its own MAC address, transform that to an IEEE EIA-64 ID, and it’ll be able to communicate with the rest of the network automatically. So there goes the need for the DHCP server that was required for automatic address configuration in IPv4 as well.

231

232

CHAPTER 8 ■ MAKING A CONNECTION

If you really need it, an IPv6 address can work with a subnet mask. By default, this subnet mask for all addresses is /64 (64 bits), which specifies that the first half of the IPv6 address refers to the network, but you can use something other than this default subnet mask as well. However, the last 64 bits of an address are always reserved for the node address, so you can’t use them in the subnet mask.

Address Types In IPv6, you can use different types of addresses: • Link local addresses: These IP addresses are used if no specific information about the network configuration could be found. They’re intended for local use only, and they always start with FE80 in the first two bytes. They aren’t routable, but they are necessary for neighbor discovery (see the next section, “The Neighbor Discovery Protocol”). Link local addresses are always created automatically if IPv6 is enabled. • Site local addresses: These are similar to addresses that are defined in the private address ranges for IPv4. They cannot be addressed from nodes outside this network. Site local addresses always start with FEC0 and have a default 48-bit subnet mask. The last 16 bits can be used for internal subnetting. Site local addresses are not created automatically. • Aggregatable global unicast addresses: These are the “normal” worldwide unique addresses that are used on IPv6 networks. They are assigned by an administrator and always start with a 2 or 3 (binary 001). • Multicast addresses: These addresses are used to address groups of nodes. They always start with FF. • Anycast addresses: This is the IPv6 alternative for a broadcast address. When using anycast, the IPv6 node gets an answer from any node that matches the anycast criteria. • In IPv6, broadcast addresses are not used. On a single Linux host that uses IPv6 (which by default is the case on Ubuntu Server), you’ll always find more than one IPv6 address: • A loopback address (::1) is used on the loopback interface. • A link local address is generated automatically for every interface. • If the administrator has configured it, every interface has a unicast address. This can be a site local address, an aggregatable global unicast address, or both.

Neighbor Discovery Protocol One of the design goals of IPv6 was to make network configuration easier. For this purpose, the neighbor discovery protocol was defined in RFC 2461 (see www.ietf.org/ rfc-rfc2461.txt). The purpose of this protocol is to provide an automatic IP address assignment: neighbor discovery makes sure that a node can automatically find routers, addresses, prefixes, and other required configuration information, just by looking at what happens on the network.

CHAPTER 8 ■ MAKING A CONNECTION

In the neighbor discovery protocol, a router advertises all relevant information such as the best next hop. Individual nodes check their neighbors and keep information about the neighbors in the neighbor cache, so that they always have current and reliable information about the rest of the network. In the neighbor cache, a node keeps information such as addresses of neighbors, a list of prefixes (IPv6 addresses) that are in use by the neighbors, and a list of routers that can be used as default routers. So, the neighbor discovery protocol really makes IPv6 a plug-and-play protocol.

Assigning IPv6 Addresses in Ubuntu Server On Ubuntu Server, you can use ip as well as ifconfig to configure an IPv6 address. All required kernel modules are loaded by default, so, with regard to that, no extra work needs to be done. Let’s look at the following examples of how to configure IPv6 on your server: • ifconfig eth0 inet6 add 2000:10:20:30:20c:29ff:fec7:82f6/64: This command configures eth0 with an IPv6 address that is an aggregatable global unicast address (a worldwide unique address). Note that the second part of the address assigned here is the IEEE EIA-64 ID of the network interface card that the address is added to. You need to configure only one address per LAN in this way, and all other nodes will get the aggregatable global unicast address assigned automatically by means of the neighbor discovery protocol. To make this a functional IPv6 address, you should also make sure that the EIA-64 ID of your network card matches your MAC address. To get to an EIA-ID from the MAC address, you should insert ff:fe in the middle of the MAC address and put a 2 in front of it. For instance, the MAC address 00:19:d1:ed:82:07 would get the EIA-ID of 219:d1ff:feed:8207. • ip address add 2000:10:20:30:20c:29ff:fec7:82f6/64 dev eth0: This is exactly the same as the command used in the previous example, with the exception that the ip tool is used instead of ifconfig. • ip address add fec0:10:20:30:29ff:fec7:82f6 dev eth0: This command adds a site address to interface eth0. Note that this also has to be performed for just one node per LAN. Instead of using the ip tool, you can do the same with ifconfig eth0 inet6 add fec0:10:20:30:29ff:fec7:82f6 dev eth0. • route -A inet6: This command shows information about current IPv6 routes. • route -A inet6 add 2000::/3 gw 3ffe:ffff:0:f101::1: This adds a route in which all addresses that start with binary 001 (decimal notes as 2) will be sent to the specified default gateway. Once your IPv6 interface is set up, you’ll probably want to test its operation as well. You can find some tools in the Linux iputils package such as the ping6 utility that you can use to ping other hosts to check for their availability. Note that when using ping6, you always need to specify the interface you want to send the ping packets from: for example, ping6 -I eth0 fe80::2e0:18ff:fe90:9205. Also in the same iputils package are the traceroute6 tool that can be used to trace the route to a given destination and the tracepath6 tool, which does more or less the same thing but without the need to use superuser privileges. You can read more about these tools in the “Troubleshooting Network Connections” section of this chapter.

233

234

CHAPTER 8 ■ MAKING A CONNECTION

Managing Routes You’ve read about how a network interface is provided with an IP address. But to be completely functional on the network, you have to specify some routes as well. These routes allow you to communicate with computers on other networks; conversely, they allow computers on other networks to communicate with your server. As a minimal requirement, you need a default route. This route specifies where to send packets that don’t have a destination on the local network. The router used for the default route is always on the same network as your server; just consider it to be the door that helps you get out of the local network. Your server typically gets the information about the default router that it should use from the /etc/network/interfaces file. To set the default route, two tools can be used: the ip tool and the route utility. The ifconfig utility was never meant to create or maintain routes, so you can’t use it for this purpose.

Setting the Default Route with route The old command to set the default route is route. If no options are used, it will display a list of all routes that are currently defined on this host. Listing 8-8 provides an example. When using the route command without options, it will always try to resolve the name for a given IP address, which takes some time. If you don’t want any name resolution to be performed, use the option -n, which makes the command a lot faster. Listing 8-8. Use the route Command to Get an Overview of All Routes That Are Currently Configured root@ZNA:~# route Kernel IP routing table Destination Gateway localnet * 10.0.0.0 * default 192.168.1.254

Genmask 255.255.255.0 255.0.0.0 0.0.0.0

Flags U U UG

Metric 0 0 0

Ref 0 0 0

Use Iface 0 eth0 0 eth0 0 eth0

Several columns are displayed in the output of the route command, as you can see in Listing 8-8. The first column provides the destination, which is the network or host that a route is defined for. Next is the gateway, which is the router that needs to be contacted to reach the specified destination. An asterisk (*) in the gateway column indicates that the localhost is the gateway for that destination. For external routers used as the destination, you’ll see the IP address (or name) of that router. Next is the genmask, which is the subnet mask used on the specified destination. Then come the flags, metric, ref, and use columns, all of which reveal more detailed information about this route. Finally, the iface column reveals what network interface is used to route packets. To specify a route, you need to provide a minimum of two pieces of information: what network you want to add an entry for and what router is used as a gateway. All the other information is added automatically. For example, if you want to specify that the router with IP address 192.168.1.254 should be used as the default gateway, use the route add default gw 192.168.1.254 command. If you need to change the default gateway, you should be aware that you first have to remove the old route. Use the route del command to do this. For example, to remove the current setting for the default gateway, use route del default gw.

CHAPTER 8 ■ MAKING A CONNECTION

Using the ip Tool to Specify the Default Gateway If you know what information has to be entered when defining a route, it’s easy to do it with either the ifconfig or the ip tool. The only differences are minor changes in the syntax that’s used. To set the default gateway to 192.168.1.254 using the ip tool, use the ip route add default via 192.168.1.254 command. This command makes sure that all packets sent to nonlocal destinations are sent through 192.168.1.254. Likewise, you can delete the default route with ip route del default.

Storing Routing Information When you enter information, such as the default gateway, from the command line, it will be lost the next time you reboot your server. To make sure that the information remains after a reboot, store it in the /etc/network/interfaces file. This file is read every time the network is activated. The entry used in this file to store the default route isn’t complex: gateway 192.168.1.254 If you have just one network card in your server, include it in the entry for your network card. If you want to specify a default route per network card (for example, one for your private internal network and one for the public external network), you can specify a default route setting for each of the network cards.

Configuring the DNS Resolver If you want to manually configure a network connection as the last part, you need to specify what DNS name server to use. This is the so-called DNS resolver. With Linux, you do this by modifying the /etc/resolv.conf file. Typically, this file will contain the IP address of at least two DNS name servers and a search domain. The name server specifications indicate what DNS name server should be contacted to translate DNS names to IP addresses, and vice versa. Specify at least two name servers, so that if the first one cannot be reached, the second one can do the job. The search domain specifies what domain name should be appended if an incomplete host name is used. On Ubuntu Server, this is typically set to lan. Listing 8-9 is an example of the content of the /etc/resolv.conf file. Listing 8-9. Example of the /etc/resolv.conf File nameserver 192.168.1.10 nameserver 193.79.237.39 search lan In this example, you see that name server 192.168.1.10 is used as the default name server, and all DNS requests will be sent to it. If this server cannot be reached, only then will the second server in the list (193.79.237.39) be contacted. Make sure to always specify the addresses of two name servers. You can specify a third name server if you like, but it probably will never be used (just because of the slim chance that the first and second names are both unavailable). You’ll see that the third line of the Listing 8-9 example specifies the search domain. For example, if a user uses the command ping ftp, which includes an incomplete host name, the name of the domain specified with the search option in resolv.conf is added automatically to it (this works only if there really is a host with the name ftp in your local network).

235

236

CHAPTER 8 ■ MAKING A CONNECTION

The Role of the nsswitch.conf File Most people take it for granted that DNS resolves host names to IP addresses, but this isn’t necessarily so. Every Linux box has the /etc/nsswitch.conf file that determines what exactly should happen when translating a host name to an IP address and vice versa. This file specifies many things, but only the following lines are important for resolving host names: hosts: networks:

files dns files

These two lines specify that when resolving host names and network names, the files should be searched first, and the DNS subsystem should be used only if the files have no information about the given host. Thus, an administrator can make sure that frequently accessed host names are resolved locally, with the DNS server being contacted only when the files don’t have information about a particular host. The most important file used for resolving names to IP addresses is the /etc/hosts file.

Using the /etc/hosts File One of the oldest ways to resolve host names to IP addresses (and vice versa) is to use the /etc/hosts file. It’s rather primitive because the file has to be maintained on every single host where you need it, and no synchronization is established between hosts. But it’s also a very efficient way to supply information that needs to be available locally.

■Note To resolve the problem of decentralized management, the Network Information Service (NIS, formerly known as Yellow Pages) was invented by Sun. It’s rarely used now because most companies keep their hosts-related information in DNS files.

Using the /etc/hosts file makes resolving names faster and reduces Internet traffic, and you can use it to add any host names that need to be available only locally. Listing 8-10 shows the contents of this file as it is created after a default installation of Ubuntu Server. Listing 8-10. Example of the /etc/hosts File root@ZNA:~# cat /etc/hosts 127.0.0.1 localhost 192.168.1.33 ZNA.lan ZNA # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts root@ZNA:~#

CHAPTER 8 ■ MAKING A CONNECTION

As you can see, the contents of this file are rather simple. First, you specify the IP address of the host, which can be an IPv4 or an IPv6 address. Next, the fully qualified host name of the host is specified. This is the name of the host itself, followed by its DNS suffix. Last, the short host name is used. Alternatively, you can just provide the IP address followed by the name of the host you want to add, as in the following line: 192.168.1.180

RNA

On a modern Linux server, it’s not necessary to set up /etc/hosts except for local name resolving. Network name resolution is typically managed by DNS. So you’ll always need your own host name and IP address in this file. This is configured automatically when installing Ubuntu Server.

Configuring Network Card Properties with the ethtool Command Up to now, we’ve talked about stuff related to IP addresses. But the network card itself has settings that you may need to modify, and you’ll use the ethtool command to do this. With it, you can change network board properties such as link speed and duplex mode. Don’t overestimate this tool, however. Some Ethernet cards are not supported, and the only way to change settings on them might be through the network card’s BIOS settings. Let’s start by displaying some information: use ethtool -i eth0 to see an overview of driver properties that are currently used, as shown in Listing 8-11. Listing 8-11. The ethtool -i Command Provides an Overview of Driver Properties root@ZNA:~# ethtool -i eth0 driver: pcnet32 version: 1.33 firmware-version: bus-info: 0000:00:11.0 To change duplex settings and link speed on your network board, you’ll use the -s option, followed by one of these arguments: • speed: This option changes the speed. Valid options are 10, 100, and 1000 (all of them expressing megabits per second). • duplex: This option changes the duplex settings. Set it to half or full. • port: This option specifies what port to use. This option is used for network interfaces with different ports available (which is not very common). Valid choices are tp, aui, bnc, mii, and fibre. • autoneg: This option indicates whether you want to use auto negotiation to discover the settings that are used on the network. Valid choices are on and off. So, for example, if you want to change the settings of your network card to full duplex and a link speed of 1000 Mbps, use ethtool -s eth0 speed 1000 duplex full. Now there is a problem when using ethtool like this because you need to enter these settings again the next

237

238

CHAPTER 8 ■ MAKING A CONNECTION

time you start your server. If you don’t want to do that all the time, you can create a simple script that contains all the settings. The following procedure describes how to do this: 1. Use Vi to create a script: vim /etc/init.d/ethtool. 2. Append the following lines (change them to reflect the settings that you need to use): #!/bin/bash ETHTOOL="/usr/sbin/ethtool" DEV="eth0" SPEED="1000 duplex full" case "$1" in start) echo –n "Setting eth0 speed…"; $ETHTOOL –s $DEV speed $SPEED; echo " done.";; stop) ;; esac exit 0 3. Make this script executable using the command chmod +x /etc/init.d/ethtool. 4. Run the update-rc.d command to update your runlevels. This process ensures that this script will be executed in all appropriate runlevels. Run it as follows: update-rc.d ethtool defaults.

■Note The update-rc.d command is available to add startup scripts to the System-V–style boot procedure. You can use it on any script, as long as the script fits these minimal requirements: the script should use a case construction; within the case, there should be a start) and a stop) section. If your script complies with these requirements, you can add it to your runlevels.

Besides the brief summary you get about your network board when you use the –i option with ethtool, there are also some other useful options. For instance, you can get some very detailed statistics about your network board when using ethtool –S, as you can see in Listing 8-12.

■Note The drivers of some network boards don’t support ethtool. Therefore, in some cases this command shows nothing at all. If that happens, your network card doesn’t support ethtool, and there’s nothing you can do about it.

CHAPTER 8 ■ MAKING A CONNECTION

Listing 8-12. The ethtool –S Tool Gives Very Detailed Statistics About Network Cards root@mel:~# ethtool -S eth0 NIC statistics: rx_packets: 1691 tx_packets: 319 rx_bytes: 183662 tx_bytes: 37876 rx_broadcast: 1441 tx_broadcast: 72 rx_multicast: 0 tx_multicast: 6 rx_errors: 0 tx_errors: 0 tx_dropped: 0 multicast: 0 collisions: 0 rx_length_errors: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 tx_restart_queue: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 183662 rx_csum_offload_good: 1504 rx_csum_offload_errors: 0

239

240

CHAPTER 8 ■ MAKING A CONNECTION

rx_header_split: 0 alloc_rx_buff_failed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0

■Note Some people seem to think that ethtool is the most important tool for proper functioning of your network. In fact, it isn’t. In the early days of switched Ethernet, auto negotiate settings didn’t work that well. Nowadays, auto negotiate works fine at almost all times. Therefore, you will find that in most situations you can do perfectly without ethtool.

Troubleshooting Network Connections Once you have finished the setup tasks I just described, you should have a working network connection. But even if it’s working fine right now, you might at some point need to perform some tuning and troubleshooting, and that’s exactly what this section is about. Here, you’ll learn how to test that everything is working the way it should and how to monitor what is happening on the network itself, as well as on the network interface. The tools I’m talking about in this section are the top-notch troubleshooting tools.

Testing Connectivity After configuring a network card, you want to make sure it’s working correctly. For this, the ping command is your friend, and more so because it’s easy to use: enter the command followed by the name or address of the host you want to test connectivity to, such as ping www.ubuntu.com. This forces ping to start continuous output, which you can interrupt by using the Ctrl+C key sequence. You can also send a limited number of packets; for example, the ping -c 3 192.168.1.254 command sends just three packets to the specified host. If you use ping in a clever way, you can test a lot of things with it. I recommend using it in the following order: 1. Ping the localhost. If you pass this test, you’ve verified that the IP stack on your local machine is working properly. 2. Ping a machine on the local network by using its IP address: if this works, you’ve verified that IP is properly bound to the network board of your server and that it can make a connection to other nodes on the network. If it fails, you need to check the information you’ve entered with the ifconfig or ip commands; you may have made an error entering the subnet mask for your network interface. 3. Ping a machine on the Internet using its IP address. A good bet is 137.65.1.1, which is a name server that hasn’t failed me in the last 15 years. Of course, you can use any other host as long as you know its IP address. If the ping is successful, you’ve verified that the routers between the localhost and the destination are all working. If it fails, there’s an error somewhere in the routing chain. Check route -n or ip route show on your localhost to see if the default route is defined.

CHAPTER 8 ■ MAKING A CONNECTION

4. Ping a machine on the Internet using its DNS name. If this succeeds, everything is working. If this step fails (but test 3 was successful), make sure you’ve entered the name of the DNS server that should be used in /etc/resolv.conf. If this is okay, check to see if your DNS server is working. In many cases, you’ll use the ping command without options. But some options can be useful, as seen in Table 8-1. Table 8-1. Useful ping Options

Option

Description

-c count

Specifies the number of packets to be sent. The ping command terminates automatically after reaching this number.

-l device

Specifies the name of the network device that should be used. Useful on a computer with several network devices.

-i seconds

Specifies the number of seconds to wait between individual ping packets. The default setting is one second.

-f

Sends packets as fast as possible, but only after a reply comes in.

-l

Sends packets without waiting for a reply. If used with the -f option, this may cause a denial-of-service attack on the target host and the host may stop functioning properly or even crash. Apart from the unknown harm that this may do to the target server, you may find yourself blacklisted or even charged with a criminal offense. Because this is such a very dangerous option, only the user root is allowed to use it.

-t ttl

Sets the time to live (TTL) for packets that are sent. This indicates the maximum number of routers that each packet may pass through on its way to a destination. The TTL is decremented by one by each router it passes until the TTL becomes 0, which means that the packet won’t be routed any more.

-b

Sends packets to the broadcast address of the network. This prompts a reply from every host that’s up and allowed to answer to ping packets.

■Note To protect against a denial-of-service attack, many hosts are configured not to answer a ping request. Therefore, when testing connectivity, make sure that you use a host that’s allowed to answer.

The ping command is not just used to test that a connection can be established; you can also use it to check the roundtrip delay between your computer and a given host. The elapsed time is an important indication of the quality of the network connection. To check the roundtrip delay, have a look at the time parameter that’s listed in the result of the ping command. Listing 8-13 provides an example in which ping is used to send four packets to www.ubuntu.com. Listing 8-13. Testing Connectivity to www.ubuntu.com root@ZNA:~# ping -c 4 www.ubuntu.com PING www.ubuntu.com (82.211.81.158) 56(84) bytes of data. 64 bytes from arctowski.ubuntu.com (82.211.81.158): icmp_seq=1 ttl=51 time=22.0 ms

241

242

CHAPTER 8 ■ MAKING A CONNECTION

64 bytes from arctowski.ubuntu.com (82.211.81.158): icmp_seq=2 ttl=51 time=10.7 ms 64 bytes from arctowski.ubuntu.com (82.211.81.158): icmp_seq=3 ttl=51 time=18.6 ms 64 bytes from arctowski.ubuntu.com (82.211.81.158): icmp_seq=4 ttl=51 time=20.8 ms --- www.ubuntu.com ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 3015ms rtt min/avg/max/mdev = 10.741/18.092/22.057/4.417 ms

Testing Routability If you can ping your default router, but you can’t ping a given host on the Internet, it’s probably obvious that something is wrong with one of the routers between your network and the destination host. You can use the traceroute command to find out exactly where things are going wrong (use apt-get install traceroute to install traceroute first). The traceroute command uses the TTL value of the UDP datagrams it sends out.

■Note A datagram is a packet sent over the OSI model network layer.

The idea is that, when the TTL value reaches 0, the packet is discarded by the router and a message is sent back to the sender. When starting, traceroute uses a TTL value of 0, which causes the packet to be discarded by the very first router. This is how traceroute identifies the first router. Next, it sends the packet to the target destination again, but with a TTL of 1, which, as you can see, causes the packet to be discarded by the second router. Things continue in this manner until the packet reaches its final destination. To use traceroute, you normally put the host name as the argument, such as traceroute www.ubuntu.com. It’s possible as well to use the IP address of a host, which will produce a result as seen in Listing 8-14. Listing 8-14. Testing a Network’s Route with traceroute root@ZNA:~# traceroute www.ubuntu.com traceroute to www.ubuntu.com (82.211.81.158), 30 hops max, 40 byte packets 1 192.168.1.254 (192.168.1.254) 72.668 ms 10.361 ms 176.306 ms 2 195.190.249.90 (195.190.249.90) 3.353 ms 9.199 ms 10.351 ms 3 42.ge-4-0-0.xr1.3d12.xs4all.net (194.109.5.49) 6.386 ms 7.237 ms 16.421 ms 4 0.so-6-0-0.xr1.tc2.xs4all.net (194.109.5.10) 11.407 ms 11.447 ms 9.599 ms 5 217.149.46.21 (217.149.46.21) 31.989 ms 29.321 ms 22.756 ms 6 sl-bb21-ams-8-0.sprintlink.net (217.149.32.41) 13.415 ms 13.244 ms 12.569 ms 7 213.206.131.46 (213.206.131.46) 11.147 ms 12.282 ms 11.222 ms 8 ae-0-56.mp2.Amsterdam1.Level3.net (4.68.120.162) 7.862 ms ae-0-54.mp2.Amster\ dam1.Level3.net (4.68.120.98) 11.796 ms ae-0-52.mp2.Amsterdam1.Level3.net\ (4.68.120.34) 11.000 ms 9 as-0-0.bbr2.London2.Level3.net (4.68.128.110) 21.047 ms ae-1-0.bbr1.London2.\

CHAPTER 8 ■ MAKING A CONNECTION

Level3.net (212.187.128.46) 35.125 ms as-0-0.bbr2.London2.Level3.net\ (4.68.128.110) 17.183 ms 10 ae-15-53.car1.London2.Level3.net (4.68.117.79) 18.917 ms 17.388 ms ae-25-52.\ car1.London2.Level3.net (4.68.117.47) 18.992 ms 11 tge9-3-146.core-r-1.lon2.\ mnet.net.uk (212.187.196.82) 14.699 ms 17.381 ms 15.293 ms 12 85.133.32.134 (85.133.32.134) 27.130 ms 33.310 ms 37.576 ms 13 82.211.81.76 (82.211.81.76) 16.784 ms 20.140 ms 17.556 ms 14 * * * 15 * * * 16 * * * 17 * * * With the traceroute command, you’ll see every router that’s passed. For each router, the name of the router is displayed, followed by its IP address and then the roundtrip times of the three packets that were sent to that router. You’ll often see that a router replies with only a string of three asterisks (* * *), which indicates that the router forwards packets normally but is configured not to reply to ping packets for security reasons.

Testing Availability of Services When the ping and traceroute commands show that everything is working, you’re the proud owner of a working network interface. Next you can test the availability of two kinds of services: those on your computer itself and those on external computers. Because so many tools are available to test service availability, I won’t try to cover them all, but I do want to discuss two of the most popular. First is the netstat tool, which you can use to test for the availability of services on the host where you run the command. And second is nmap, which is used to test availability on other hosts.

■Caution Some administrators consider any use of nmap on their hosts or their network as an attack against their security, so they don’t allow it. I once used it in a hotel room in the United States to see if my server in Amsterdam was still offering all its services, and the hotel network shut me off immediately. In these circumstances, it can be a real pain to get your connection back, so be careful.

Using netstat to Check Your Server If you want to know what services are available on your server and what these services are doing, the netstat command is an excellent choice. However, because many of its options require you to be root, I recommend that you use netstat as root only. To see the most useful information offered by netstat, use the -platune options, which make sure that you see information about programs connected to ports (-p) and what ports are actually listening (-l). Other options show you everything there is to show (-a), do that for TCP (-t) as well as UDP (-u), without translating IP addresses to DNS names (-n), and with extended information (-e).

243

244

CHAPTER 8 ■ MAKING A CONNECTION

If you think that netstat -platune offers too much information, use netstat -tulp instead. The results are slightly less verbose, which makes it easier to get the data you really need. Listing 8-15 shows the first screen of output generated by netstat -platune. Listing 8-15. The netstat -platune Command Provides an Exhaustive Overview of Everything Happening on Your Computer root@ZNA:~# netstat -platune Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State User\ Inode PID/Program name tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 103\ 12937 3839/mysqld tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 0\ 13209 3965/apache2 tcp 0 0 10.0.0.20:53 0.0.0.0:* LISTEN 104\ 13737 3737/named tcp 0 0 10.0.0.30:53 0.0.0.0:* LISTEN 104\ 13735 3737/named tcp 0 0 10.0.0.10:53 0.0.0.0:* LISTEN 104\ 13733 3737/named tcp 0 0 192.168.1.33:53 0.0.0.0:* LISTEN 104\ 12821 3737/named tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 104\ 12819 3737/named tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 104\ 12824 3737/named tcp6 0 0 :::53 :::* LISTEN\ 104 12816 3737/named tcp6 0 0 :::22 :::* LISTEN\ 0 13585 4150/sshd tcp6 0 0 ::1:953 :::* LISTEN\ 104 12825 3737/named tcp6 0 0 ::ffff:192.168.1.33:22 ::ffff:192.168.1.6:4197 ESTABLISHED0\ 13761 4229/1 tcp6 0 164 ::ffff:192.168.1.33:22 ::ffff:192.168.1.7:9688 ESTABLISHED0\ 13609 4158/0 udp 0 0 0.0.0.0:1024 0.0.0.0:* 104\ 12822 3737/named udp 0 0 10.0.0.20:53 0.0.0.0:* 104\ 13736 3737/named udp 0 0 10.0.0.30:53 0.0.0.0:* 104\ 13734 3737/named udp 0 0 10.0.0.10:53 0.0.0.0:* 104\ 13732 3737/named udp 0 0 192.168.1.33:53 0.0.0.0:* 104\ 12820 3737/named udp 0 0 127.0.0.1:53 0.0.0.0:* 104\

CHAPTER 8 ■ MAKING A CONNECTION

udp6 udp6

12818 0 12823 0 12815

3737/named 0 :::1025 3737/named 0 :::53 3737/named

:::*

104\

:::*

104\

As you can see, the netstat command yields a lot of information when used with the -platune options. Table 8-2 explains the information displayed in Listing 8-15. Table 8-2. Information Offered by netstat -platune

Item

Explanation

Proto

The protocol that’s used; can be TCP or UDP.

Recv-Q

The number of packets waiting in the receive queue for this port at the moment that netstat was used.

Send-Q

The number of packets waiting to be sent from this port at the moment that netstat was used.

Local address

The local socket address (the local IP address followed by the port number that’s used).

Foreign address

The address of the foreign host (if any) that currently has an open connection to this host.

State

The current state of the protocol connected to the mentioned port.

User

The numeric user ID of the user with whose permissions the process was started.

Inode

The inode(s) of files that currently are opened by the process.

PID/program name

The PID and name of the program that has currently claimed the mentioned port.

As you can see, netstat provides a complete overview of what’s happening on your server. It’s especially useful if you get error messages like “port already in use.” In combination with the grep utility, it’s easy to learn what port program is currently holding a port open and, if required, to terminate that program. For example, to find out what program is currently occupying port 80, use netstat -platune | grep 80. This returns a line like this: root@ZNA:~# netstat -platune | grep 80 tcp 0 0 0.0.0.0:80 0.0.0.0:* 13209 3965/apache2

LISTEN

0\

From this line, you can see that an Apache web server with a PID of 3965 is currently listening on port 80. Want to remove it? Use kill 3965 and it’s gone.

Using nmap to Check Service Availability on Remote Servers The netstat command is a useful tool, but it works only on the host where you run it. Sometimes, when you cannot connect to a given service on a given host, you’d like to know if the service is available at all. You can do this with the nmap command. (Use apt-get install nmap to install it.) Like most powerful network tools, nmap also works best if you are root.

245

246

CHAPTER 8 ■ MAKING A CONNECTION

The nmap command is an expert tool that helps you find out exactly what’s happening at another host. If you use it properly, the owner of that host will never even know that you were there. However, you should be aware that running a so-called port scan to monitor open ports on a given host is considered an intrusion by many administrators, so be careful about what you’re doing with it because you might run into trouble if you use nmap on a host that isn’t yours and you haven’t notified its owner. If you really want to keep things simple, just use nmap without arguments. For example, nmap 192.168.1.69 performs a basic scan on host 192.168.1.69 to find what common ports are open on it. This gives good results for day-to-day use; see Listing 8-16 for an example. Listing 8-16. The nmap Command Shows You What Services Are Offered by a Host root@ZNA:~# nmap 192.168.1.69 Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-01 11:08 EDT Interesting ports on 192.168.1.69: Not shown: 1693 closed ports PORT STATE SERVICE 22/tcp open ssh 111/tcp open rpcbind 139/tcp open netbios-ssn 445/tcp open microsoft-ds MAC Address: 00:18:8B:AC:C9:54 (Dell) Nmap finished: 1 IP address (1 host up) scanned in 0.626 seconds A very common reason why the test shown in Listing 8-16 could fail is that nmap normally tries to ping its targets first. On many hosts, ping commands are administratively prohibited, dropped, or ignored. And these hosts won’t reveal anything when you issue nmap on them. To make sure that they’re working even when you cannot ping, use the -P0 option to disable ping. Another useful option is -O, which tries to guess the operating system that is on the target host. And if you want to make sure that both TCP and UDP ports are scanned, you should include -sT and -sU as well. So the command becomes somewhat longer: nmap -sT -sU -P0 -O 192.168.1.69 would scan the target host with all those options. You’ll notice that because nmap has to do a lot more work with these options, it takes considerably longer for the command to complete. Listing 8-17 shows the result of this scan. Listing 8-17. You Have Lots of Options to Specify How nmap Should Do Its Work root@ZNA:~# nmap -sT -sU -P0 -O 192.168.1.69 Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-01 11:11 EDT Interesting ports on 192.168.1.69: Not shown: 3176 closed ports PORT STATE SERVICE 22/tcp open ssh 111/tcp open rpcbind 139/tcp open netbios-ssn

CHAPTER 8 ■ MAKING A CONNECTION

445/tcp open microsoft-ds 68/udp open|filtered dhcpc 111/udp open|filtered rpcbind 631/udp open|filtered unknown 5353/udp open|filtered zeroconf 32768/udp open|filtered omad MAC Address: 00:18:8B:AC:C9:54 (Dell) Device type: general purpose Running: Linux 2.6.X OS details: Linux 2.6.14 - 2.6.17 Uptime: 0.176 days (since Wed Aug 1 07:23:05 2007) Network Distance: 1 hop OS detection performed. Please report any incorrect results at➥ http://insecure.org/ nmap/submit/ . Nmap finished: 1 IP address (1 host up) scanned in 1482.860 seconds In the last command, you’ll most likely get a better result, but there’s still a problem: the scan is rather noisy, and so the target host might log messages to tell its owner that you’re using nmap on it. There’s nothing wrong with that in most cases, but if you really want to put nmap through a thorough security test, you should use some stealth options such as -sF (FINscan), -sX (X-mas tree scan), or -sN (NULL-scan). All of these options use specific properties of the IP protocol to perform a stealth scan so that the target host never knows you were there. The disadvantage of these scan options is that they don’t always work! On many modern operating systems, you’ll find that the operating system ignores them, so you’ll end up waiting a long time without a result.

Monitoring the Network Interface Two useful tools are available to monitor what’s happening on your servers’ network cards. IPTraf offers a menu-driven interface from which you can monitor protocol activity, and the iftop utility shows how much bandwidth is used by a given connection.

Monitoring Protocol Activity with IPTraf IPTraf is another useful tool to see what’s happening on the network. It’s not installed by default, however, so make sure that it’s installed before you try to launch it from the command line with the iptraf command. (If it’s not installed yet, use apt-get install iptraf.) After launching it as root, you’ll see the menu interface (see Figure 8-1). You have several options in this interface: • IP traffic monitor: This option tells IPTraf to monitor what’s happening on the network interfaces in your server. You can select one particular network interface, but it’s possible to check all the interfaces as well. When a connection is established, you’ll see the connection happening in real time, indicating with what other node the connection is established and how many packets are flowing across that connection.

247

248

CHAPTER 8 ■ MAKING A CONNECTION

Figure 8-1. The IPTraf tool offers different menu options to see what’s happening on your server. • General interface statistics: This option provides generic information on what’s happening on a network board. You’ll see information such as the number of packets sent and received by the network interface, which is a good statistical overview of what’s happening on a network board. • Detailed interface statistics: As you would guess, this option provides more detail, such as the number of sent packets of a specific protocol type (see Figure 8-2).

Figure 8-2. If you choose to view the detailed interface statistics, you’ll see how many packets of a given protocol type are sent on an interface.

CHAPTER 8 ■ MAKING A CONNECTION

• Statistical breakdown: This option lets you divide the incoming information into different columns, sorted by the protocols in use. • LAN station monitor: This option provides an overview of the most active stations on the LAN. However, be aware that only those packets coming in on the host where you are running IPTraf are seen, unless you’re connected directly to the monitoring port of a switch. Apart from these options that you can use to specify how IPTraf should do its work, you also have a filter option and a configure option. The filter option is used to specify what kind of packets you want to see, and the configure option is used to configure IPTraf itself. For example, there’s an option that allows you to specify what colors are used in the IPTraf interface.

Monitoring Bandwidth Usage with the iftop Utility The iftop utility is simple but efficient. It shows you who has an open connection to a network card on your server and how much bandwidth they’re consuming. It displays a summary total of transmitted and received packets for the selected network card, but a progress bar also provides a visual indication of the actual bandwidth usage of the given connection. As root, run iftop from the command line (if it’s not installed yet, use apt-get install iftop to install it), and it will display the results window shown in Figure 8-3. On servers with more than one network card, don’t forget to specify on which network card you want to run iftop. You can do that with the option –i; for instance, iftop –i eth1 would show you usage statistics for eth1.

Figure 8-3. The iftop utility displays actual bandwidth usage on your server.

249

250

CHAPTER 8 ■ MAKING A CONNECTION

Monitoring Network Traffic So, you’ve seen the tools that will help you monitor the local network cards of your server, but there are also some excellent tools to see what’s happening on the network. The mother of all of these tools is tcpdump, which just dumps IP packets on the console you run it from. This tool is for the hardcore system administrator because it provides lots of information that normally scrolls by much too fast to see what’s happening. Listing 8-18 shows the results of the tcpdump command. Listing 8-18. When Using tcpdump, You’ll See Packet Headers Flying by on Your Server’s Console root@RNA:/ # tcpdump 16:00:21.044803 IP ida.lan.9603 > RNA.lan.ssh: 16:00:21.044856 IP RNA.lan.ssh > ida.lan.9603: 13936 16:00:21.044945 IP RNA.lan.ssh > ida.lan.9603: 13936 16:00:21.045023 IP ida.lan.9603 > RNA.lan.ssh: 64159 16:00:21.045076 IP RNA.lan.ssh > ida.lan.9603: 13936 16:00:21.045166 IP RNA.lan.ssh > ida.lan.9603: 13936 16:00:21.045220 IP RNA.lan.ssh > ida.lan.9603: 13936 16:00:21.045267 IP ida.lan.9603 > RNA.lan.ssh: 16:00:21.045336 IP RNA.lan.ssh > ida.lan.9603: 13936 ... 23826 packets captured 24116 packets received by filter 288 packets dropped by kernel

. ack 2705288 win 64503 P 2705420:2705632(212) ack 12377 win\ P 2705632:2705844(212) ack 12377 win\ . ack 2705632 win\ P 2705844:2705976(132) ack 12377 win\ P 2705976:2706188(212) ack 12377 win\ P 2706188:2706320(132) ack 12377 win\ . ack 2705976 win 65535 P 2706320:2706452(132) ack 12377 win\

Wireshark is built on top of tcpdump and can be used to view network packets from a graphical interface. This allows you to see what protocols are used, who is sending the packets, and even what is inside them. Before starting with these tools, however, you should know one thing: you can monitor only what you can see. If you’re on a shared network in which every node sees every packet coming by, it’s easy to monitor everything sent by all hosts on the network. But this isn’t the case on modern switched networks. If your computer is connected to a switch, you can see only those packets that are addressed to the host from where you run the monitoring software. To see the packets sent by others, you need a specialized tool, like the ARP poisoning tool Ettercap. (This is a very dangerous tool that can severely disturb network communications, and I won’t be covering it in this book.) Another way of seeing all packets that are sent on the network is to connect the computer on which you’re capturing packets to your switch’s management port. This allows you to see all the packets sent on the network.

CHAPTER 8 ■ MAKING A CONNECTION

Using tcpdump Because tcpdump is a very straightforward tool, it does exactly what its name promises: it dumps TCP packets on the console of your machine so you can see all packets received by the network board of your server. By default, it shows the first 96 bytes of every packet, but, if you need more details, you can start it with the -v option, or even the -vv option, so that it will be more verbose. On a very busy server, tcpdump isn’t very useful. The information passes by way too fast to see what’s happening, so it makes sense to pipe its output to a file and grep that file for the information that you really need. Although tcpdump is an excellent tool for capturing packets, it isn’t the best solution if you want to do something with the captured packets afterward. That would be Wireshark.

Analyzing Packets with Wireshark Wireshark provides a graphical interface that you can use to capture and analyze packets, so it doesn’t work directly on a server that’s running without X. Also, you wouldn’t want to install a complete GUI environment on your server just to run Wireshark. Instead of using it from your server directly, I recommend running it from a graphical workstation that is connected on the same network. It would show you the same information, anyway. To start Wireshark, as root install it using apt-get install wireshark. Then run the wireshark command from the run command box, which you can open from the graphical console with Ctrl+F2. To start a Wireshark capture session, select Capture ➤ Interfaces. From the pop-up window, select the interface that you want to use to perform your packet capture (see Figure 8-4). Click Start to start the packet capture.

Figure 8-4. Before starting a Wireshark packet capture, select the interface you want to perform the packet capture on and then click Start. Wireshark now starts filling its buffers. You won’t see any packet contents while this is happening. To see the content of the packet buffer, you need to click the Stop button. As a result, you’ll see the window shown in Figure 8-5. From this window you can sort packets and see packet details. To sort packets, click one of the columns. By default, they’re sorted on the number they came in with, but if you click the Source column, you can sort the packets by their source address; and if you click the Protocol column, you can sort the packets by the protocol that was used. Any of the columns in the results window can be clicked to filter the information.

251

252

CHAPTER 8 ■ MAKING A CONNECTION

Figure 8-5. You can browse the contents of packets from this results window. Click one of the packets to display more detail. For every packet that’s captured, you can analyze all its layers. The top part of the Wireshark capture results window displays just the list of packets, but after selecting a packet in the lower part, you’ll see the packet’s different headers. If you really need to see details of any of these parts, click the part you want to zoom in on to display its contents. You might even see passwords being sent in plain text over the network.

Connecting Remotely with SSH The essence of SSH is its security, and public and private keys naturally play an important role in it. On first making contact, the client and the server exchange public and private keys. In this communication, the server creates a key based on its private key—the so-called host key— and uses it as its proof of identity. When connecting, the server sends its public key to the client. If this is the first time the client has connected to this host, the host replies with the message shown in Listing 8-19. Listing 8-19. Establishing an SSH Session with an Unknown Host root@ZNA:~# ssh 192.168.1.70 The authenticity of host '192.168.1.70 (192.168.1.70)' can't be established. RSA key fingerprint is fd:07:f6:ce:5d:df:6f:a2:84:38:c7:89:f1:3a:a6:34. Are you sure you want to continue connecting (yes/no)? yes

CHAPTER 8 ■ MAKING A CONNECTION

Warning: Permanently added '192.168.1.70' (RSA) to the list of known hosts. Password: Last login: Tue Jul 31 15:34:15 2007 from ida.lan root@RNA:~# If the client trusts that this is really the intended host, it should answer yes to the request, in which case the host is then added to the .ssh/known_hosts file in the home directory of the user who initiated the SSH session. The next time the client connects to the host, this file is checked to see if the host is already known. The check is based on the public key fingerprint of the host, which is a unique checksum related to the public key of the host. The connection is established only if this check matches the name and public key of the server that the client is connecting to. If these two pieces of data don’t match, it’s very likely that the host the client is trying to connect to isn’t the intended host, and the connection is refused. Once the identity of the server you want to connect to is established, a secured channel is set up between the client and server. These secured channels are established by a session key, which is an encryption key that’s the same on both the server and the client and encrypts all data sent between the two machines. The client and the server negotiate this session key based on their public keys. One of the things determined in this negotiation is the protocol that should be used. For example, session keys can use different encryption protocols such as 3DES, Blowfish, or IDEA. After establishing the secured channel, the user on the client is asked for credentials; if nothing is configured, a prompt asks the user to enter his user name and password. Alternatively, the user can authenticate with his public/private key pair, thus proving that he really is the user that he says he is, but some more things have to be configured before that can happen. All this might sound pretty complicated, but the nice thing is that the user doesn’t notice any of it. The user just has to enter a user name and a password. If, however, you want to move beyond simple password-based authentication, it’s necessary to understand what’s happening.

Working with Public/Private Key Pairs The security of SSH relies on the use of public/private key pairs. By default, the client tries to authenticate using RSA or DSA key pairs. To make this work, the server must have the client’s public key, which is something that you have to configure by hand, as you’ll see later. When the client has a public/private key pair, it generates an encrypted string with its private key. If the server can decrypt this string using the client’s public key, the client’s identity is authenticated. When using public/private key pairs, you can configure different things. First, the user needs to determine what cryptographic algorithm she wants to use. For this purpose, she can choose between RSA and DSA (DSA is considered stronger). Next, she has to decide if she wants to protect her private key with a passphrase. Using a passphrase is important because the private key really is used as the identity of the user. Should anyone steal this private key, it would be possible to forge the identity of the key’s owner, so it’s a very good idea to secure private keys with a passphrase.

253

254

CHAPTER 8 ■ MAKING A CONNECTION

Working with Secure Shell Basically, Secure Shell is a suite of tools that consists of three main programs and a daemon: sshd. Before being able to use it, of course, you have to install it using apt-get install openssh-server (you might have installed it already using the SSH installation pattern when you installed your server). The tools are ssh, scp, and sftp. The first, ssh, is used to establish a secured remote session. Let’s say that it’s like telnet but cryptographically secured. The second, scp, is a very useful command that’s used to copy files to and from another server where the SSH process is running. The third, sftp, is a secure FTP client interface. Using it establishes a secured FTP session to a server that’s running the sshd. Two of the best things of all these tools are that they can be used without any preparation or setup, and you can set them up to work entirely according to your needs. They are at once easy-to-use and very specialized tools.

Using the ssh Command The simplest way to work with SSH is to just enter the ssh command, followed by the name of the host you want to connect to. For example, to connect to the host AMS.sandervanvugt.com, use ssh AMS.sandervanvugt.com. Depending on whether you’ve connected to that host before, it may check the host credentials or just ask for your password. The ssh command doesn’t ask for a user name because it assumes that you want to connect to the other host with the same identity that you’re logged in with locally. If you’d rather log in with another user account, you can indicate this intention in one of two ways. You can specify the user name and follow it with an at sign (@) when establishing the connection to the remote host, and you can also use the -l option followed by the name of the user account you want to use to connect to the other host. So ssh [email protected] and ssh -l linda AMS.sandervanvugt.com accomplish the same thing. After establishing a session, use the exit command (or Ctrl+D) to close the session and return to your own machine. Now, it seems a lot of trouble to log in to a remote host if you just need to enter one or two commands. If you face this situation often, it’s good to know that you can just specify the name of the command at the end of the ssh command: ssh -l [email protected] ls -l provides a long listing of files that user linda has in her home directory at the other host. Of course, this isn’t the most realistic example of how to use “one command only” sessions to a host, but you probably can see its value when working from shell scripts.

Using scp to Copy Files Securely The scp command is another part of the SSH suite that you’ll definitely like. It’s used to copy files securely. If you know how the cp command works, you’ll know how to handle scp. The only difference is that it requires a complete network pathname, including the names of the host and the file you want to copy. Also, if you don’t want to use the name of the user you are currently logged in as, a user name should be included as well. Consider the following example: scp /some/file [email protected]:/some/file This easy command copies /some/file to AMS.sandervanvugt.com and places it in the directory /some/file on that host. Of course, it’s possible to do the opposite as well: scp

CHAPTER 8 ■ MAKING A CONNECTION

[email protected]:/some/file /some/file copies /some/file from a remote host with the name SFO.sandervanvugt.com to the localhost. You’ll like the -r option as well because it allows you to copy a complete subdirectory structure.

Using sftp for Secured FTP Sessions As an alternative to copying files with scp, you can use the sftp command. This command is used to connect to servers running the sshd program and to establish a secured FTP session with it. From the sftp command, you have an interface that really looks a lot like the normal FTP client interface. All the standard FTP commands work here as well, except that it’s secure in this case. For example, you can use the ls and cd commands to browse to a directory and see what files are available and use the get command from there to copy a file to the current local directory.

Configuring SSH In an SSH environment, a node can be client and server simultaneously. So, as you can imagine, there’s a configuration file for both of these aspects. The client is configured in /etc/ssh/ ssh_config, and the server uses /etc/ssh/sshd_config. Setting options for the server isn’t hard to understand: just put them in the configuration file for the /etc/ssh/sshd_config daemon. For the client settings, however, the situation is more complicated because there are several ways of overwriting the default client settings: • The generic /etc/ssh/ssh_config file is applied to all users initiating an SSH session. An individual user can overwrite them if he creates a .ssh_config file in the .ssh directory of his home directory. • An option in /etc/ssh/ssh_config has to be supported by the sshd_config file on the server you are connecting to. For example, if you’re allowing password-based authentication from the client side, but the server doesn’t allow it, it won’t work. • Options in both files can be overwritten with command-line options. Table 8-3 is an overview of some of the most useful options that you can use to configure the client in ssh_config. Table 8-3. Useful options in ssh_config

Option

Description

Host

This option restricts the following declarations (up to the next Host keyword) to a specific host. Therefore, this option is applied on a host that a user is connecting to. The host name is taken as specified on the command line. Use this parameter to add some extra security to specific hosts. You can also use wildcards such as * and ? to refer to more than one host name.

CheckHostIP

If this option is set to yes (the default value), SSH will check the host IP address in the known_hosts file. Use this as a protection against DNS or IP address spoofing. Continued

255

256

CHAPTER 8 ■ MAKING A CONNECTION

Table 8-3. Continued

Option

Description

Ciphers

This option, which takes multiple arguments, is used to specify the order in which the different encryption algorithms should be tried to use in an SSHv2 session (version 2 is the default SSH version nowadays).

Compression

The yes/no values for this option specify whether to use compression. The default is no.

ForwardX11

This very useful option specifies if X11 connections will be forwarded. If set to yes, graphical screens from an SSH session can be forwarded through a secure tunnel. The result is that the DISPLAY environment variable that determines where to draw graphical screens is set correctly. If you don’t want to enable X forwarding by default, use the -X option on the command line when establishing an SSH session.

LocalForward

This option specifies that a TCP/IP port on the local machine is forwarded over SSH to the specified port on a remote machine. (See “Generic TCP Port Forwarding” later in this chapter for more details.)

LogLevel

Use this option to specify the level of verbosity for log messages. The default value is INFO. If this doesn’t go deep enough, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3 provide progressively more information.

PasswordAuthentication

Use this option to specify whether or not you want to use password authentication. By default, password authentication is used. In a secure environment in which keys are used for authentication, you can safely set this option to “no” to disable password authentication completely.

Protocol

This option specifies the protocol version that SSH should use. The default value is set to 2,1 (which indicates that version 2 should be used first and, if that doesn’t work, version 1 is tried). It’s a good idea to disable version 1 completely because it has some known security issues.

PubkeyAuthentication

Use this option to specify whether you want to use public key–based authentication. This option should always be set to the default value (yes) because public key–based authentication is the safest way of authenticating.

The counterpart of ssh_config on the client computer is the sshd_config file on the server. Many options that you can use in the ssh_config file are also available in the sshd_ config file. However, some options are specific to the server side of SSH. Table 8-4 gives an overview of some of these options.

CHAPTER 8 ■ MAKING A CONNECTION

Table 8-4. Important Options in sshd_config

Option

Description

AllowTcpForwarding

Use this option to specify whether you want to allow clients to do TCP port forwarding. This is a very useful feature, and you’ll probably want to leave it at its default value (yes).

Port

Use this option to specify the port that the server is listening on. By default, sshd is listening on port 22. If the SSH process is connected directly to the Internet, this will cause many people to try a brute-force attack on your server. Consider running the SSH process on some other port for increased security.

PermitRootLogin

Use this option to specify whether you want to allow root logins. To add additional security to your server, consider setting this option to the no value. If set to no, the root user has to establish a connection as a normal user and from there use su to become root or use sudo to perform certain tasks with root permissions.

PermitEmptyPasswords

Use this option to specify if you want to allow users to log in with an empty password. From a security perspective, this isn’t a very good idea, so the default no value is suitable in most cases. If, however, you want to run SSH from a script and establish a connection without entering a password, it can be useful to change the value of this parameter to yes.

ChallengeResponseAuthentication

This option specifies whether users are allowed to log in using passwords. If you want to add additional security to your server by forcing users to log in with public/private key pairs only, give this parameter the value no.

X11Forwarding

Use this option to specify if you want to allow clients to use X11 forwarding. On Ubuntu Server, the default value for this parameter is yes.

Using Key-Based Authentication Now that you know all about the basics of SSH, let’s look at some of the more advanced options. One of the most important is key-based authentication, which SSH uses via public/private key–based authentication. Before diving into the configuration of key-based authentication, let’s first have a look on how these keys are used.

A Short Introduction to Cryptography In general, you can use two methods for encryption: symmetric and asymmetric. Symmetric encryption is faster but less secure, and asymmetric encryption is slower but more secure. In a symmetric key environment, both parties use the same key to encrypt and decrypt messages. With asymmetric keys, a public and a private key are used, and this is the important technique that’s used for SSH.

257

258

CHAPTER 8 ■ MAKING A CONNECTION

If asymmetric keys are used, every user needs his own public/private key pair and every server needs a pair of them as well. Of these keys, the private key must be protected at all times; if the private key is compromised, the identity of the owner of the private key is compromised as well. In short, stealing a user’s private key is like stealing their identity. Therefore, a private key is normally stored in a very secure place where no one other than its owner can access it; typically this is in ~/.ssh. The public key, on the other hand, is available to everyone. Public/private keys are generally used for three purposes: encryption, authentication, and non-repudiation. To send an encrypted message, the sender encrypts the message with the public key of the receiver who can decrypt it with the matching private key. This scenario requires that before you send an encrypted message, you have the public key of the person you want to send the message to. The other options are to use public/private keys for authentication or to prove that a message has not changed since it was created. This method is known as nonrepudiation. In the example of authentication, the private key is used to generate an encrypted token, the salt. If this salt can be decrypted with the public key of the person who wants to authenticate, that proves that the server really is dealing with the right person, and access can be granted. However, this technique requires the public key to be copied to the server before any authentication can occur, which is also the case when keys are used to prove that a message hasn’t been tampered with.

Using Public/Private Key–Based Authentication in an SSH Environment When SSH key-based authentication is used, you must make sure that for all users who need to use this technology, the public key is available on the servers they want to log in to. When logging in, the user creates an authentication request that’s signed with the user’s private key. This authentication request is matched to the public key of the same user on the server where that user wants to be authenticated. If it matches, the user is allowed access; if it doesn’t, user access is denied. Public/private key–based authentication is enabled by default on Ubuntu Server, so it’s only when no keys are present that the server prompts users for a password. The following steps provide a summary of what happens when a user tries to establish an SSH session with a server: 1. If public key authentication is enabled (the default), SSH checks the .ssh directory in the user’s home directory to see if a private key is present. 2. If a private key is found, SSH creates a packet with some data in it (the salt), encrypts that packet with the private key, and sends it to the server. The public key is also sent with this packet. 3. The server now checks whether a file with the name authorized_keys exists in the home directory of the user. If it doesn’t, the user can’t be authenticated with his keys. If the file does exist, and the public key is an allowed key (and also is identical to the key that was previously stored on the server), the server uses this key to check the signature.

CHAPTER 8 ■ MAKING A CONNECTION

4. If the signature is verified, the user is granted access. If the signature can’t be verified, the server prompts the user for a password instead. All this sounds pretty complicated, but it really isn’t. Everything happens transparently if it has been set up right. Also, there’s hardly any noticeable delay when establishing a connection. It normally takes no more than a second.

Setting Up SSH for Key-Based Authentication The best way to explain how to set up SSH for key-based authentication is by working through an example. In the following procedure, key-based authentication is enabled for the user root. 1. On the desktop where root is working, use the ssh-keygen -t dsa -b 1024 command. This generates a public/private key pair of 1,024 bits. Listing 8-20 shows what happens. Listing 8-20. Generating a Public/Private Key Pair with ssh-keygen workstation # ssh-keygen -t dsa -b 1024 Generating public/private dsa key pair. Enter file in which to save the key (/root/.ssh/id_dsa) : Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_dsa. Your public key has been saved in /root/.ssh/id_dsa.pub. The key fingerprint is: 59:63:b5:a0:c5:2c:b5:b8:2f:99:80:5b:43:77:3c:dd root@workstation I’ll explain what happens. The user in this example uses the ssh-keygen command to generate a public and a private key. The encryption algorithm used to generate this key is DSA, which is considered more secure than its alternative, RSA. The option -b 1024 specifies that 1024-bit encryption should be used for the key. The longer this number, the more secure it is. Notice, however, that a many-bits encryption algorithm also requires more system resources to use it. After generating the keys, the command prompts you to save it somewhere. By default, a directory with the name .ssh is created in your home directory and, within this directory, a file with the name id_dsa. This file contains the private key. Next, you’re prompted to enter a passphrase, which is an important extra layer of protection that can be added to the key. Because anyone who has access to your private key (which isn’t that easy) can forge your identity, your private key should always be protected with a passphrase. After entering the same passphrase twice, the private key is saved, and the related public key is generated and saved in the /root/.ssh/id_dsa.pub file. Also, a key fingerprint is generated. This fingerprint is a summary of your key, a checksum that’s calculated on the key to alert you if the key has been changed. Make sure that your passphrase is not too easy to guess; a weak passphrase makes a strong key useless.

259

260

CHAPTER 8 ■ MAKING A CONNECTION

2. After creating the public/private key pair, you must transfer the public key to the server. The ultimate goal is to place the contents of the id_dsa.pub file in the /root/ .shh/authorized_keys file on the server. But you can’t simply copy the file to the authorized_keys destination file because other keys may already be stored there. Therefore, first use scp to copy the file to a temporary location. The command scp /root/.ssh/id_dsa.pub root@server:/root/from_workstation_key.pub would do the job. 3. Now that the public key is on the server, you have to put it in the authorized_keys file. Before doing this, though, make sure that the .ssh directory exists on the server in the home directory of the user root, and that it has user and group root as its owner and permission mode 700. Then, on the server with /root as your current directory, use cat from_workstation_key.pub >> .ssh/authorized_keys. This command appends the content of the public key file to the authorized_keys file, thus not overwriting any file that may have been there already. 4. Hopefully, no errors have occurred, and you’ve been successful. Go back to your workstation and start an SSH session to the server in which you just copied your public key to the authorized_keys file. You’ll notice that you are no longer prompted for a password, but for a passphrase instead. This proves that everything worked. Do notice, however, that you need to repeat this procedure for every key-secured server with which you want to be able to establish a session. Working with keys as described in these steps is an excellent way to make SSH authentication more secure. But there’s a drawback: if you need to establish an SSH session automatically from a shell script or cron job, it’s not very handy if you’re first prompted for a key. Therefore, some method is needed to execute such jobs automatically. One solution is to create a special user account with limited permissions and without a passphrase on its private key. Another solution is to run ssh-agent, which caches the keys before they are used (you’ll learn how to do this in the next section).

Caching Keys with ssh-agent You can use ssh-agent to save yourself from constantly having to enter private keys. With this program, you can cache keys for a given shell environment. After starting ssh-agent from a shell prompt, you need to add the passphrase for the private key that belongs to it. This is something that you’ll do for a specific shell, so after you close that specific shell or load another shell, you’ll need to add the passphrase to that shell again. After adding a passphrase to ssh-agent, the passphrase is stored in RAM, and only the user who added the key to RAM can read it from there. Also, ssh-agent listens only to the ssh and scp commands that you’ve started locally, so there’s no way you can access a key that is kept by ssh-agent over the network. So you can be sure that using ssh-agent is pretty secure. Apart from being secure, it’s pretty easy as well. Enabling ssh-agent and adding a passphrase to it is a simple two-step procedure:

CHAPTER 8 ■ MAKING A CONNECTION

1. From the shell prompt, use ssh-agent followed by the name of the shell you want to use it from. For example, use ssh-agent /bin/bash to activate ssh-agent for the Bash shell. 2. Now type ssh-add. You’ll be prompted for the passphrase of your current private key, and you’ll then see the message identity added, followed by the private key whose passphrase is added to ssh-agent.

■Tip Secure Shell is a great way of accessing other hosts. But did you know that you can also use it to mount a file system on a remote computer? All modern versions of SSH support this feature: just use sshfs for access to all the files and directories on the remote server, just like a local user on that server. If you know how to mount a directory with the mount command, working with sshfs is easy. For example, the command sshfs linda@AMS:/data /mnt allows access to the /data directory on the remote server and connects that directory to /mnt on the local server. Secure Shell is not installed by default, so use apt-get install sshfs to install it on your server.

Tunneling Traffic with SSH Apart from establishing remote login sessions, copying files, and executing commands on remote hosts, you can also use SSH for TCP port forwarding. When used like this, SSH is a simple VPN solution with the capability of tunneling to almost any unsecured protocol over a secured connection. In this section, I’ll first talk about X forwarding and then you’ll see how to forward almost any protocol using SSH.

X Forwarding Wouldn’t it be useful if you could start an application on a server, where all the workload is performed by the server while you control the application from your client? Well, you can with SSH X forwarding. To use X forwarding, you first must establish an SSH session to the server you want to connect to. Next, from this SSH session, you start the graphical application, which will draw its screen on your workstation while doing all the work on the server itself. Sounds good? Establishing such an environment has only two requirements: • Make sure that the X11Forwarding option is set to yes in /etc/ssh/sshd_config on the server. • Connect to the server with the ssh -X command from your client. Alternatively, you can set the X11Forwarding option in the client configuration file /etc/ssh/ssh_config, which allows you to forward graphical sessions by default. This poses a minor security problem, however, so this setting is not enabled by default on Ubuntu Server. Now that you have established the SSH session with your server, start your favorite graphical program. The program itself will be executed at the remote host, and you’ll see the screen locally.

261

262

CHAPTER 8 ■ MAKING A CONNECTION

■Note X-forwarding sessions with SSH are really cool, but there is a limitation: you need an X server on the client from which you are establishing the SSH session. This X server is used as the driver for your graphical hardware, and the application that you want to run on your client needs it to display its screens. This won’t be a problem on Linux, UNIX, or Macintosh machines because an X server is present by default. It’s a problem on Windows, however. The most common SSH client for Windows is PuTTY, which, although very useful, doesn’t contain an X server. A good X server for Windows is Xming, which is a free X server that you can download from the Internet.

Generic TCP Port Forwarding X is the only service for which port forwarding is hard-coded in the SSH software. For everything else, you need to do it by hand using the -L (local forwarding) or the -R (remote port forwarding) options. Let’s have a look at the example in Figure 8-6.

Figure 8-6. Example network This network has three nodes: AMS is the node in which the administrator is working; ATL is the node in the middle; and AMS has a direct connection to ATL, but not to SLC which is behind a firewall. ATL does have a direct connection to SLC and is not obstructed by any firewall. The following command illustrates a simple case of port forwarding: linda@AMS:~> ssh -L 4444:ATL:110 linda@ATL In this example, user linda forwards connections to port 4444 on her localhost to port 110 on the host ATL as user linda on that host. This is how you would establish a secure session to the insecure POP service on that host, for example. The localhost first establishes a connection to the SSH server running on ATL. This SSH server connects to port 110 at ATL, whereas ssh binds to port 4444 on the localhost. Now an encrypted session is established between local port 4444 and server port 110: everything sent to port 4444 on the localhost really goes to port 110 at the server. If, for example, you configured your POP mail client to get its mail from local port 4444, it would really get it from port 110 at ATL. Notice that a nonprivileged port is used in this example. Only user root can connect to a privileged port with a port number lower than 1024. No matter what port you are connecting to, you should always check in the /etc/services services configuration file, in which port numbers are matched to names of services, what the port is normally used for (if anything), and use netstat -platune | grep to make sure that the port is not already in use.

CHAPTER 8 ■ MAKING A CONNECTION

A little variation on local port forwarding, as just seen, is remote port forwarding. If you want to try it, forward all connections on a remote port at a remote server to a local port on your machine. To do this, use the -R option as in the following example: linda@AMS:~> ssh -R 4444:AMS:110 linda@ATL In this example, user linda connects to host ATL (see the last part of the command). On this remote host, port 4444 is addressed by using the construction -R 4444. This remote port is redirected to port 110 on the localhost. As a result, anything going to port 4444 on ATL is redirected to port 110 on AMS. This example would be useful if ATL were the client and AMS were the server running a POP mail server that user linda wants to connect to. Another very useful instance is when the host you want to forward to cannot be reached directly, perhaps because it is behind a firewall. In this case, you can establish a tunnel to another host that is reachable with SSH. Imagine that in Figure 8-6, the host SLC is running a POP mail server that our user linda wants to connect to. This user would use the following command: linda@AMS:~> ssh -L 4444:SLC:110 linda@ATL In this example, linda forwards connections to port 4444 on her localhost to server ATL that is running SSH. This server, in turn, forwards the connection to port 110 on server SLC. Note that, in this scenario, the only requirement is that ATL has the SSH service activated; no sshd is needed on SLC for this to work. Also note that there is no need for host AMS to get in direct contact with SLC because that’s what ATL is used for. In these examples, you learned how to use the ssh command to accomplish port forwarding, but this isn’t the only way of doing it. If a port-forwarding connection needs to be available all the time, you can put it in the ssh configuration file at the client computer. Put it in .ssh/config in your home directory if you want it to work for your user account only, or put it in /etc/ssh/ssh_config if you want it to apply for all users on your machine. The parameter that should be used as an alternative to ssh -L 4444:ATL:110 would be LocalForward 4444 ATL:110.

Summary In this chapter you learned how to set up a network connection. First, we explored how an IP address is assigned to a network interface card. We covered IPv4 addresses as well as IPv6 addresses. Following that, you read how to troubleshoot a network connection using basic commands such as ping and traceroute, or advanced tools such as nmap and Wireshark. In the last part of this section, you learned how to create a remote session with SSH. In the next chapter, you’ll find out how to set up networking services such as NTP, DHCP, and DNS on your server.

263

CHAPTER

9

Configuring Network Infrastructure Services Using DNS, DHCP, and NTP L

inux servers are often used to configure services that help make networking happen. These services include DNS for name resolution, DHCP for IP address configuration, and NTP for time services. In this chapter, you’ll read how to configure them. You’ll also read how to enable some common Linux services using xinetd.

Configuring DNS As you would expect, IP (Internet protocol) is used for all communications on the Internet. This protocol specifies unique IP addresses that computers use to talk to one another. To contact a computer, you just need to know its IP address. One of the most important reasons why the domain name system (DNS) was developed is because computers work better with numbers than humans do, and humans tend to prefer names. So, DNS translates IP addresses to DNS names (and back from DNS names to IP addresses). In this chapter, you’ll learn how to configure DNS on Ubuntu Server.

Methods of Name Resolution Before going into detail about configuring DNS servers, you first need to learn exactly what DNS is and how it works. In this section, you’ll read about the differences between DNS and other methods of resolving names. You’ll also find out how the DNS hierarchy is structured and what roles the different types of DNS servers play in this hierarchy. DNS is not the only solution that you can use for name resolving. Let’s have a quick look at two of the alternative methods: the /etc/hosts file and Sun’s Network Information System (NIS).

Managing Host Name Information with the /etc/hosts File Before centralized systems such as NIS and DNS were introduced, every host kept its own file that mapped IP addresses to names. In the days when the Internet was called (D)ARPANet and was still a very small network, this was a feasible solution, although the administrator had to

265

266

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

make sure that these files were updated properly. Such a mechanism still exists, but in the form of the /etc/hosts file. In this file, you can keep a list of commonly used names and their IP addresses. Ubuntu Server creates this file by default to make sure that the localhost can be resolved. Listing 9-1 shows an example of the file. Note that you can still use this file as an addition to DNS. Depending on the settings in /etc/nsswitch.conf, its contents will be checked first before any DNS lookup. Listing 9-1. Displaying the Contents of /etc/hosts root@RNA:~# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 RNA.lan RNA # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

Using NIS to Manage Name Resolution A more convenient method that you can use to keep mappings between host names and IP addresses is Sun’s NIS, also known as the Yellow Pages. This system uses a database for important configuration files on a server, such as the /etc/hosts file, the /etc/passwd file, and /etc/shadow. As an administrator, you can determine for yourself what files to manage with NIS. These files are converted to NIS maps, which are the indexed files that comprise the NIS database. In NIS, one server is configured as the master server, which maintains the NIS database. All nodes are configured as NIS clients and send their name resolution requests to the NIS master. To provide redundancy, NIS can also use slave servers, which offer a read-only copy of the NIS master database. However, the master server is the single point of administration. Although NIS was a good solution to manage relevant information within a network, it never became very popular as an Internet-level name service mainly because NIS does not provide a hierarchical solution, only flat databases. All these flat databases are managed by local administrators, and there’s no relation among the databases that are used in different NIS domains. The large amount of information on the Internet today makes it impossible to get quick results from a structure like NIS. For this reason, most organizations that still use NIS are phasing it out and configuring DNS to resolve host names to IP addresses and LDAP to manage user information (therefore, NIS is not covered in this book).

Managing Search Order with the /etc/nsswitch.conf File Although DNS is the main system used for name resolution, it’s not the only one. You can set it up in parallel with a NIS system and the /etc/hosts file. If you do this, the order in which the

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

different systems are searched is important. The search order is determined by the /etc/nsswitch.conf file; see Listing 9-2 for an example. Listing 9-2. Contents of the /etc/nsswitch.conf File root@RNA:~# cat /etc/nsswitch.conf # /etc/nsswitch.conf # # Example configuration of GNU Name Service Switch functionality. # If you have the 'glibc-doc-reference' and 'info' packages installed, try: # 'info libc "Name Service Switch"' for information about this file. passwd: group: shadow:

compat compat compat

hosts: networks:

files dns files

protocols: services: ethers: rpc:

db db db db

netgroup:

nis

files files files files

For all the important information on your server, the nsswitch.conf file contains an indication of where it should be searched. In the case of hosts and network information, the example file is pretty clear: it first checks local configuration files and only after that does it check the DNS hierarchy. This means that you can use /etc/hosts to override information as defined in DNS.

Structure of the DNS Hierarchy The most important advantage offered by DNS is that it’s organized in a hierarchical manner. This makes the system very scalable because it can be extended by simply adding another branch to the tree-like hierarchy. On top of the hierarchy are the root servers, which have one purpose only: to provide information about the top-level domains (TLDs). Some fixed domain names are used for toplevel domains, including .com, .org, and .info. TLDs exist for all countries as well, such as .nl, .uk, .fr, and so on. Within these TLDs, persons and organizations can create their own domains, which can contain subdomains as well. For example, an organization could create a domain called example.com and, within the structure of example.com, it could create some subdomains as well, such as east.example.com and west.example.com. The number of subdomains is virtually unlimited, although it becomes hard to work with more than four or five levels of domains. No one wants to type www.servers.east.nl. sandervanvugt.com all the time, do they? Figure 9-1 provides an example of the partial DNS hierarchy.

267

268

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

Figure 9-1. Example of a part of the DNS hierarchy

Master and Slave Servers Within the DNS hierarchy, different servers are responsible for the data in specific domains (and sometimes subdomains as well). These are the so-called name servers and the part of the hierarchy that they are responsible for is the zone. A zone can include more than just one domain; for example, if one name server is responsible for everything in sandervanvugt.nl, including the subdomain’s servers and workstations, the complete zone is sandervanvugt.nl. If, however, there’s a subdomain called sales.sandervanvugt.com that has its own name server, the subdomain would be a zone by itself that is not part of sandervanvugt.com. Speaking in a very generic way, a zone is just a branch of the DNS hierarchy. All zones should have at least two name servers. The first is the master name server, which is ultimately responsible for the data in a zone. For fault tolerance and to make the information more accessible, it’s a good idea to use one or more slave servers as well. These slave servers will periodically get an update of all the data on the master server by means of a zone transfer; this is the process the master server uses to update the database on the slave server. Note that DNS uses a single-master model: updates are performed on the master server and nowhere else, and the databases on the slave servers are read-only. You should also know that the name servers do not need to be in the zone that they are responsible for. For example, the name server of a given domain will often be hosted by the Internet provider that (of course) has its own domain. You can maintain your own DNS server, and it’s useful to do so if your organization is larger than average, but you don’t have to. You can also just purchase a domain and have your Internet server do the name server maintenance work.

Connecting the Name Servers in the Hierarchy Because DNS uses a hierarchy, the servers in DNS need to know about each other, and this is a two-way process by its very nature. First, all servers in subordinate zones need to know where to find the root servers of the DNS hierarchy. Equally, the servers of the upper-level zones need to know how to find the servers of lower-level zones. You can very well create your own DNS domain called mynicednsdomain.com and run your DNS server in it, but it doesn’t make sense if

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

the DNS server that’s responsible for the .com domain doesn’t know about it. This is because a client trying to find your server will first ask the name server of the domain above your zone if it knows where to find authoritative information for your domain. This is why DNS domain names need to be registered. Only then can the manager of the domain above yours configure your name server as the responsible name server for your domain. This is the delegation of authority. It also helps to understand what happens when a user tries to resolve a DNS name that it doesn’t know about already. The next procedure describes what happens: 1. To resolve DNS names, you need to configure the DNS resolver on the user’s workstation or on the server that needs to be part of the DNS hierarchy. The DNS resolver is the part of the workstation where the user has configured how to find a DNS server. On a Linux system, this happens in the /etc/resolv.conf file. 2. Based on the information in the DNS resolver, the client contacts its preferred name server and asks that server to resolve the DNS name, no matter what server it is and where on Earth the server is running. So, if the client tries to resolve the name www.sandervanvugt.nl, it first asks its preferred name server. The advantage is that the client’s name server can consult its cache to find out whether it has recently resolved that name for the client. If it knows the IP address of the requested server, the DNS name server returns that information to the client immediately. 3. If the name server of the client doesn’t know the IP address of the requested server, it sees whether a forwarder is configured. (A forwarder is just a server that a name server contacts if it can’t resolve a name by itself.) 4. If no forwarder is configured, the DNS name server contacts a name server of the root domain and asks that name server how to contact the name server of the top-level domain it needs. In the example in which you want to reach the host www.sandervanvugt.nl, this is the name server for the .nl domain. 5. Once the name server of the client finds the name server address of the top-level domain, it contacts it and asks for the IP address of the authoritative name server for the domain it’s looking for. In our example, this would be the name server for sandervanvugt.nl. 6. Once the name server of the client finds out how to reach the authoritative name server for the domain the client asks for, it contacts that name server and asks to resolve its name. In return, the name server of the client receives the IP address it needs. 7. Ultimately, the IP address of the desired server is returned to the client, and contact can be established.

Resource Records To answer all name resolution requests, your DNS server needs to maintain a database in which it maintains the resource records, which contain different types of data to find specific information for a domain. Table 9-1 presents some of the most important types of data that can be maintained in that database. Later in this chapter you’ll learn how to add these

269

270

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

resource records to the DNS database. Listing 9-3 shows an example of the DNS database in which different resource record types are used. You’ll find an explanation of the resource record types in Table 9-1. Listing 9-3. Contents of the example.com Zone File RNA:/ # cat /etc/bind/db.example.com $TTL 2D @ IN SOA SFO.example.com. 2006080700 3H 1H 1W 1D ) example.com. example.com. sfo lax web

IN IN IN IN IN

MX NS A A CNAME

root.SFO.example.com. ( ; serial ; refresh ; retry ; expiry ; minimum

10 mail.example.com. lax.example.com. 201.100.100.10 201.100.100.40 sfo.example.com.

Table 9-1. Using the Important Resource Records

Resource Record

Use

MX

This resource record finds the mail servers for your domain. In the first column, you’ll find the name of the domain they are used for, and the fourth column reveals the primary mail server. The number 10 indicates the priority of this mail server. If more than one mail server is present in the domain, the mail server with the lowest priority number is used first. Following the priority number is the DNS name of the mail server.

NS

This resource record provides a list of name servers for this domain. Typically, you must have this resource record for all master and slave name servers of the domain. The first column reveals the name of the domain, and the fourth column provides the name of the server itself. Notice the dot at the end of the server name, which indicates it as an absolute name (a name that refers to the root of the DNS hierarchy directly).

A

The A resource record is used to define the relation between a host name and an IP address. The first column mentions the name of the host as it occurs in this domain, and the fourth column provides the IP address of this host.

CNAME

The CNAME (“common name”) resource record is used to define an alias, which is just a nickname that is used for a host. A CNAME should always refer to the real name of the host. Aliases can be useful if one server hosts many DNS names. In that case, use an A resource record for “myserver” and create CNAMEs that refer to the A resource record for all services provided by your server. This way, if you have to change the IP address of your server, you’ll change it only once.

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

Introducing Forward and Reverse DNS Before I start talking about the actual configuration of DNS, you need to know about reverse DNS. Translating names into IP addresses is one task of the DNS server, and its other task is translating IP addresses to names. This translation from address to name is called reverse DNS, and it’s necessary if you want to find the real name that is used by a given IP address. This feature is useful if you want names in your log files instead of IP addresses, but, if you want all IP addresses translated to names, you should realize that this comes at a cost in performance. For this reason, many services and commands allow you to specify whether to use reversed name resolution. To make name resolution for your domain possible, you should always configure it when setting up a DNS hierarchy. To create a reverse DNS structure, you need to configure a zone in the in-addr.arpa domain, under which a structure is created that contains the inverse IP addresses for your network. If, for example, you’re using the class C network 201.10.19.0/24, you should create a DNS domain with the name 19.10.201.in-addr.arpa. Within this zone, you next have to create a pointer (PTR) resource record for all of the hosts that you want to include in the DNS hierarchy. When working with reverse DNS, you should be aware of one important limitation: it doesn’t know how to handle non-default subnet masks. In other words, it works only if you have the complete network, and it doesn’t work if you’ve registered a couple of IP addresses only with your IP. If you have only one (or very few) IP addresses out of a complete range, you should ask your IP to set up reverse DNS for you.

Configuring DNS When setting up DNS, you have to configure a series of configuration files, and in this section you’ll learn how these relate to each other. At this point, make sure that the DNS server is installed by using apt-get install bind9 as root.

/etc/bind/named.conf The /etc/bind/named.conf file is the master configuration file for your DNS server. Listing 9-4 provides an example. The named.conf file is a master configuration file that contains all you need to get a working DNS set up. To set up your own additional zones, you have to use the /etc/bind/named.conf.local file. Listing 9-4. Default /etc/bind/named.conf File root@RNA:~# cat /etc/bind/named.conf // This is the primary configuration file for the BIND DNS server named. // // Please read /usr/share/doc/bind9/README.Debian.gz for information on the // structure of BIND configuration files in Debian, *BEFORE* you customize // this configuration file. // // If you are just adding zones, please do that in /etc/bind/named.conf.local

271

272

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

include "/etc/bind/named.conf.options"; // prime the server with knowledge of the root servers zone "." { type hint; file "/etc/bind/db.root"; }; // be authoritative for the localhost forward and reverse zones, and for // broadcast zones as per RFC 1912 zone "localhost" { type master; file "/etc/bind/db.local"; }; zone "127.in-addr.arpa" { type master; file "/etc/bind/db.127"; }; zone "0.in-addr.arpa" { type master; file "/etc/bind/db.0"; }; zone "255.in-addr.arpa" { type master; file "/etc/bind/db.255"; };

include "/etc/bind/named.conf.local"; Several other files are called from the main configuration file (/etc/bind/named.conf). Before starting to configure your own DNS server, let’s look at how these files relate to each other: • /etc/bind/named.conf.local: This file contains the DNS zones that you set up on your server. • /etc/bind/named.conf.options: In this file you’d put generic options that define the working of your DNS server. • The db files: These are database files that store the information for specific zones. Every zone has its own db file, so there should be many of these db files on a completely configured DNS server. For example, the following code lines that come from /etc/bind/named.conf refer to the database for the localhost zone:

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

zone "localhost" { type master; file "/etc/bind/db.local"; }; When setting up your own DNS server, it can be quite hard to configure the right files in the right way. So let’s do a setup for the example.com zone.

■Tip If you want to set up a working DNS environment, you should have your own DNS domain, which you can get from your IP. If you don’t have your own DNS domain, you can use the example.com domain. This domain is not used for real on the Internet and can therefore be used by anyone who wants to set up a localonly DNS test environment.

1. Don’t touch the /etc/bind/named.conf file. It contains default settings, and you never need to modify it on Ubuntu Server. 2. Open the /etc/bind/named.conf.local file with an editor and use the following code: zone "example.com" in { allow-transfer { any; }; file "/etc/bind/db.example.com"; type master; }; 3. In this example configuration, the zone "example.com" statement is used as a definition of the zone that you want to use. After the definition of the zone itself and between brackets, specify the options for that zone. In this example, they are as follows: • allow-transfer { any; };: This option specifies what name servers are allowed to synchronize their databases with the information in this database. • file "/etc/bind/db.example.com";: This line indicates what file contains the specific configuration for this zone. • type master;: This option indicates the definition of the master name server for this zone. 4. You’ve now defined the file in which the DNS server can find the specific configuration for example.com. Next, you need to set up reversed name resolution as well. If example.com is using the IP network 201.100.100.0, you should open /etc/bind/ named.conf.local once more and enter the following code: zone "100.100.201.in-addr.arpa" { type master; file "/etc/bind/db.100.100.201"; };

273

274

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

5. Now that you’ve set up the basic structure for DNS, you need to create the database files in /etc/bind that contain the actual configuration of the DNS zones. I’ll explain how to do this and all your available options in the next few sections.

Using named.conf Options Before you create the database files that you refer to in the named.conf file and its related files, let’s have a look at some of the options that you can use in the /etc/bind/named.conf file and its related /etc/bind/named.conf.local and /etc/bind/named.conf.options files. Ubuntu Server uses the /etc/bind/named.conf.options file to include options in the DNS configuration. This file is included with the line include "/etc/bind/named.conf.options"; in /etc/bind/named.conf. Listing 9-5 shows the file as it is by default. Listing 9-5. The /etc/bind/named.conf.options File Contains Generic Options for Your bind Name Server root@RNA:~# cat /etc/bind/named.conf.options options { directory "/var/cache/bind"; // // // // //

If there is a firewall between you and name servers you want to talk to, you might need to uncomment the query-source directive below. Previous versions of BIND always asked questions using port 53, but BIND 8.1 and later use an unprivileged port by default.

// query-source address * port 53; // // // //

If your ISP provided one or more IP addresses for stable name servers, you probably want to use them as forwarders. Uncomment the following block, and insert the addresses replacing the all-0's placeholder.

// forwarders { // 0.0.0.0; // }; auth-nxdomain no; # conform to RFC1035 listen-on-v6 { any; }; // // // // // //

By default, name servers should only perform recursive domain lookups for their direct clients. If recursion is left open to the entire Internet, your name server could be used to perform distributed denial-of-service attacks against other innocent computers. For more information on DDoS recursion: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-0987

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

allow-recursion { localnets; }; // // // //

If you have DNS clients on other subnets outside of your server's "localnets", you can explicitly add their networks without opening up your server to the Internet at large: allow-recursion { localnets; 192.168.0.0/24; };

// If your name server is only listening on 127.0.0.1, consider: // allow-recursion { 127.0.0.1; }; }; As you see in the example file in Listing 9-5, you have quite a few options. In the example file, many options are disabled by default ,and others are just not available. Let’s have a look at some of the more common options: • options { .... };: Use this statement to indicate the start and the end of the section that contains the options. All generic options need to be in this section, so notice the structure used by this statement. It starts with a bracket, it ends with a bracket, and all specific options are defined between the brackets. When putting this in manually, do not forget the semicolon after the last bracket. • directory "/var/cache/bind";: You can use this parameter to define the location in which all DNS configuration files are stored. If an incomplete file name is used anywhere in one of the DNS configuration files, the DNS name server looks for it in this directory. If, however, an absolute file name (a file with a complete directory reference) is used, it just follows the absolute file name. Also note the semicolon at the end of the line; this is an important syntax feature. • notify no;: This option indicates that slave servers should not be notified of changes, which leaves it completely to the slave server to make sure that it is up to date. If you want an alert to be sent to a slave server when a change occurs, change this setting to notify yes;. • forwarders;: By default, if your DNS server gets a request to resolve a name for which it is not responsible, it starts querying a root server of the DNS hierarchy to find the required information. You can change this behavior by using a forwarder, which is another DNS name server that typically has a large cache that it uses to resolve names very quickly. You could, for example, use your IP’s DNS name server as a DNS forwarder.

Zone Definition in /etc/bind/named.conf.local Among the most important DNS server options is the definition of zones. As you can see in the example in Listing 9-4, the first zone that is defined is the zone “.”. This refers to the root of the DNS domain. The definition is required to hook up your DNS server to the rest of the DNS hierarchy. To do this, the zone definition in /etc/bind/named.conf indicates that a list of name servers for the root domain can be found in the db.root file. Listing 9-6 is a portion of the contents of that file.

275

276

CHAPTER 9 ■ CONFIGURING NETWORK INFRASTRUCTURE SERVICES

Listing 9-6. The db.root File Makes Sure That Your DNS Server Can Contact Other Servers in the DNS Hierarchy root@RNA:~# cat /etc/bind/db.root ; DiG 9.2.3 ns . @a.root-servers.net. ;; global options: printcmd ;; Got answer: ;; ->>HEADER