IBM Tivoli Monitoring Version 5.1 : Advanced Resource Monitoring 9780738426938

Citation preview

Front cover

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring Best practices built into ready-to-use resource models Migration from Tivoli Distributed Monitoring (Classic Edition) TEC and TBSM integration

Stephen Hochstetler Ghufran Shah Jamie Carl Jason Shamroski John Willis Murilo Goncalves Aguiar

ibm.com/redbooks

International Technical Support Organization IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring October 2002

SG24-5519-02

Note: Before using this information and the product it supports, read the information in “Notices” on page xix.

Third Edition (October 2002) This edition applies to Version 5 Release 1 of IBM Tivoli Monitoring (product number 5698-EMN). Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. JN9B Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright International Business Machines Corporation 2000-2002. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi The team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Comments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv October 2002, Third Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv Part 1. Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Integration with Common Information Model. . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 Resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 New features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.5.1 Mdist2 support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.5.2 Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.5.3 Migrating from Tivoli Distributed Monitoring (Classic Edition) . . . . . . 10 1.5.4 Migrating from Tivoli Web Component Manager. . . . . . . . . . . . . . . . 10 1.5.5 Tivoli Enterprise Data Warehouse Support . . . . . . . . . . . . . . . . . . . . 10 1.5.6 Additional response actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Chapter 2. Planning the implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1 Systems management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Information Technology Infrastructure Library . . . . . . . . . . . . . . . . . 12 2.1.2 Availability Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Systems monitoring design considerations . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2.1 High-level design considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.2 Solution objectives and requirements gathering . . . . . . . . . . . . . . . . 14 2.2.3 Environment review and gap analysis. . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.4 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

© Copyright IBM Corp. 2002

iii

2.2.5 Limiting factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3 Monitoring systems resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4 Resource model overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.4.1 Resource models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.4.2 Understanding the IBM Tivoli Monitoring 5.1 engine . . . . . . . . . . . . 22 2.4.3 Thresholds and indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.4 Windows resource models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.5 Correlated events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.4.6 UNIX resource models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.5 Installation and configuration planning . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.5.1 Hardware requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.5.2 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.6 Planning for migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.6.1 Think migration, not upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2.6.2 Auditing your current environment . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.6.3 Tivoli Monitoring versions that can coexist . . . . . . . . . . . . . . . . . . . . 58 2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Chapter 3. Firewall considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 3.1 The Tivoli Firewall Security Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.2 Special considerations for firewall environments . . . . . . . . . . . . . . . . . . . 65 3.2.1 Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2.2 Distributing Resource Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2.3 Endpoint-generated TEC events . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Part 2. Installation and migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Chapter 4. Installation and migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.1 Installation and migration guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.1.1 Project environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.1.2 Framework requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.3 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.4 Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.2 Installation and upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.2.1 Fresh Installation of IBM Tivoli Monitoring 5.1 . . . . . . . . . . . . . . . . . 80 4.2.2 Upgrade to IBM Tivoli Monitoring 5.1 . . . . . . . . . . . . . . . . . . . . . . . . 87 4.2.3 Installing prerequisite software on endpoints . . . . . . . . . . . . . . . . . . 98 4.2.4 Installing JRE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.3 Uninstalling the product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.3.1 Uninstalling IBM Tivoli Monitoring 5.1 from endpoints . . . . . . . . . . 102 4.3.2 Uninstalling IBM Tivoli Monitoring 5.1 from the server/gateway . . . 104 4.4 Installing TBSM IBM Tivoli Monitoring 5.1. . . . . . . . . . . . . . . . . . . . . . . . 105 4.5 Installing Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.6 Installing IBM Tivoli Monitoring Historical Data Gathering . . . . . . . . . . . 110

iv

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Chapter 5. Migrating Classic Edition monitors. . . . . . . . . . . . . . . . . . . . . 111 5.1 Migration process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.1.1 Migration overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.1.2 Custom monitors overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.1.3 Sentry profile analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.2 Classic Edition monitor conversion example . . . . . . . . . . . . . . . . . . . . . . 119 5.3 Custom numeric monitor conversion example . . . . . . . . . . . . . . . . . . . . 137 5.4 Converting a CSL-based monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Chapter 6. Migrating from Tivoli Web Component Manager . . . . . . . . . . 161 6.1 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Part 3. Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Chapter 7. Heartbeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 7.1 Heartbeat overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 7.1.1 Endpoint registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 7.1.2 Heartbeat monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 7.1.3 Viewing the endpoint cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 7.2 Installing and configuring heartbeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 7.3 TEC event classes and rule sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 7.3.1 HeartBeat_Off . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 7.3.2 HeartBeat_EndpointUnreachable . . . . . . . . . . . . . . . . . . . . . . . . . . 175 7.3.3 HeartBeat_DMAgentDown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 7.3.4 HeartBeat_DMAgentAlive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 7.3.5 HeartBeat_ResourceModelsInError . . . . . . . . . . . . . . . . . . . . . . . . 176 7.3.6 HeartBeat_DMAgentRestarted . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 7.3.7 HeartBeat_EndpointMigrated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 7.3.8 Heartbeat-related TEC Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 Chapter 8. Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 8.1 Overview of the Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 8.2 Web Health Console components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 8.3 Defining health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 8.4 Using the Web Health Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 8.4.1 Logging on to the Web Health Console . . . . . . . . . . . . . . . . . . . . . 185 8.4.2 The Web Health Console main view . . . . . . . . . . . . . . . . . . . . . . . . 191 8.4.3 The Endpoint List View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 8.4.4 The Resource Model List View . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1 9.1 Configuring IBM Tivoli Monitoring Version 5.1 . . . . . . . . . . . . . . . . . . . 9.1.1 Creating and customizing profiles . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.2 Profile distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

Contents

205 206 206 220

v

9.2 Maintaining your environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 9.2.1 IBM Tivoli Monitoring Version 5.1 command line . . . . . . . . . . . . . . 230 9.2.2 Heartbeat maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 9.2.3 Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 9.2.4 Logs and files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 9.3 New classes in the Tivoli database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 9.3.1 Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 9.3.2 Distinguished. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 9.3.3 Name registry resource types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Chapter 10. Workbench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 10.1 New features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 10.1.1 Compatibility mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 10.1.2 Custom scripting support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 10.1.3 New response actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 10.1.4 Customizable messages for TEC and TBSM events . . . . . . . . . . 247 10.1.5 Wizard for UNIX resource models. . . . . . . . . . . . . . . . . . . . . . . . . 248 10.1.6 New service object APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 10.1.7 New CLI support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 10.2 Using the Workbench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 10.2.1 The Workbench panes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 10.2.2 Elements of a resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 10.2.3 Looking at the PhysicalDiskModel . . . . . . . . . . . . . . . . . . . . . . . . 257 10.3 Resource model examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 10.3.1 ITSO_PhysicalDiskModel example . . . . . . . . . . . . . . . . . . . . . . . . 282 10.3.2 ITSO_LogicalDiskModel example . . . . . . . . . . . . . . . . . . . . . . . . . 301 10.3.3 ITSO_LoadAvg example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 10.3.4 ITSO_FileSystem Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 10.4 Tools and extra information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 10.4.1 WorkBench Command Line Interface . . . . . . . . . . . . . . . . . . . . . . 376 10.4.2 Resource model internal format . . . . . . . . . . . . . . . . . . . . . . . . . . 377 10.4.3 Microsoft tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 10.4.4 Saxsoft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 10.4.5 Rhino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 Part 4. Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 Chapter 11. TEC integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 11.1 Enabling IBM Tivoli Monitoring Version 5.1 to forward events to TEC384 11.1.1 Configuring a profile to send events to TEC . . . . . . . . . . . . . . . . . 384 11.1.2 Configuring the heartbeat function to send events to TEC . . . . . . 389 11.2 Configure TEC to receive events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.2.1 Using the Tivoli-provided script . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.2.2 Manual update of TEC rule base . . . . . . . . . . . . . . . . . . . . . . . . . 393

vi

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

11.2.3 Viewing IBM Tivoli Monitoring Version 5.1 events in TEC . . . . . . 394 11.2.4 IBM Tivoli Monitoring Version 5.1 event aggregation and TEC . . 394 11.2.5 IBM Tivoli Monitoring Version 5.1 event classes. . . . . . . . . . . . . . 395 11.2.6 TMW_Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.2.7 Clearing events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 11.2.8 The supplied rule sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 11.2.9 Event definitions - Baroc files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 11.2.10 Event caching on endpoints and gateways . . . . . . . . . . . . . . . . . 404 Chapter 12. Tivoli Business Systems Manager integration . . . . . . . . . . 407 12.1 Overview of Tivoli Business Systems Manager . . . . . . . . . . . . . . . . . . 408 12.2 Tivoli Business Systems Manager integration. . . . . . . . . . . . . . . . . . . . 408 12.3 Install/configure IBM Tivoli Monitoring Version 5.1 TBSM Adapter . . . . 409 12.3.1 JRE Version 1.3.0 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 12.3.2 IBM Tivoli Monitoring 5.1 TBSM Adapter installation . . . . . . . . . . 411 12.3.3 IBM Tivoli Monitoring 5.1 TBSM Adapter configuration . . . . . . . . 414 12.4 IBM Tivoli Monitoring 5.1 TBSM Adapter . . . . . . . . . . . . . . . . . . . . . . . 415 12.4.1 Endpoint discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 12.4.2 Event forwarding to Tivoli Business Systems Manager . . . . . . . . 422 Chapter 13. Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 13.1 Introduction to Tivoli Enterprise Data Warehouse . . . . . . . . . . . . . . . . . 426 13.2 IBM Tivoli Monitoring Version 5.1 data flow . . . . . . . . . . . . . . . . . . . . . 428 13.3 Installing IBM Tivoli Monitoring Version 5.1 PAC . . . . . . . . . . . . . . . . . 429 13.3.1 Installing IBM Tivoli Monitoring Version 5.1 PAC in TEDW . . . . . 429 13.3.2 Installing IBM Tivoli Monitoring Version 5.1 Data Gathering . . . . . 435 13.3.3 Explanation of installed tasks/jobs . . . . . . . . . . . . . . . . . . . . . . . . 440 13.4 Configuring TEDW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 13.5 Creating and configuring the data mart . . . . . . . . . . . . . . . . . . . . . . . . . 462 13.6 Creating a report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468 Part 5. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Chapter 14. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 14.1 Logs and traces format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482 14.2 TMR Server logs and traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 14.2.1 Profile distribution process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 14.3 Gateway logs and traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 14.3.1 Heartbeat engine traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 14.3.2 Task engine trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486 14.3.3 Tivoli Business Systems Manager engine traces . . . . . . . . . . . . . 486 14.3.4 Tivoli Business Systems Manager adapter trace . . . . . . . . . . . . . 487 14.3.5 Tivoli Business Systems Manager transport trace . . . . . . . . . . . . 487 14.3.6 Endpoint upcall traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488

Contents

vii

14.3.7 Request manager trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488 14.3.8 Profile core trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489 14.4 Endpoint logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 14.4.1 Common endpoint logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 14.4.2 Windows endpoint logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 14.4.3 Non-Windows endpoints logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 14.4.4 Web Health Console logs and traces . . . . . . . . . . . . . . . . . . . . . . 496 14.5 Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 14.5.1 Tool to generate XML file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 14.5.2 Autotrace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 14.5.3 Serviceability tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 14.6 Known problems and their resolutions . . . . . . . . . . . . . . . . . . . . . . . . . 503 14.7 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503 Appendix A. Additional material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 Locating the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 Using the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530 How to use the Web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530 Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 Related publications . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other resources . . . . . . . . . . . . . . . . . . . . . . . . Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . IBM Redbooks collections . . . . . . . . . . . . . . . . .

...... ...... ...... ...... ...... ......

....... ....... ....... ....... ....... .......

...... ...... ...... ...... ...... ......

. . . . . .

533 533 533 534 535 535

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537

viii

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figures 1-1 1-2 2-1 2-2 2-3 3-1 3-2 3-3 3-4 3-5 4-1 4-2 4-3 4-4 4-5 4-6 4-7 4-8 4-9 4-10 4-11 4-12 4-13 4-14 4-15 4-16 4-17 4-18 4-19 5-1 5-2 5-3 5-4 5-5 5-6 5-7 5-8 5-9

© Copyright IBM Corp. 2002

High-level overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 CIM implementation for systems management . . . . . . . . . . . . . . . . . . . . 7 Dynamic versus reference model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Sampling of volatile metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Customized user ID in a classic Distributed Monitoring Profile . . . . . . . 57 Data flow from end user to TMR Server for Web Health Console . . . . . 63 Data flow from Web Application Server to gateway proxy . . . . . . . . . . . 64 Data flow from Relay to Tivoli Endpoint/LCF . . . . . . . . . . . . . . . . . . . . . 65 Properties of a profile configured for TME (Secure) Delivery . . . . . . . . 67 Properties of a profile configured for non-TME (Unsecure) Delivery . . . 68 Lab environment tividc11-region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Products to install. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 IBM Tivoli Monitoring 5.1 policy region . . . . . . . . . . . . . . . . . . . . . . . . . 84 Profile manager and task library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Tivoli Distributed Monitoring (Advanced Edition) Tasks window . . . . . . 85 IBM Tivoli Monitoring Utility Tasks window . . . . . . . . . . . . . . . . . . . . . . 86 Extracting dependencies from resource model DMXCPU . . . . . . . . . . . 93 Removing dependencies from DMXUsrCpu . . . . . . . . . . . . . . . . . . . . . 94 Adding dependencies in DMXUsrCpu . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Building the DMXUsrCpu with the updated dependencies . . . . . . . . . . 96 Adding the custom resource model to the profile. . . . . . . . . . . . . . . . . . 98 Task options panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 DMLinkJre configure task arguments . . . . . . . . . . . . . . . . . . . . . . . . . 102 Initial Web Health Console installation. . . . . . . . . . . . . . . . . . . . . . . . . 107 Web Health Console license agreement . . . . . . . . . . . . . . . . . . . . . . . 108 Directory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 User and password settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Web Health Console products to be installed . . . . . . . . . . . . . . . . . . . 109 Web Health Console final installation . . . . . . . . . . . . . . . . . . . . . . . . . 110 Migration from Tivoli Distributed Monitoring (Classic Edition) 3.7 . . . . 113 Unix_Sentry numeric script monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 MCSL-created monitor collection and monitor . . . . . . . . . . . . . . . . . . . 117 MCSL -c example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Example mcsl -p command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Unix_Sentry/diskioratek MCSl output . . . . . . . . . . . . . . . . . . . . . . . . . 122 Example owgetmon.pl output. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Editing the monitors_rm_table.txt file to find a monitor . . . . . . . . . . . . 124 Converting MCSL-based monitor collections . . . . . . . . . . . . . . . . . . . . 125

ix

5-10 5-11 5-12 5-13 5-14 5-15 5-16 5-17 5-18 5-19 5-20 5-21 5-22 5-23 5-24 5-25 5-26 5-27 5-28 5-29 5-30 5-31 5-32 5-33 5-34 5-35 7-1 7-2 7-3 7-4 8-1 8-2 8-3 8-4 8-5 8-6 8-7 8-8 8-9 8-10 8-11 8-12 8-13

x

Selecting the Dumped Monitoring Collection option . . . . . . . . . . . . . . 126 Listing all of the monitors in the Unix_Sentry monitor collection . . . . . 127 Selecting the Monitor Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Selecting event and threshold elements . . . . . . . . . . . . . . . . . . . . . . . 129 Selecting logging elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Wizard-generated resource model for diskioratek after modifications . 130 General Settings window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Diskioratek Source Details window . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Creating custom script resource models . . . . . . . . . . . . . . . . . . . . . . . 139 Import the Custom Script window . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Setting the event and threshold elements . . . . . . . . . . . . . . . . . . . . . . 141 Setting the logging elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 ITSO_Processor.sh resource model . . . . . . . . . . . . . . . . . . . . . . . . . . 142 ITSO_Processor.sh General Settings window . . . . . . . . . . . . . . . . . . . 143 MCSL-generated monitor collection. . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Converting CSL monitors using the resource model wizard . . . . . . . . 150 Importing the processor.csl file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Preprocessor settings window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Select the Monitoring Sources window . . . . . . . . . . . . . . . . . . . . . . . . 152 Processor monitor arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Specify the Event Triggering Conditions window . . . . . . . . . . . . . . . . . 154 Selecting log elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 ITSO_Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 ITSO_Processor General Settings window . . . . . . . . . . . . . . . . . . . . . 156 Monitor Source Details window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 The extracted tar file - Probe file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Heartbeat endpoint registration data flow . . . . . . . . . . . . . . . . . . . . . . 169 Heartbeat monitoring data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 Endpoint cache retrieval data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Heartbeat configuration and control data flow . . . . . . . . . . . . . . . . . . . 173 High-level data flow of the Web Health Console . . . . . . . . . . . . . . . . . 181 Web Health Console screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 Web Health Console login panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 Web Health Console Preferences -> Endpoint view . . . . . . . . . . . . . . 187 Tivoli Administrator with no user Tivoli role . . . . . . . . . . . . . . . . . . . . . 189 Web Health Console Preferences General view. . . . . . . . . . . . . . . . . 190 Web Health Console Preferences Chart view . . . . . . . . . . . . . . . . . . . 191 Web Health Console common menu bar . . . . . . . . . . . . . . . . . . . . . . . 192 Endpoint List View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Icons showing the Endpoint Health . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Web Health Console showing that the engine is unreachable. . . . . . . 194 Web Health Console and the endpoint resource model information . . 195 Starting or stopping the IBM Tivoli Monitoring 5.1 engine . . . . . . . . . . 196

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

8-14 8-15 8-16 8-17 8-18 8-19 8-20 9-1 9-2 9-3 9-4 9-5 9-6 9-7 9-8 9-9 9-10 9-11 9-12 9-13 9-14 10-1 10-2 10-3 10-4 10-5 10-6 10-7 10-8 10-9 10-10 10-11 10-12 10-13 10-14 10-15 10-16 10-17 10-18 10-19 10-20 10-21 10-22

Resource Model List View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Managing the resource model on an endpoint. . . . . . . . . . . . . . . . . . . 198 Viewing online data for an indication . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Online data for indications generated by a resource model. . . . . . . . . 200 Historical data options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Sample historical data graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Available graph layouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Setting managed resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Creating profile manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Profile creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Empty Tmw2kProfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Add/configure resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Threshold and cycle time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 Configuring indications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Task dm_mn_send_email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Task dm_mn_send_notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Scheduler customization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 Data logging panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Changing parameters of resource model. . . . . . . . . . . . . . . . . . . . . . . 219 Profile properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Profile distribution options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Opening the IBM Tivoli Monitoring Workbench . . . . . . . . . . . . . . . . . . 249 Opening the browser in IBM Tivoli Monitoring . . . . . . . . . . . . . . . . . . . 249 Navigating to the samples directory . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Browser listing of the Windows sample (dmws) files . . . . . . . . . . . . . . 251 PhysicalDiskModel opened in Workbench . . . . . . . . . . . . . . . . . . . . . . 252 Decision tree script logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 TMW_PhsyicalDiskModel general settings . . . . . . . . . . . . . . . . . . . . . 257 Example of descriptive name in a profile . . . . . . . . . . . . . . . . . . . . . . . 258 Category profile pull-down option . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 PhysicalDiskModel General Settings . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Opening the CIM browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 PhysicalDisk resource class definitions . . . . . . . . . . . . . . . . . . . . . . . . 262 Using WMI WQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 Profile indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 TMW_SlowPhysicalDrive event element settings . . . . . . . . . . . . . . . . 267 PhysicalDiskModel thresholds in a profile . . . . . . . . . . . . . . . . . . . . . . 270 PhysicalDiskModel HighPercetnUsage threshold profile display . . . . . 271 PhysicalDiskModel logging element profile display . . . . . . . . . . . . . . . 273 PhysicalDiskModel percent disk usage element profile display . . . . . . 274 PhysicalDiskModel Web Health Console display. . . . . . . . . . . . . . . . . 275 PhysicalDiskModel Web Health Console graphic display . . . . . . . . . . 276 Perfmon GUI display of PercentDiskTime (not capped) . . . . . . . . . . . 282

Figures

xi

10-23 10-24 10-25 10-26 10-27 10-28 10-29 10-30 10-31 10-32 10-33 10-34 10-35 10-36 10-37 10-38 10-39 10-40 10-41 10-42 10-43 10-44 10-45 10-46 10-47 10-48 10-49 10-50 10-51 10-52 10-53 10-54 10-55 10-56 10-57 10-58 10-59 10-60 10-61 10-62 10-63 10-64 10-65

xii

Cloning TMW_PhysicalDiskModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 ITSO_PhysicalDiskModel general settings . . . . . . . . . . . . . . . . . . . . . 284 ITSO resource models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Renaming the event elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Recalculating PercentDiskTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Changing the HighPercentUsage threshold element . . . . . . . . . . . . . . 288 Starting the debugger for Visual Basic based resource models . . . . . 289 Running the Debugger in the Workbench . . . . . . . . . . . . . . . . . . . . . . 290 Adding variables to the Watch window . . . . . . . . . . . . . . . . . . . . . . . . 291 Watch window error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Using the Step to Cursor option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Watch window example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Watch window example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 Indication Sent from the Debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Build menu items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 ITSO_PhysicalDiskModel HTML documentation . . . . . . . . . . . . . . . . . 299 wdmlseng command for the ITSO_PhysicalDiskModel . . . . . . . . . . . . 300 ITSO_PhysicalDiskModel TEC event . . . . . . . . . . . . . . . . . . . . . . . . . 301 CIM Studio MOF Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Compiling a MOF from the command line . . . . . . . . . . . . . . . . . . . . . . 305 Using the MOF Compiler from the Workbench . . . . . . . . . . . . . . . . . . 306 Resource model wizard languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 Resource model wizard methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 Resource model wizard data source type . . . . . . . . . . . . . . . . . . . . . . 309 Resource model wizard - Finding ITSO_LogicalDisk resource class. . 311 Resource Model Wizard: Select a Class window. . . . . . . . . . . . . . . . . 312 Resource model wizard - Defining properties . . . . . . . . . . . . . . . . . . . 313 Resource Model Wizard: Filtering window . . . . . . . . . . . . . . . . . . . . . . 314 Resource model wizard event elements . . . . . . . . . . . . . . . . . . . . . . . 315 Resource model wizard logging elements . . . . . . . . . . . . . . . . . . . . . . 316 Resource model wizard default cycle time. . . . . . . . . . . . . . . . . . . . . . 317 Resource model wizard generated model . . . . . . . . . . . . . . . . . . . . . . 318 ITSO_LogicalDisk General Settings window . . . . . . . . . . . . . . . . . . . . 319 Changing the event elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 Changing the threshold elements in ITSO_LogicalDisk. . . . . . . . . . . . 322 Changing the logging element in ITSO_LogicalDisk . . . . . . . . . . . . . . 323 Adding a dependency element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 Testing the ITSO_LogicalDisk resource model . . . . . . . . . . . . . . . . . . 328 Extracting Resource Class MOF from UNIX/Linux resource model. . . 330 Resource model creation wizards . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 ITSO_LoadAvg General Settings window . . . . . . . . . . . . . . . . . . . . . . 333 Adding dynamic model elements for ITSO_LoadAvg . . . . . . . . . . . . . 334 Compiling a MOF from the Workbench CIM browser . . . . . . . . . . . . . 335

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

10-66 10-67 10-68 10-69 10-70 10-71 10-72 10-73 10-74 10-75 10-76 10-77 10-78 10-79 10-80 10-81 10-82 10-83 10-84 10-85 10-86 10-87 10-88 10-89 10-90 10-91 10-92 10-93 10-94 10-95 11-1 11-2 11-3 11-4 11-5 11-6 11-7 11-8 12-1 12-2 12-3 12-4 12-5

Compiling a MOF from the Workbench . . . . . . . . . . . . . . . . . . . . . . . . 336 Selecting the DMXCpu Resource Class . . . . . . . . . . . . . . . . . . . . . . . 337 Creating event elements for ITSO_LoadAvg . . . . . . . . . . . . . . . . . . . . 339 Creating threshold elements for ITSO_LoadAvg . . . . . . . . . . . . . . . . . 340 Creating logging elements for ITSO_LoadAvg . . . . . . . . . . . . . . . . . . 341 Extracting dependency elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 LoadAvg development directory structure . . . . . . . . . . . . . . . . . . . . . . 343 Adding the DMXCpu.mof file as a dependency . . . . . . . . . . . . . . . . . . 344 Adding the DMXCpu.tar file as a dependency . . . . . . . . . . . . . . . . . . . 345 ITSO_LoadAvg step-by-step wizard-generated resource model . . . . . 346 Invoking the Mozilla Rhino JavaScript Debugger . . . . . . . . . . . . . . . . 350 Syntax checking a JavaScript with the Rhino script debugger . . . . . . 351 trace_dmxengine.log debugging example . . . . . . . . . . . . . . . . . . . . . . 353 Compiling the DMXFileSystem.mof . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 ITSO_FileSystem selecting data source types . . . . . . . . . . . . . . . . . . 357 ITSO_FileSystem selecting the resource class . . . . . . . . . . . . . . . . . . 358 ITSO_FileSystem Selected Properties window . . . . . . . . . . . . . . . . . . 359 ITSO_FileSystem specifying event triggering conditions . . . . . . . . . . . 360 ITSO_FileSystem wizard-generated resource model . . . . . . . . . . . . . 361 ITSO_FileSystem modifying the general settings . . . . . . . . . . . . . . . . 362 ITSO_FileSystem modifying the event element . . . . . . . . . . . . . . . . . . 363 ITSO_FileSystem response actions. . . . . . . . . . . . . . . . . . . . . . . . . . . 365 ITSO_FileSystem variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 ITSO_FileSystem modifying the threshold element . . . . . . . . . . . . . . . 367 ITSO_FileSystem modifying the logging element . . . . . . . . . . . . . . . . 368 ITSO_FileSystem adding parameter elements . . . . . . . . . . . . . . . . . . 369 IBM Tivoli Monitoring 5.1 Profile parameters . . . . . . . . . . . . . . . . . . . . 370 ITSO_FileSystem resource model elements . . . . . . . . . . . . . . . . . . . . 371 Browsing the ITSO_FileSystem.tar file . . . . . . . . . . . . . . . . . . . . . . . . 378 Browsing the ITSO_FileSystem zip file . . . . . . . . . . . . . . . . . . . . . . . . 378 Profile with a resource model added . . . . . . . . . . . . . . . . . . . . . . . . . . 385 Profile Properties window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Indications and Actions panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 TEC Console with events received from a resource model . . . . . . . . . 389 Heartbeat events for the engine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 Heartbeat events for the endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 ProcessKilledOrNotExisting event . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 Clearing event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 JRE Install Product panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 IBM Tivoli Monitoring Version 5.1 TBSM Adapter Install Product . . . . 412 Tivoli Distributed Monitoring 4.1 TBSM Adapter JRE install path . . . . 413 Data flow for bulk/delta discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 Profile settings - Send events to TBSM . . . . . . . . . . . . . . . . . . . . . . . . 418

Figures

xiii

12-6 12-7 12-8 12-9 13-1 13-2 13-3 13-4 13-5 13-6 13-7 13-8 13-9 13-10 13-11 13-12 13-13 13-14 13-15 13-16 13-17 13-18 13-19 13-20 13-21 13-22 13-23 13-24 13-25 13-26 13-27 13-28 13-29 13-30 13-31 13-32 13-33 13-34 13-35 13-36 13-37 13-38 13-39

xiv

Profile settings - Indications to send TBSM events . . . . . . . . . . . . . . . 419 Tivoli Business Systems Manager All Resources view . . . . . . . . . . . . 421 Endpoints discovery in Tivoli Business Systems Manager . . . . . . . . . 422 Properties of Tivoli Distributed Monitoring 4.1 resource . . . . . . . . . . . 423 Tivoli Data Warehouse data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 IBM Tivoli Monitoring Version 5.1 data flow to TEDW . . . . . . . . . . . . . 428 Install welcome screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430 Install type dialog (application only) . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 Confirmation of local host name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 Specifying DB2 user name and password . . . . . . . . . . . . . . . . . . . . . . 432 Set directory name for IBM Tivoli Monitoring Version 5.1 PAC . . . . . . 432 Install additional packages dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Confirm installation options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Installing application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 Installation complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 Installing Data Gathering Component on TMR Server . . . . . . . . . . . . 436 RIM options for Data Gathering installation . . . . . . . . . . . . . . . . . . . . . 437 PolicyRegion SPR_Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 PolicyRegion tividc11_SPR_Region . . . . . . . . . . . . . . . . . . . . . . . . . . 439 TaskLibrary SPR_TaskLib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 Scheduled jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 Tmw2kProfiles inside SPR_ProfileMgr . . . . . . . . . . . . . . . . . . . . . . . . 445 Tmw2kProfile SPR_NtProfile settings . . . . . . . . . . . . . . . . . . . . . . . . . 446 Tmw2kProfile SPR_NtProfile parameters . . . . . . . . . . . . . . . . . . . . . . 447 Tmw2kProfile SPR_UNIXProfile settings. . . . . . . . . . . . . . . . . . . . . . . 448 Tmw2kProfile SPR_UNIXProfile parameters . . . . . . . . . . . . . . . . . . . . 449 Connecting to AMW database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 Setting the user ID and password in the ODBC. . . . . . . . . . . . . . . . . . 450 Creating the user DB2 in DB2 Data Warehouse . . . . . . . . . . . . . . . . . 451 Creating an account in DB2 Data Warehouse Center . . . . . . . . . . . . . 452 Getting the authorities from the database . . . . . . . . . . . . . . . . . . . . . . 453 Permissions granted to db2admin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 Getting to AMW_RIM_Source properties. . . . . . . . . . . . . . . . . . . . . . . 455 Setting AMW_Rim_Source user ID and password . . . . . . . . . . . . . . . 456 Schedule option for AMW_c10_loadDMData_Process . . . . . . . . . . . . 457 Configuring daily schedule for AMW_c10_loadDMData_Process . . . . 458 Promoting schedule to Production mode . . . . . . . . . . . . . . . . . . . . . . . 459 Using DB2 Control Center to view the AMW database schema . . . . . 460 Getting DM_METRICS properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 Setting the right schema for the table DM_METRICS . . . . . . . . . . . . . 461 Data Warehouse main page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 Begin creation of new data mart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 Filling out general information for new data mart . . . . . . . . . . . . . . . . . 464

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

13-40 13-41 13-42 13-43 13-44 13-45 13-46 13-47 13-48 13-49 13-50 13-51 13-52 13-53 13-54 13-55 13-56 13-57 13-58 13-59

Configuring user groups within data mart . . . . . . . . . . . . . . . . . . . . . . 465 Selecting the TWHAdmin user group. . . . . . . . . . . . . . . . . . . . . . . . . . 465 Adding a star schema to the data mart . . . . . . . . . . . . . . . . . . . . . . . . 466 Selecting the AMW star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 Finalizing creation of data mart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 Before refreshing data mart view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 After refreshing data mart view - Creation complete . . . . . . . . . . . . . . 468 Creating a report - Selecting data mart . . . . . . . . . . . . . . . . . . . . . . . . 469 Creating a report - Filling out general information . . . . . . . . . . . . . . . . 470 Creating a report - Adding metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470 Creating a report - Selecting metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 471 Creating a report - Specify aggregations . . . . . . . . . . . . . . . . . . . . . . . 472 Creating a report - Specify group by and order by . . . . . . . . . . . . . . . . 473 Creating a report - Confirming metrics and aggregation . . . . . . . . . . . 473 Creating a report - Specify general time frame . . . . . . . . . . . . . . . . . . 474 Creating a report - Confirmation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 Running the report - Selecting from the Manage Reports view . . . . . . 475 Running the report - Specify general time frame . . . . . . . . . . . . . . . . . 476 Running the report - Specify report name and description . . . . . . . . . 477 Running the report - My First Report completed . . . . . . . . . . . . . . . . . 477

Figures

xv

xvi

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Tables 2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 2-9 2-10 2-11 2-12 2-13 2-14 2-15 2-16 2-17 2-18 2-19 2-20 2-21 2-22 2-23 2-24 2-25 2-26 4-1 4-2 4-3 4-4 4-5 4-6 4-7 4-8 8-1 8-2 8-3 9-1

© Copyright IBM Corp. 2002

ITIL processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Sample server list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Windows resource model selection worksheet . . . . . . . . . . . . . . . . . . . 18 UNIX resource model selection worksheet . . . . . . . . . . . . . . . . . . . . . . 19 Resource model selection worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Memory resource model dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 27 Network Interface Card resource model dependencies . . . . . . . . . . . . . 29 Parametric event log resource model dependencies . . . . . . . . . . . . . . . 31 Parametric Services resource model dependencies . . . . . . . . . . . . . . . 31 Parametric TCP/IP Ports resource model dependencies . . . . . . . . . . . 32 Logical Disk resource model dependencies . . . . . . . . . . . . . . . . . . . . . 32 Physical Disk resource model dependencies Printing resource model . 34 Printing resource model dependencies Process resource model . . . . . 35 Process resource model dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 35 Processor resource model dependencies . . . . . . . . . . . . . . . . . . . . . . . 36 Services resource model dependencies . . . . . . . . . . . . . . . . . . . . . . . . 37 TCP/IP resource model dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 38 Table of correlated indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 CPU resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Memory resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 File resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Process resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Network Interface resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 File system resource model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Security resource model and file monitoring . . . . . . . . . . . . . . . . . . . . . 46 Network RPC/NFS resource model (Sun Solaris specific). . . . . . . . . . . 48 Lab configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Itsodev region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Ghufran region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Jason environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Prerequisite software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Advanced Edition version upgrade path . . . . . . . . . . . . . . . . . . . . . . . . 78 Baroc files compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Resource models compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Defining health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 Health values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Health determination example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 IBM Tivoli Monitoring Version 5.1 CLI . . . . . . . . . . . . . . . . . . . . . . . . . 230

xvii

11-1 13-1 13-2 13-3 13-4 14-1

xviii

IBM Tivoli Monitoring Version 5.1 event classes . . . . . . . . . . . . . . . . . 395 Job DMAE_ReadDataInDB configuration . . . . . . . . . . . . . . . . . . . . . . 440 Job DMAE_NewDataAggregation configuration . . . . . . . . . . . . . . . . . 441 Job DMAE_RollupIntoDB configuration . . . . . . . . . . . . . . . . . . . . . . . . 442 Job DMAE_DataAggregation configuration . . . . . . . . . . . . . . . . . . . . . 443 Gateway trace components summary . . . . . . . . . . . . . . . . . . . . . . . . . 489

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2002

xix

Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX® DB2® IBM® IMS™ OS/390® PAL®

Perform™ Planet Tivoli® Redbooks™ Redbooks(logo)™ RMF™ S/390®

SP™ Tivoli® Tivoli Enterprise Console® Tivoli Enterprise™ Tivoli Management

Environment® Tivoli/Sentry® TME 10™ TME® WebSphere® z/OS™

The following terms are trademarks of International Business Machines Corporation and Lotus Development Corporation in the United States, other countries, or both: Lotus

Word Pro

Notes®

ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. C-bus is a trademark of Corollary, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. Other company, product, and service names may be trademarks or service marks of others.

xx

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Preface IBM Tivoli Monitoring Version 5.1 is a policy based implementation that provides centralized control, reducing the need for a large distributed IT staff. A comprehensive sef of predefined resource monitors, providing default monitoring values for basic system resources, supplies a quick return on investment. IBM Tivoli Monitoring Version 5.1 enables you to reduce the time and effort needed to receive meaningful information regarding a distributed system’s status. This redbook describes the new functions and features of IBM Tivoli Monitoring Version 5.1, as well as how to migrate from previous levels of Tivoli Distributed Monitoring (Advanced Edition) 4.1, Tivoli Distributed Monitoring (Classic Edition) 3.7, Tivoli Distributed Monitoring for Windows 3.7 and Tivoli Manager for NT 3.6.2. This redbook will help Tivoli architects and solution providers migrate custom monitors into resource models used by IBM Tivoli Monitoring Version 5.1.

The team that wrote this redbook This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center. Stephen Hochstetler is a Project Leader at the International Technical Support Organization, Austin Center. He writes extensively and teaches IBM classes worldwide on all areas of Systems Management and Linux Clusters. Before joining the ITSO two years ago, Stephen worked in the Tivoli Services Organization of Tivoli as a Network Management Specialist. He is a certified ITIL Service Manager. Ghufran Shah is a Tivoli Certified Enterprise Consultant and Instructor based in the United Kingdom with os-security.com. He has eight years of experience in Systems Development and Enterprise Systems Management. He holds a degree in Computer Science from the University of Bradford. His areas of expertise include Tivoli Systems Management Architecture, Implementation, and Tivoli Training, together with Business Process Improvement and Return on Investment modeling. He has written extensively on event Management, Monitoring, and Business Systems Management integration and has taught Tivoli courses worldwide.

© Copyright IBM Corp. 2002

xxi

Jamie Carl is a member of Tivoli Software's Global Response Team in Austin, Texas. A two-year veteran of IBM, Jamie was a customer prior to coming to IBM, bringing with him three years of experience with Tivoli doing global deployment as an Architect, Implementer, and Support Person. Since joining IBM, Jamie has made valuable contributions to the company through his knowledge and expertise with Windows NT/2000, including the most recent work he has done with Tivoli Framework and Active Directory environments. Jamie has made several appearances at SHARE, Tivoli User Groups, and Planet Tivoli presenting Distributed Monitoring 3.7/4.1 and various other topics. Jason Shamroski is an Enterprise Systems Management Architect in the United States with over five years of Tivoli and other ESM-related experience. His areas of expertise include ESM Architecture, wireless notification, two-way wireless device integration to the enterprise, and data mobilization. He has written extensively on the use of mobile and wireless devices to manage enterprise IT resources and wireless data synchronization. John Willis is a Tivoli Enterprise Certified Instructor, Consultant, and Lead Architect for Gulf Breeze Software (gulfsoft.com). He has over 20 years in IT with five years of experience working with Tivoli. John is a frequent speaker at SHARE.org on DM,TEC, and the Workbench. His current area of expertise includes working with CIM and WMI. Murilo Goncalves Aguiar is a Advisory IT Specialist working in Tivoli Customer Support at IBM Brazil. He has three years of experience in Tivoli Systems Management with focus on the DM product family. He holds a degree in System Analysis from the Universidade Metodista de Piracicaba, Brazil. His areas of expertise include operational systems, Windows, UNIX, Linux, and networking. Thanks to the following people for their contributions to this project: International Technical Support Organization, Austin Center Budi Darmawan, Vasfi Gucer, Julie Czubik IBM Germany Ingo Averdunk IBM Italy Antonella Fidanza and Arcangelo Di Balsamo IBM USA Randy Hill Dale Ullrich, Theo Winkelmann, Peter LoBrutto, David Smith

xxii

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html

Comments welcome Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: 򐂰 Use the online Contact us review redbook form found at: ibm.com/redbooks

򐂰 Send your comments in an Internet note to: [email protected]

򐂰 Mail your comments to: IBM Corporation, International Technical Support Organization Dept. JN9B Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493

Preface

xxiii

xxiv

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Summary of changes This section describes the technical changes made in this edition of the book and in previous editions. This edition may also include minor corrections and editorial changes that are not identified. Summary of Changes for SG24-5519-02 for IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring as created or updated on October 3, 2002.

October 2002, Third Edition This revision reflects the addition, deletion, or modification of new and changed information described below.

New information 򐂰 Chapter 3, “Firewall considerations” on page 61 is a new chapter. 򐂰 Maintaining IBM Tivoli Monitoring 5.1 recommendations were added in Chapter 9, “Configure and maintain IBM Tivoli Monitoring Version 5.1” on page 205. 򐂰 Chapter 13, “Data Warehouse” on page 425 is a new chapter. 򐂰 Chapter 10, “Workbench” on page 245 is a chapter instead of a second redbook. 򐂰 A new topic was added in Chapter 5, “Migrating Classic Edition monitors” on page 111. 򐂰 A new topic was added in Chapter 6, “Migrating from Tivoli Web Component Manager” on page 161.

Changed information 򐂰 The following was added to Chapter 1, “Introduction” on page 3: – New features 򐂰 The following was added to Chapter 2, “Planning the implementation” on page 11: – Resource model information – Hardware and software requirements – Migration planning

© Copyright IBM Corp. 2002

xxv

򐂰 The installation section of Chapter 3 “Installation and Configuration” and Chapter 7 “Migration Considerations” became Chapter 4, “Installation and migration” on page 71. 򐂰 Additional Heartbeat information was added to section 4.4 and became Chapter 7, “Heartbeat” on page 167. 򐂰 Chapter 4 “Health Console” was updated and changed to Chapter 8, “Web Health Console” on page 179. 򐂰 Chapter 5 “Integration with Tivoli Enterprise Console” was updated and moved to Chapter 11, “TEC integration” on page 383. 򐂰 Chapter 6 “Integration with Tivoli Business Systems Manager” was updated and moved to Chapter 12, “Tivoli Business Systems Manager integration” on page 407. 򐂰 Chapter 8 “Troubleshooting” was updated and moved to Chapter 14, “Troubleshooting” on page 481.

xxvi

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Part 1

Part

1

Planning In this part we introduce IBM Tivoli Monitoring 5.1 and cover topics such as firewalls and other planning considerations.

© Copyright IBM Corp. 2002

1

2

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

1

Chapter 1.

Introduction This chapter gives a brief outline of new features in IBM Tivoli Monitoring 5.1. IBM has attached a new naming scheme to its products, but more than just the name has changed in the case of IBM Tivoli Monitoring 5.1. Additional functionality and integration to new features of Tivoli Framework have been included in this new release as well. If you are new to IBM Tivoli Monitoring 5.1 you will want to read: 򐂰 򐂰 򐂰 򐂰

Section 1.1, “Overview” on page 4 Section 1.2, “Architecture” on page 4 Section 1.3, “Integration with Common Information Model” on page 5 Section 1.4, “Resource model” on page 7

Perhaps one of the most noticeable and interesting changes is the replacement of the Java Health Console with the Web Health Console. The Web Health Console now uses the IBM WebSphere Application Server to provide interaction with the IBM Tivoli Monitoring data on the endpoints. IBM Tivoli Monitoring 5.1 is also the first release of the Distributed Monitoring (Advanced Edition) family of products that includes tools for migrating from a Distributed Monitoring (Classic Edition) product. Other new features are described in 1.5, “New features” on page 8.

© Copyright IBM Corp. 2002

3

1.1 Overview The Tivoli Management Environment framework provides a means to manage distributed resources through centralized control and configuration. IBM Tivoli Monitoring 5.1 is now the backbone for availability monitoring across operating systems and applications and more tightly integrates into the Framework environment. IBM Tivoli Monitoring 5.1 maintains many of the advances of Tivoli Distributed Monitoring (Advanced Edition) 4.1, such as integration with the Common Information Model (CIM) and Windows Management Instrumentation (WMI) to manage event and performance data closer to the source. More intelligent decisions are made based on the resource model approach introduced in Tivoli Distributed Monitoring (Advanced Edition) 4.1. IBM Tivoli Monitoring 5.1 continues the paradigm in resource monitoring by reducing the processing of large amounts of raw data generated from sources at a single central location. Instead, IBM Tivoli Monitoring 5.1 uses Java-based technology to integrate with industry standard information models to correlate event data at the server level with other event data from monitors on the server. Overall this will reduce the number of events at the central display. By performing functions locally at the managed resource that includes local event analysis, event correlation, event aggregation, event management, and event logging, it is able to process the information intelligently to recognize critical situations. IBM Tivoli Monitoring 5.1 continues the approach of building monitors based on resource models, which in turn are based on the CIM, which provides industry standards for vendors and application developers that wish to instrument their software. New features are described in 1.5, “New features” on page 8.

1.2 Architecture The following sections describe the high-level architecture of IBM Tivoli Monitoring 5.1. This is the same content as Tivoli Distributed Monitoring (Advanced Edition) 4.1, as the basic architecture remains the same. Figure 1-1 on page 5 presents a high-level overview of the interaction between various components of IBM Tivoli Monitoring 5.1. The IBM Tivoli Monitoring 5.1 profile contains, among other information, a resource model. The resource model is a collection of monitors that correlate amongst themselves before attempting to perform a notification action. The IBM Tivoli Monitoring 5.1 profile is distributed to the endpoints to monitor one or more resources (examples of typical

4

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

resources are hard disk space, paging space, and process/service). Based on configuration settings in the IBM Tivoli Monitoring 5.1 profile, the engine runs on the endpoint and performs the necessary monitoring on the resources that are specified in the distributed resource model(s). The Web Health Console obtains logged data from selected endpoints and displays the "health" of the endpoints for their resources.

Cu sto

Profile

Trend Analysis

m iz e /D ist

De fa ul

rib

Data Warehouse ut e

Rollup

ts

Display

TMR TMR ITM HeartBeat

tall Ins

Web Health Console

Get Data

Resource Model

Distribute

Design Create Debug

Endpoint Windows NT/2000 ITM Engine

Endpoint Unix/Linux ITM Engine

Workbench Figure 1-1 High-level overview

The following sections describe CIM architecture and the three key components of IBM Tivoli Monitoring 5.1, which are: 򐂰 Profile 򐂰 Resource models 򐂰 Engine

1.3 Integration with Common Information Model The Common Information Model (CIM) is part of the industry-wide initiative, called Distributed Management Task Force (DMTF), for providing a common model for the management of environments across multiple vendor specific products. The Web-based Enterprise Management (WBEM) is another initiative

Chapter 1. Introduction

5

that includes the CIM standard. CIM is a conceptual model for describing implementation-independent management. The ultimate goal of CIM compliance is to build applications that can extract and use management data from products released by different vendors, be it hardware or software. IBM Tivoli Monitoring 5.1 is CIM-compliant and hence can collect, store, and analyze management data from other CIM-compliant base products in a common format (CIM). The definite advantage of using CIM-based resource monitoring is that when newer versions of the product whose resources are already being monitored are released, the monitor is not required to know the implementation details of the system, but only interacts with it through an already accepted management interface (CIM Schema). CIM is basically an object-oriented modeling of both hardware and software elements. CIM consists of two parts: The CIM Specification and the CIM Schema. While the CIM Specification describes the language, naming, meta-schema, and mapping techniques to other management models such as SNMP, MIBs, and so on, the CIM Schema provides the actual model descriptions by giving the set of classes with properties and associations to model the particular environment. The CIM Schema is comprised of the core model and common model. The core model is the platform-independent base model and other models are built as extensions of the core model. It is described by the core schema and contains a top-level set of classes applicable to all management domains. The common model is an extension of the core model. The common model is also platform-independent, but specifies the management area. Presently, there are five areas that are defined in the common model: Systems, devices, applications, networks, and physical. Beyond the CIM Schema are extension schemas. The extension schemas are specific to the technologies that are defined in them. Examples of extension schema include schemas for Microsoft Windows operating systems, various UNIX environments, and so on. Each schema is a collection of one or more classes, and each class is a collection of similar objects. A class is defined as the basic unit of management. Observing the object-oriented paradigm, classes are either defined as static or dynamic. Static classes are stored by the CIM Object Manager in the CIM Object Management (CIMOM) Repository. Applications query object data from the CIMOM Repository through the CIM Object Manager. The basic CIM-compliant instrumentation layer is either provided by the operating systems vendor or a software that is compliant with CIM specifications that is installed on the operating system, to enable management applications (like Tivoli Distributed Monitoring) to retrieve the management data about various resources in the system.

6

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

On Windows 2000 systems, the CIM-compliant Microsoft implementation called Windows Management Instrumentation is already installed as part of the operating system. Windows NT 4.0 servers require WMI software to be installed prior to the use of IBM Tivoli Monitoring; this software is available as a download from Microsoft. Therefore, IBM Tivoli Monitoring 5.1 directly interacts with WMI to pull management data at runtime. In UNIX systems, a CIM-compliant instrumentation called TouchPoint is embedded in the IBM Tivoli Monitoring 5.1 engine to facilitate the retrieval of management data. The TouchPoint infrastructure has two interfaces: Touchpoint Service Layer (TSL) and Resource Interface Instrumentation Library Type (ILT). Figure 1-2 shows a layout of CIM implementation for systems management.

Management Applications

•producing reports of network inventory •displaying system information •responding to events •starting or stopping system services

Database Database Applications Applications

Web Web Browsers Browsers

C/C++/Java C/C++/Java Applications Applications

Language for modeled data Managed Object Format

CIM

MOF compiler

MOF

CIMOM CIMOM APIs APIs / TSL Management Infrastructure

CIMOM CIMOM Repository Repository

CIM CIM Objects Manager Manager (CIMOM (CIMOM))

•Class providers

•Property providers WBEM •Event providers •Methods providers •Event Consumer Providers Providers

Provider Provider // ILTs ILTs

•Instance providers

Managed Objects

•logical enterprise components •physical enterprise components

System System Resources Resources

BackOffice BackOffice Apps Apps

MiddleWare MiddleWare

Figure 1-2 CIM implementation for systems management

For a detailed discussion of CIM and CIM certification, please refer to the DMTF Web site: http://dmtf.org

1.4 Resource model The resource model is a monitoring model that contains the program scheme necessary to determine what data is to be accessed from an endpoint at runtime and how this data is to be handled. In other words, the resource model is equivalent to an implementation of monitors from previous editions of Tivoli

Chapter 1. Introduction

7

Distributed Monitoring, albeit in a different way, using the object-oriented modeling approach and integration with CIM. Each resource model obtains resource data from the endpoint it is distributed to, performs root cause analysis using a built-in algorithm, and reacts accordingly in the form of built-in actions or user-defined tasks. An example of a resource model is the TMW_Services resource model, which is used to monitor the Windows Services like server, browser, event log, and so on. The major components of a resource model are the configuration file, MOF file, decision tree file, message catalog for building the Tivoli Enterprise Console, and Tivoli Business Systems Manager events, and class files to interface with the native library. The MOF file contains the particular resource model's class description and event definitions. The classes are derived from the corresponding base classes of the resources. The base classes for the model-to-resources association, and classes that describe physical resources used in the resource models with respect to the instance providers, are provided in a base MOF file a and resources MOF file, respectively. These additional MOF files are distributed to the endpoint the first time an IBM Tivoli Monitoring 5.1 profile is distributed to it. The configuration details of the resource model are obtained by the engine from the configuration file. These configuration details include resource model name, platform name, name of the decision tree file, and names of the MOF files. The decision tree file (VB script or JavaScript for Windows platform and JavaScript for UNIX/LINUX platforms) contains both the initialization settings for the resource model and the algorithm that is used by the IBM Tivoli Monitoring 5.1 analyzer to determine if a problem is encountered in the resources described by the resource model. When an IBM Tivoli Monitoring 5.1 profile containing a resource model is distributed to the endpoint, the following actions take place: 򐂰 Resource model files are unzipped from the resource model specific zip file. 򐂰 MOF files specific to the resource model are complied. 򐂰 All components of the IBM Tivoli Monitoring 5.1 engine are created, loaded in memory, and started (logger, analyzer, action manager, event aggregator, and event correlator).

1.5 New features Just to briefly touch on the new features, read on. The general architecture and functionality have remained the same. The key features are Mdist2 support, Web Health Console, migration tools from previous versions of Tivoli Distributed Monitoring, Tivoli Enterprise Data Warehouse support, and some additional response actions.

8

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

1.5.1 Mdist2 support The product now uses Mdist2 to distribute profiles to the subscribers. Mdist2 introduced new stability and features when attempting to distribute a profile to an endpoint. IBM Tivoli Monitoring Version 5.1 now takes full advantage of the same Framework features as Inventory 4.0. MDIST2 maintains a history of distributions to your endpoints and provides a graphical representation of the profile distribution statuses. MDIST2 will allow a distribution to be paused and canceled, and will also perform retries based on architecture definitions. While this change is not a direct reflection of the product, it is a nice feature to be able to see the statuses of distributions, have a configurable retry, and maintain a history in a standard database. For more information see 9.1.2, “Profile distribution” on page 220.

1.5.2 Web Health Console The former Health Console has become a Web-based Health Console. The Health Console was a nice start and the Web Health Console is another major step forward. IBM Tivoli Monitoring Version 5.1 now ships with IBM WebSphere Application Server to serve the Health Console pages. Many advantages come from making connections through WebSphere, one of the first thoughts that comes to mind is security. Another is performance, and another is standardization. WebSphere’s HTTP services are provided by the IBM HTTP Web Server which is based on Apache, so customization and integration with enterprise security standards should not be an issue. Once connected you will be able to define the endpoints that you want to view data on. The idea of the Web-based Health Console helps us all out a lot. Used in conjunction with data aggregation, a systems administrator can view their own history without having to go to the Tivoli administrators. This is a time-saver in the long run for most of us. For more information see Chapter 8, “Web Health Console” on page 179.

Chapter 1. Introduction

9

1.5.3 Migrating from Tivoli Distributed Monitoring (Classic Edition) Tools have been developed and are included for migrating monitors from the Tivoli Distributed Monitoring Classic Edition products into IBM Tivoli Monitoring 5.1. Features include a compatibility mode for running Classic Edition type monitors within IBM Tivoli Monitoring 5.1 resource models. For more information see Chapter 5, “Migrating Classic Edition monitors” on page 111.

1.5.4 Migrating from Tivoli Web Component Manager Additionally, a tool has been developed to assist in the migration of Tivoli Web Component Manager monitors into IBM Tivoli Monitoring Version 5.1. While the process is not automated, it is much easier than switching back and forth between profiles. The process for migrating from TWCM to IBM Tivoli Monitoring 5.1 is very well documented in the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569, User’s Guide; however, we will attempt to fill in a few blanks. For more information see Chapter 6, “Migrating from Tivoli Web Component Manager” on page 161.

1.5.5 Tivoli Enterprise Data Warehouse Support The product now uses the Gathering Historical Data component to store data in the Tivoli Enterprise Data Warehouse schema. As Tivoli stated in its earlier product releases, all new products will have the ability to store historical data in the Tivoli Enterprise Data Warehouse. This feature once again focuses on standardization of data repositories and reusing the toolset wherever applicable.

1.5.6 Additional response actions The product is now able to act on an event by sending an e-mail to a specified address, sending a notice to a notice group, and running a program. These notification methods are executed as tasks to achieve the desired notification.

10

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

2

Chapter 2.

Planning the implementation In this chapter, we provide an overview of the steps needed to ensure an effective implementation of IBM Tivoli Monitoring 5.1. As part of our planning approach, we describe the importance of systems management processes as they relate to the design of a systems monitoring solution, such as IBM Tivoli Monitoring 5.1. Some guidelines and recommendations are then offered in regards to configuring and using the product. The following are the main points we cover in this chapter: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

Systems management Systems monitoring design considerations Monitoring systems resources Resource model overview Installation and configuration planning Planning for migration

© Copyright IBM Corp. 2002

11

2.1 Systems management With the demands of the new economy and to satisfy the growing business needs, organizations are becoming increasingly dependent on IT services. This growing dependency also continues to raise expectations and requirements, calling for a closer management and control of IT resources, such as servers, workstations, applications, and Internet services. IT services mainly deals with purchasing, distributing, configuring, maintaining, and tracking problems of IT resources. To provide these services and to measure their effectiveness, a systems management solution, which consists of processes, along with organizational skills and the tools to support them, must be implemented. When developing a systems management solution, it is often useful to adopt a best-practices approach and to utilize a standard framework model. Such a model will simplify the design process and can help overcome common difficulties associated with the growth of IT systems. One representation of the IT best practices is the Information Technology Infrastructure Library (ITIL) standard, described in the following section.

2.1.1 Information Technology Infrastructure Library ITIL consists of books and best practices for managing the information technology infrastructure. Developed by the United Kingdom’s Central Computer and Telecommunications Agency, it has become a widely accepted approach toward service management. ITIL provides guidelines for delivering efficient and effective IT services. These service management guidelines are organized into two main areas, Service Support and Service Delivery. Each of these two areas cover different management disciplines, as described in Table 2-1. Table 2-1 ITIL processes

12

Service Delivery

Service Support

Service Level Management

Configuration Management

Capacity Management

Problem Management

Contingency Planning

Change Management

Availability Management

Help Desk

Cost Management for IT Services

Software Control and Distribution

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Depending on how it is implemented, IBM Tivoli Monitoring 5.1 can support a number of service management processes including availability, capacity, problem, and service level management. However, we will discuss IBM Tivoli Monitoring 5.1 as a tool that would primarily support the Availability Management process, as it also integrates with other systems management functions.

2.1.2 Availability Management Availability Management, as it refers to IT resources, involves proactively monitoring and responding to network and system events. It may also involve reporting management information in the form of availability statistics to service level management. System events are primarily the result of failures or alerts within system components, such as CPUs, memory, disk space, applications, or network connectivity. The ability to monitor such components is vital to providing availability information on systems, applications, and business processes.

2.2 Systems monitoring design considerations Systems monitoring involves the monitoring of networked resources in the enterprise, such as application servers, print servers, database servers, and possibly the client workstations. It is likely that the systems monitoring design within your environment is addressed as part of an overall systems management solution. This solution, as mentioned in the previous section, should define processes, organizational roles, and the enabling technology. In the following sections we describe some of the design considerations when implementing IBM Tivoli Monitoring 5.1 as the enterprise systems monitoring tool.

Chapter 2. Planning the implementation

13

2.2.1 High-level design considerations Implementation planning for IBM Tivoli Monitoring 5.1 would entail answering some basic questions, such as: 򐂰 How will IBM Tivoli Monitoring 5.1 be used in your environment? Systems monitoring can be used solely to alert support personnel about a failed resource. It can also be used to proactively respond to minor events that can potentially cause larger problems. Information gathered by the monitoring tool can also be used for trend analysis, service level reporting, and capacity planning. In short, you need to clearly define which problems you are going to solve, because this will determine how IBM Tivoli Monitoring 5.1 will be configured and how it will integrate with other systems. 򐂰 Which resources should be monitored? This will partly depend on the previous point, which is how you will use IBM Tivoli Monitoring 5.1. Your decision will also be influenced by your organization’s monitoring requirement, as we will discuss in the following sections. 򐂰 How should the resource be monitored? IBM Tivoli Monitoring 5.1 provides some out-of-the-box monitoring capabilities. However, as you gain more experience and become familiar with your environment, you may decide to modify the default monitoring parameters.

2.2.2 Solution objectives and requirements gathering The primary objective of a systems monitoring solution is to meet the pertinent goals of the organization, which may include the following: 򐂰 򐂰 򐂰 򐂰 򐂰

Proactive systems and application monitoring Systems availability reporting Collection of historical performance data for trend analysis Performance management and efficient utilization of IT resources Capacity planning

Supporting such strategic goals will require careful and complete requirements gathering, which may involve various business areas within your organization. Understanding the requirements is the key to providing optimum systems monitoring services.

14

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

2.2.3 Environment review and gap analysis Most systems management solutions are implemented to support an existing infrastructure. Therefore, it is important to have a clear understanding of the current environment and the resources you plan to manage. Any potential gaps must also be identified so they can either be corrected or be considered as the limiting factors. The environment review should include the following areas: 򐂰 Systems management processes: Knowing which processes currently exist is important in that you can determine which processes will have to be added, modified, or eliminated. 򐂰 Organizational issues: You may already have an idea of the currently available skills within your organization. This will be helpful when planning the implementation and future support needs. 򐂰 Tivoli environment: Ensure that your current Tivoli environment is suitable for the new install or upgrade to IBM Tivoli Monitoring 5.1. 򐂰 Network environment: An up-to-date diagram along with other network documentation can provide information regarding bandwidth, traffic flows, and firewall configurations. 򐂰 Systems: Perhaps the most important part of the environment review is to document the number and configuration of the systems that are potentially going to be managed by IBM Tivoli Monitoring 5.1. For each system, the following information should be gathered and documented: – – – – – – – – –

Hardware configuration Platform Processor Memory Hard disks/file systems Network interface Function (that is, file, print, database, and so on) Ownership and support information Availability impact on business

򐂰 Applications: Gather information on the applications used within the environment, including any special monitoring requirements.

2.2.4 Testing Of course, the solution you are going to implement should not interfere with the existing IT processes and initiatives. The impact of the monitoring tools, although very minimal, must be taken into consideration before you install the product in a production environment.

Chapter 2. Planning the implementation

15

It is likely that you already have access to a Tivoli test environment that is similar to your production environment. A detailed systems monitoring test plan, in accordance with your organization’s guidelines and procedures, must be developed and implemented. Your testing effort may cover the following: 򐂰 Concurrence with other system processes and applications 򐂰 Processor load on the endpoints to be managed 򐂰 Impact on the network traffic 򐂰 Integration with other management tools such as Tivoli Business Systems Manager and Tivoli Enterprise Console

2.2.5 Limiting factors There may be limitations within your environment that cannot be solved at this time. You may have to accept these as the limiting factors and design your solution accordingly. They may include the following: 򐂰 Environment limitations, such as the systems management processes and organization issues. 򐂰 Tivoli limitations may include the number of ManagedNodes, endpoints, and the overall architecture. 򐂰 Network issues. Firewalls, security, and bandwidth thresholds may also limit your IBM Tivoli Monitoring 5.1 implementation.

2.3 Monitoring systems resources Hopefully by this time you have answered the basic questions in regards to the systems monitoring solution you are going to implement. You can now begin the task of determining how to monitor the specific systems resources and, more importantly, what to do with the gathered information. We know that IBM Tivoli Monitoring 5.1 has the capability to monitor a large number of systems resources. In most cases the technical ability to manage a certain resource is not the issue. The more important question is which resources do you manage and what actions do you take when a fault is detected. When IBM Tivoli Monitoring 5.1 detects a fault, it can optionally forward an event to Tivoli Enterprise Console or Tivoli Business Systems Manager. In the case of Tivoli Enterprise Console, the event is then either processed by a series of rules that reflect the overall event management process, or they are reactively responded to by the support personnel. We know that there is always some cost associated with processing, and even more so when responding to an event.

16

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

This cost may be defined in terms of time and money, or just taking the attention away from the more important tasks. The cost may be justified for some resources and not for others. Calculating the monitoring cost for every server in your environment can itself prove to be a daunting task. We recommend that you first categorize all your systems based on how critical the availability of that particular system is for your business. Your assessment may be influenced by estimates such as loss of revenue, affected business processes, public perception, number of people effected, and so on. You may decide to have a number of categories or only two (that is, Critical and Not Critical). Then you can organize your servers based on their type or the application they support, such as mail, Web, database, and so on. An example of this is given in Table 2-2. Depending on the size of your environment, you may even skip this step.

MKT_SVR_10

YES

DEV_SVR_11

NO

Application B

YES

Application A

HR_SVR_03

Mail server

YES

Web service

ORA_UNIX

Transaction server

Business critical?

Print server

Server name

File server

Table 2-2 Sample server list

X X X X

Your next task is to determine which resources you are going to monitor. By now, you are familiar with the notion of resources and resource models. Summarize the information you have on your systems and the available resource models in a matrix, similar to Table 2-3 on page 18 or Table 2-4 on page 19. Since there are different resource models for Windows and UNIX, we have created two tables. If you skipped the previous step, Table 2-2, and decide to map the resource models directly to the server names, then your table may look similar to Table 2-5 on page 19.

Chapter 2. Planning the implementation

17

TMW_EventLog TMW_LogicalDisk TMW_Memory TMW_NetworkIntCard TMW_ParamPorts TMW_ParamServices TMW_PhysicalDisk TMW_Print TMW_Process TMW_Processor TMW_Services TMW_TCPIP

18

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Application B

Application A

Mail server

Web service

Transaction server

Print server

Windows NT resource models

File server

Table 2-3 Windows resource model selection worksheet

ORA_UNIX

Y

HR_SVR_03

Y

MKT_SVR_10

N

DEV_SVR_11

N

Chapter 2. Planning the implementation

Windows UNIX

resource models resource models

DMXNetworkRPCNFS (Solaris)

DMXSecurity

DMXProcess

DMXNetworkInterface

DMXMemory

DMXFileSystem

DMXFile

DMXCpu

TMW_TCPIPModel

TMW_ServicesModel

TMW_ProcessorModel

TMW_ProcessModel

TMW_PrintModel

Application B

Application A

Mail server

Web service

Transaction server

Print server

File server

UNIX resource models

TMW_PhysicalDiskModel

TMW_NetworkIntCardModel

TMW_MemoryModel

TMW_LogicalDiskModel

TMW_EventLogModel

Business critical?

Server name

Table 2-4 UNIX resource model selection worksheet

DMXCpu

DMXFile

DMXFileSystem

DMXMemory

DMXNetworkInterface

DMXProcess

DMXSecurity

DMXNetworkRPCNFS (Solaris)

Table 2-5 Resource model selection worksheet

19

When implementing resource monitoring for the first time, it is important to start only with the resource models most critical to your business. IBM Tivoli Monitoring 5.1 comes with a set of monitoring parameters that can be used out-of-the-box. You can always add more monitoring capabilities as you become more familiar with the environment and have tested your event management processes. Once the above tasks are completed, you can use the information to implement resource monitoring on your systems. Basically, for each cell in the resource model selection worksheet, you will have to create a profile for IBM Tivoli Monitoring 5.1. You will then subscribe the machines to the profile according to the worksheet. You will also be able to decide how often IBM Tivoli Monitoring 5.1 is going to check the system, how many incidents have to occur before a notification is sent, and what thresholds are to be considered. To see which resource models have been installed, use the following command: wdmrm -list

You may end up with profiles that have no resource models selected. These should also be distributed in order to prepare the systems for future monitoring. Remember, the IBM Tivoli Monitoring 5.1 endpoint components are installed the first time you distribute a profile to that endpoint. The profile creation procedures are explained in more detail in 9.1.1, “Creating and customizing profiles” on page 206.

2.4 Resource model overview In order to properly utilize the resource model architecture implemented by IBM Tivoli Monitoring 5.1, it is important to have a solid understanding of the components, terminology, and design behind this architecture. The following sections will provide details on what resource models are, the components of the IBM Tivoli Monitoring 5.1 engine, and how all of the pieces fit together.

2.4.1 Resource models The resource model is the key concept of the IBM Tivoli Monitoring 5.1 product. Resource models are used to model a business-relevant object, such as the CPU of a machine or the memory of a machine. Resource models basically have two components, named dynamic model and reference model, as shown in Figure 2-1 on page 21.

20

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The dynamic model is the definition of a set of attributes that specifies the current state of the resource. The attribute values typically will change dynamically while the system is working. IBM Tivoli Monitoring 5.1 leverages on the WMI infrastructure on Windows systems, and the Touchpoint infrastructure on UNIX systems. Touchpoint is Tivoli’s CIM implementation for UNIX, which includes the provider CIMOM and data repositories. The reference model has the responsibility to: 򐂰 Interpret the quality of an object against a defined threshold, as well as to discover the root cause of the quality reduction. 򐂰 Process the properties of the physical objects associated in the business object. 򐂰 Generate the indications.

1- Dynamic Model

2- Reference Model

Range of Contained References

Range of Containing Reference

Metric Analysis

Contained

TMW_Resource

TMW_ResourceModel

1..n

1..n

Containing

TMW_Resource +Containing:TMW_ResourceModel +Contained: TMW_Resouce

Association:Domain of References

Recovery Action

Figure 2-1 Dynamic versus reference model

Chapter 2. Planning the implementation

21

2.4.2 Understanding the IBM Tivoli Monitoring 5.1 engine In order to configure the various components of the IBM Tivoli Monitoring 5.1 profile, it is important to have a basic understanding of the internal workings of the IBM Tivoli Monitoring 5.1 engine and how the different components interact with each other. Understanding the internals will also make it easier to understand the impact of changing the “out-of-the-box” parameters of the IBM Tivoli Monitoring 5.1 profile. We briefly cover the following six components that make up the IBM Tivoli Monitoring 5.1 engine: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

Analyzer Event aggregator Event correlator Action manager Logger Scheduler

For more information regarding these components, see 14.4.2, “Windows endpoint logs” on page 491, which includes detailed logging messages illustrating the conversations between components.

Analyzer The analyzer component receives the resource model from the IBM Tivoli Monitoring 5.1 engine and has the responsibility to run the best practices contained in the reference part of the model. The reference models gets, through the WMI/Touchpoint, the performance data of all physical and logical resource instances. Basically, it collects and analyzes the performance data to verify the Service Level of the endpoint with respect to a determined class of problems. The best practices or knowledge paths are coded in Visual Basic (VB) on Windows and JavaScript on UNIX. JavaScript will also be available on the Windows platform in Version 5.1.1. The VB/JavaScripts are executed during every cycle. If a threshold is exceeded, the analyzer sends an indication to the event aggregator. In a simple case, we would compare one threshold (resource model property) to one resource property (“the real world”) to trigger indications. However, in most cases, to find the root cause of a problem you would have to look at any number of thresholds and compare them to the resource property values. In 2.4.3, “Thresholds and indications” on page 26, we provide cross-referenced tables of all the events that can be generated and the thresholds that make up those events, for both Windows and UNIX resource models that ship with IBM Tivoli Monitoring 5.1.

22

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Event aggregator When a service level is exceeded at a determined time, it does not always represent a problem, but its persistence may represent a real problem. The event aggregator measures the persistence of the indications that are being generated by the analyzer. The event aggregator collects all indications coming from all decision trees currently running, and consolidates them based on the aggregation rules that are configured in the profile. The aggregation rules are controlled by the occurrences and holes that are configured in the IBM Tivoli Monitoring 5.1 profile. An occurrence is the term used to refer to a cycle during which an indication occurs for a given resource model (an “above threshold” cycle). A hole is the term used to refer to a cycle during which an indication does not occur (a “below threshold” cycle). If the persistence of an indication meets the configured number of occurrences, an event is generated and sent to the event correlator. If the profile is configured to send events to Tivoli Enterprise Console for a particular indication, then the event aggregator is responsible for sending the event to Tivoli Enterprise Console. However, all events generated by the event aggregator are sent to the event correlator, irrespective of whether it is configured to send events to Tivoli Enterprise Console. Assume that there is a resource property that changes its value rapidly. The decision tree would be visited every cycle, and as part of this, the value of the resource property is retrieved. In Figure 2-2 on page 24, the vertical dashed lines represent the moments of the queries, and the point at which the dotted lines meet the graph are values that are the results of the inquiries. The one horizontal dashed line represents the threshold. Values above that line are considered potential problems and will trigger an indication. Every time the values of the resource properties exceed the thresholds, the resource model will generate an indication. If the value of the resource property drops below the threshold for a particular cycle then no indication is generated and the event aggregator counts the non-occurrence of an indication as a hole. So, if we define an event as four occurrences and one hole when we configure our profile, then based on the indications generated in Figure 2-2 on page 24, the event will be generated when the fourth indication occurs. If we had two consecutive holes, then the occurrence count would be reset to zero and a clearing event would be sent if it was configured to send a clearing event. Tip: The concept of holes can be a bit confusing if you do not realize that the number of holes specified in your profile represent the acceptable number of consecutive holes, not the accumulated number of holes over a sliding window of time.

Chapter 2. Planning the implementation

23

In dica tion # 4

metric values

Indication #3 In dica tion # 1

Ind ication #2

.

threshold

.

.

. 2 consecutive holes

}

C learing eve nt

tim e

C ycle Time Ma x # o f hole s is 1 and they are not co nse cutive

Figure 2-2 Sampling of volatile metric

Event correlator The event correlator is responsible for consolidating and correlating the events that are generated by the event aggregator between different resource models. It receives all events from the event aggregator irrespective of whether the event aggregator has been configured to send events to Tivoli Enterprise Console. It uses a set of static rules to correlate between the different resource models. This static correlation between resource models is only available for the Windows resource models.

Action manager This component is responsible for executing corrective actions (tasks and built-in actions) when a resource model detects a problem. It executes the actions that the user has associated to a particular indication. There are a number of ways that the action manager executes the actions. Built-in actions are implemented as methods of resource models and they are executed through the WMI. Tivoli tasks are the classic Tivoli tasks that the user can associate with a particular indication through the IBM Tivoli Monitoring 5.1 profile GUI. The action manager also generates an ActionResult or a TaskResult indication that is received by the Tivoli Enterprise Console adapter and then forwarded to the Tivoli Enterprise Console server.

24

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Logger The logger is responsible for collecting performance data on the endpoint and storing the data locally. The logger handles multiple resource models that are distributed to an endpoint. It also handles the way data is collected, that is, aggregated data (minimum, maximum, or average values) or raw data. The logger is also responsible for clearing the oldest records in the database. Nightly, around midnight, a data purging process is executed, which removes all data over 24 hours old. You can only view 24 hours of data; however, there are times when this database has almost 48 hours of data (just prior to midnight’s purging process). By default, logging is turned off. The data can also be consolidated to a central RDBMS for trending and reporting; see 13.1, “Introduction to Tivoli Enterprise Data Warehouse” on page 426, for more details. On Windows NT/2000/XP systems, this information is stored in the format of a Microsoft Access database. The OS components necessary for reading/writing to this database are included in Windows 2000/XP, but must be separately downloaded and installed on Windows NT (see 2.5.2, “Software requirements” on page 50). On the UNIX platforms, IBM Tivoli Monitoring 5.1 is utilizing an open-source database from Quadcap Software. Quadcap is a pure Java RDBMS designed to be embedded into applications. For more information and documentation about the Quadcap database, visit their Web site at: http://www.quadcap.com/home.html

Scheduler IBM Tivoli Monitoring 5.1 contains a scheduling feature that allows you to determine a period within which monitoring should take place, and specific scheduling rules. The monitoring period is determined by defining a from and a to date. The scheduling rules allow you to define time periods on specific weekdays during which monitoring will take place. Any number of rules can be defined, letting you set up a complex pattern of resource monitoring for a profile, covering the periods important to you. The scheduled times are always interpreted as local times, allowing you to set up a single rule that will monitor the same local time period in different time zones. For example, if your region covers several time zones, but you wish to monitor morning activities in each time zone, a single rule defining the monitoring period as being between 08:00 and 13:00 is interpreted locally in each of the time zones, so that you monitor the same relative period. You should note also that all times of events or activities reported from endpoints or gateways are also logged in the local time of the system from where they originated.

Chapter 2. Planning the implementation

25

2.4.3 Thresholds and indications As mentioned above, indications are sent to signal the occurrence of a situation that might be a problem. This assessment is based on a number of thresholds that can be configured by an experienced user. But an indication usually does not depend on only one threshold value, and the same threshold may be applied to different resource properties. When you want to adjust the models, it is necessary to understand the dependencies, and when events are generated, it will be useful to know what resource properties triggered it off. In this section, we provide you with cross-reference tables for all of the resource models that are shipped with IBM Tivoli Monitoring 5.1 for both UNIX and Windows resource models. The tables are to be interpreted as follows.

How to read the tables The header row contains the name of the thresholds that exist for a certain resource model. The left cell contains the name of the indication that is generated. The Correlation column may contain a number, which indicates a correlation with another indication, as shown in Table 2-18 on page 39. The other cells of the table contain either T (true, which means that the threshold has been reached or exceeded) or F (false, which means the threshold has not been reached or exceeded). If it is blank, then it means that this threshold is not used to determine this indicator. When the dependent thresholds match, whether T or F, an indication is generated by the analyzer and sent to the Aggregator. Example: 򐂰 The indication Low Logical Disk Space is generated when the threshold for Low Disk Space value is true. 򐂰 The indication Slow Logical Drive is generated when both the thresholds for High Queue Length and Total disk Bytes Read Per Second are exceeded. When you change a threshold, you may want to check which indications are influenced by your adaptation.

2.4.4 Windows resource models Each of the following sections will describe the threshold logic that will generate indicators from Windows resource models.

26

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Memory resource model In this model, some indications are triggered due to some calculations based upon resource properties. Therefore, we included the column Calculation, which does not reflect a threshold value that can be adjusted by a user. The dependencies for the memory resource model are shown in Table 2-6.

High paging TMW_HighPaging

Calculated

Low available memory TMW_LowAvail

Calculation

Low cache hits percent (moving average of...)

T

T b

Low available causing hard paging TMW_LowAvailCausingHardPaging Low available causing many problems TMW_LowAvailCausingManyProblems

2

Low available causing soft paging and page file resizing TMW_LowAvailCausingSoftPagePagefile Resize

4

Low available memory is causing excessive soft paging TMW_LowAvailCausingSoftPaging Low available with a high working set TMW_LowAvailHighWS

PinReadHits

a

1

MDLReadHits

F

Correlation

Indications

DataMapHits

Minimum available bytes

T

Properties

CopyReadHits

Excessive paging

Excessive page faults

Thresholds

Minimum committed bytes

Table 2-6 Memory resource model dependencies

5

T

T

T

F

T

T

T

T

T

F

T

T

T

F

T

F

T

T c

Chapter 2. Planning the implementation

27

Low available memory with a small page file TMW_LowAvailWithSmallPageFile

T

Low available with high cache TMW_LowAvailHighCache

T

Calculation

PinReadHits

MDLReadHits

T d

T

Low data map hits TMW_LowDataMapHits

T

Low MDL read hits TMW_LowMDLReadHits

T

Low pin read hits TMW_LowPinReadHits

T 5

T e

Memory leak in system code TMW_MemoryLeakInSC

f

Memory leak in system drivers TMW_MemoryLeakInSD

g

The page file is resizing TMW_PageFileResizing a. Calculated field: Commit Limit - Commit Bytes

28

DataMapHits

T

Low copy read hits TMW_LowCopyReadHits

Memory leak in private bytes TMW_MemoryLeakInPB

CopyReadHits

Calculated

Correlation

Indications

a

Properties

Low cache hits percent (moving average of...)

Minimum committed bytes

Minimum available bytes

Excessive paging

Excessive page faults

Thresholds

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

T T F

T

b. Working set is larger than cache but less than available memory or working set is less than cache, and cache is less than available memory (that means not enough physical RAM) c. Working set is larger than cache and larger than available memory d. Working set is less than cache, but cache is larger than available memory e. Process increases private bytes since last visit and is among top five consumers f. Amount of system memory increases g. Amount of memory in device drivers increases

Network Interface Card resource model The dependencies for the Network Interface Card (NIC) resource model are shown in Table 2-7.

High errored ratio TMW_HighErroredRatio Network Interface Card overworked TMW_NICOverworked Network Interface Card too slow TMW_NICTooSlow High broadcast frames TMW_HighBroadcastFrames Adjust work items TMW_AdjustWorkItems

High work item shortage

High current

High percent utilization Network Interface Card

High percent bytes sec Redirector

Indications

Server

High output queue

High errored out ratio

BroadcastFrames

Properties

Correlation

Thresholds

High percent broadcast

Table 2-7 Network Interface Card resource model dependencies

T 3

T

T

T

F T

T F

F

Chapter 2. Planning the implementation

T

29

F

T

F

F

T

Redirector overloaded TMW_RedirectorOverloaded

T

F

T

T

T

T

T

T

T

Redirector overloaded affecting segment TMW_RedirectorOverloadedAffecting Segment

F

F

F

High work item shortage

High current

High percent utilization Network Interface Card

Redirector

T

High current commands TMW_HighCurrentCommands

Redirector affecting server TMW_RedirectorAffectingServer

30

High percent bytes sec

High percent broadcast

High output queue

High errored out ratio

Server overloaded TMW_ServerOverloaded

Server

Indications

BroadcastFrames

Properties

Correlation

Thresholds

T

Segment affecting redirector TMW_SegmentAffectingRedirector

F

F

F

T

Segment affecting server TMW_SegmentAffectingServer

F

F

F

T

T

Server overloaded affecting segment TMW_ServerOverloadedAffectingSeg ment

F

T

T

T

Server affecting redirector TMW_ServerAffectingRedirector

F

T

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

F

T

T

Parametric event log resource model This resource model examines the Windows NT or Windows 2000 event log and will send an indication if one of the user-specified events occur and is logged in the Windows event log. The dependencies for the parametric event log resource model are shown in Table 2-8.

Correlation

Thresholds

Indications

NT event log occurred TMW_NTEventLogOccurred

Windows event logged

Table 2-8 Parametric event log resource model dependencies

T

This resource model is used to examine the user-specified services for availability and any non-stable conditions. The dependencies for the Parametric Services resource model are shown in Table 2-9.

Parametric services failing service TMW_ParamServicesFailingService

Service failing service

Indications

Correlation

Thresholds

Service stopped service

Table 2-9 Parametric Services resource model dependencies

T

Parametric services stopped service TMW_ParamServicesStoppedService

T

Chapter 2. Planning the implementation

31

Parametric TCP/IP Ports resource model This resource model checks the user-specified TCP and UDP port numbers and generates events whenever these ports are in a specified state or states. The dependencies for the Parametric TCP/IP Ports resource model are shown in Table 2-10.

Correlation

Thresholds

Indications

Parametric Port status TMW_ParamPortStatus

Status of the defined port

Table 2-10 Parametric TCP/IP Ports resource model dependencies

T

Logical Disk resource model The dependencies for the Logical Disk resource model are shown in Table 2-11.

Correlation

Read

Write

Total disk

High percent usage

High queue length

High read bytes per second TMW_HighLogicalDiskReadBytesSec

7

T

F

T

T

F

High write bytes per second TMW_HighLogicalDiskWriteBytesSec

8

F

T

T

T

F

Indications

32

High bytes per second

Thresholds

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Low disk space

Table 2-11 Logical Disk resource model dependencies

High percent usage

High queue length

T

F

1, 2, 6, 10

F

T

T

Logical disk possible fragmentation TMW_LogicalPossibleFrag

11

F

T

F

Slow logical drive TMW_SlowLogicalDrive

13

T

Low logical disk space TMW_LowLogicalDiskSpace

4

High transfer rate TMW_HighLogicalDiskXferRate

9

T

High percent disk time TMW_HighLogicalPercentDiskTime

Indications

Low disk space

Total disk

T

Read

T

Correlation

Write

High bytes per second

Thresholds

T T

Physical Disk resource model The dependencies for the Physical Disk resource model are shown in Table 2-12 on page 34.

Chapter 2. Planning the implementation

33

Table 2-12 Physical Disk resource model dependencies Printing resource model

High queue length

T

T

T

T

T

F

T

F

T

T

F

F

T

T

T

F

11

F

T

F

13

T

High percent disk time TMW_HighPhysicalPercentDiskTime

6, 10

High transfer rate TMW_HighPhysicalDiskXferRate

9

T

High read bytes per second TMW_HighPhysicalDiskReadBytesSec

7

High write bytes per second TMW_HighPhysicalDiskWriteBytesSec

8

Physical disk possible fragmentation TMW_PhysicalPossibleFrag Slow physical drive TMW_SlowPhysicalDrive

Write

Read

Properties indications

Total disk

F

Correlation

High percent usage

High bytes per second

Thresholds

T

In addition to the thresholds, you may specify the number of printer queues and jobs within each printer queue that will be examined by the resource model. These settings influence the performance and CPU load generated by Tivoli Distribute Monitoring (Advanced Edition) 4.1. The dependencies for the Printing resource model are shown in Table 2-13 on page 35.

34

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Table 2-13 Printing resource model dependencies Process resource model

High job errors per day TMW_HighJobErrorsPerDay

Percent processor

Out of paper errors

Out of paper errors per day

Not ready errors

Not ready errors per day

Job errors

Job errors per day

Indications

Correlation

Thresholds

T

High job errors TMW_HighJobErrors

T

High not ready errors per day TMW_HighNotReadyErrorsPerDay

T

High not ready errors TMW_HighNotReadyErrors

T

High out of paper errors per day TMW_HighOutOfPaperErrorsPerDay

T

High out of paper errors TMW_HighOutOfPaperErrors

T

High current percent time TMW_HighCurrentPercentTime

T

In addition to the thresholds, it is possible to specify the number of processes that are analyzed. The dependencies for the Process resource model are shown in Table 2-14.

Process high CPU TMW_ProcessHighCPU

High CPU use

Indications

Correlation

Thresholds

12

T

Chapter 2. Planning the implementation

Max handles

Table 2-14 Process resource model dependencies

35

Indications

Max handles

Correlation

High CPU use

Thresholds

Process handle leak TMW_ProcessHandleleak

T

Processor resource model The dependencies for the Processor resource model are shown in Table 2-15.

T

CPU Cannot Keep Up With HW TMW_CPUCantKeepUpWithHW

T

T

F

T

HW Keeping CPU Busy TMW_HWKeepingCPUBusy

T

T

F

Fa

T

F

T

T

F

F

High Processes TMW_HighProcesses

12

Processor Busy TMW_ProcessorBusy High Percent Usage Delta TMW_HighPercentUsageDelta

Total CPUs mod

T

High percent usage delta

High CPU usage interrupt

T

Busy Hardware TMW_BusyHardware

High interrupts sec

High CPU usage

5

Indications

High CPU usage process

Correlation

Thresholds

High CPU usage user priv

Table 2-15 Processor resource model dependencies

T

a. The processor queue length is greater than the total number of CPUs plus the TotalCPUsMod threshold.

36

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Services resource model The services model works in a slightly different way. The threshold values have a logical value set of {0, 1}. In contrast to the usual meaning, these values are Boolean variables. Everything greater than zero is equal to true; zero means false. If the value is set to 1 or a number greater than that, then the indications may be triggered when the corresponding service either failed or was stopped. In addition to that, IBM Tivoli Monitoring 5.1 will try to restart a stopped service. If that fails, another event is generated and sent to TEC. The dependencies for the Services resource model are shown in Table 2-16.

Any failing service TMW_ServicesFailingService

Lcfd

LanmanServer

NtLmSsp

Netlogon

EventLog

Indications

Browser

Thresholds

LanmanWorkstation

Table 2-16 Services resource model dependencies

T T T T T T T

Services stopped service TMW_ServicesStoppedService

T T T T T T T

Chapter 2. Planning the implementation

37

Note that NT or Windows 2000 machines that are part of a workgroup do not run the Netlogon service; starting this service will fail. Therefore, to avoid large numbers of reoccurring service events on workgroup members, you should turn off monitoring for this service by setting the resource model property to zero for the Netlogon service.

TCP/IP resource model The dependencies for the TCP/IP resource model are shown in Table 2-17. Table 2-17 TCP/IP resource model dependencies

High fragment ratio TMW_HighFragRatio Segments resubmitted TMW_SegmentsReXmit

Low segments

High fragment ratio

High segment retransmitted

SegmentsReceived

FragmentsReceived, DatagramsReceived

SegementsRetransmitted

High ping TMW_HighPing

Moderate DG

Indications

Datagramsreceived

Properties

Correlation

Thresholds

T

F T

3

T

EventLog resource model This model works in a different way. It scans the NT event log and sends indications when it finds an event of a specific type. It does not depend on thresholds. You can specify how many events are examined. This number will affect the performance of IBM Tivoli Monitoring 5.1.

38

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The event log indications are: 򐂰 NT event log 9 (TMW_EventID9) The device, [name], did not respond within the timeout period. 򐂰 NT event log 11 (TMW_EventID11) The driver detected a controller error on [text]. 򐂰 NT event log 15 (TMW_EventID15) The device, [name], is not ready for access yet. 򐂰 NT event log 2011 (TMW_EventID2011) The server's configuration parameter ipstacksize is too small for the server to use a local device. Please increase the value of this parameter. 򐂰 NT event log 2511 (TMW_EventID2511) The server service was unable to recreate the share name because the directory path no longer exists. 򐂰 NT event log 3013 (TMW_EventID3013) The redirector has timed out a request to [text]. 򐂰 NT event log 7023 (TMW_EventID7023) The [name] service terminated with the following error: [text].

2.4.5 Correlated events Correlated events are generated from two indications, in most cases, originated in different resource models. Table 2-18 provides an overview of the dependencies of the correlated events. For a more detailed description of the correlated events, refer to the IBM Tivoli Monitoring Resource Model Reference Version 5.1, SH19-4570. Table 2-18 Table of correlated indications

1

Generated indication

Correlated indication

TMW_BusyDriveFromPaging

TMW_HighLogicalPercentDiskTime TMW_HighPaging indication

2

TMW_BusyDriveFromLowAvail

TMW_LowAvailCausingManyProblems TMW_HighLogicalPercentDiskTime

3

TMW_CongestedTCPNetwork

TMW_NICOverworked TMW_SegmentsReXmit

Chapter 2. Planning the implementation

39

4

Generated indication

Correlated indication

TMW_CriticallyLowDiskSpace

TMW_LowAvailCausingSoftPagePagefileR esize TMW_LowLogicalDiskSpace

5

TMW_CriticalMemoryLeakInWS

TMW_MemoryLeakInPB TMW_LowAvailHighWS

6

TMW_FaultyDiskSubsystem

TMW_HighPhysicalPercentDiskTime TMW_BusyHardware indication TMW_HighLogicalPercentDiskTime

7

TMW_HighDiskReadBytesSec

TMW_HighPhysicalDiskReadBytesSec TMW_HighLogicalDiskReadBytesSec

8

TMW_HighDiskWriteBytesSec

TMW_HighLogicalDiskWriteBytesSec TMW_HighPhysicalDiskWriteBytesSec

9

TMW_HighDriveXferRate

TMW_HighLogicalDiskXferRate TMW_HighPhysicalDiskXferRate

10

TMW_HighPercentDiskTime

TMW_HighPhysicalPercentDiskTime TMW_HighLogicalPercentDiskTime

11

TMW_PossibleFrag

TMW_LogicalPossibleFrag TMW_PhysicalPossibleFrag

12

TMW_ProcessHoggingCPU

TMW_HighProcesses TMW_ProcessHighCPU

13

TMW_SlowHardDrive

TMW_SlowPhysicalDrive TMW_SlowLogicalDrive

2.4.6 UNIX resource models The tables in this section are for the UNIX/Linux resource models. Each table can be interpreted in the same way as the Windows reference models. Any differences or unique characteristics to a specific table will be documented with the relevant tables.

40

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

CPU resource model This resource model detects problems with the central processing unit of a computer, for example, how long processes wait in the queue to be processed. The dependencies for the CPU resource model are shown in Table 2-19. Table 2-19 CPU resource model

Percentage of CPU used by system

Percentage of CPU idle

Thresholds

Low

Properties Indications

High CPU overload Low_IdleCPUUsage

T

High CPU usage by system High_SysCPUUsage

T

Memory resource model This resource model provides information about how the memory is used. The dependencies for the resource model are shown in Table 2-20.

Low storage space LowStorage Low swap space LowSwap

Memory page-in rate

Memory page-out rate

Indications

Percentage of available swap space

Thresholds

Percentage of available virtual storage

Table 2-20 Memory resource model

T T

Chapter 2. Planning the implementation

41

Memory page-out rate

Memory page-in rate

Percentage of available swap space

Indications

Percentage of available virtual storage

Thresholds

T

T

System thrashing Thrashing

File resource model The file resource model gives information about files in the system. This resource model does not have thresholds that certain conditions are compared against. Instead it checks for changes in files, file attributes, and so on. We provide the system resource that it uses to calculate which event should be generated. The dependencies for this resource model are shown in Table 2-21. Table 2-21 File resource model

File not present FileNotPresent

42

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Change in CRC /etc/hosts

File attributes changed FilesAttributeChange

Change in file attributes

File changed FileChanged

/etc/group

Indications

Change in file Properties

/etc/passwd

Thresholds

T

T

T

T

F

This event is generated if the file does not exist.

Process resource model The Process resource model looks for bottlenecks that occur in running processes. Problems highlighted include: 򐂰 򐂰 򐂰 򐂰

A process uses too much CPU time. Too many zombie processes in the system. A process is stopped or killed. A process that was requested does not exist.

This resource model monitors processes that are specified in the parameter list. By default, it monitors lcfd and syslogd. The dependencies for this resource model are shown in Table 2-22. Table 2-22 Process resource model

lcfd

Properties

Indications

High number of zombie processes HighZombieProcesses

syslogd

Percent of CPU used

Max zombie processes

Thresholds

T

Process consuming high CPU ProcessHighCPU

T

Process killed or nonexistent ProcessKilledOrNotExisting

Sent if a process is killed or does not exist.

Process stopped ProcessStopped

Sent for each monitored process that is stopped.

Network Interface resource model The Network Interface resource model detects problems with all the installed network interfaces. Events are generated when performance data, such as bytes per second in and out and sessions with errors or requests, becomes critical. The dependencies for this resource model are shown in Table 2-23 on page 44.

Chapter 2. Planning the implementation

43

High percentage packet collision HighPacktsCollision High output packets in error HighOutErrorPacks

T

T

High input packets in error HighInputErPacks

UP&RUNNING

T

T

T

T

T

T

T

T

T

Interface not operational InterfaceNotOperat

T

Unknown interface status IntStatUnknown

T

Network Interface Card not supported IntNotSupported

F

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

UNKNOWN

Ta

Interface not enabled InterfaceNotEnabled

a. Ethernet interface only.

44

T

UP&NOTRUNNING

Indications

Down

Token Ring, Ethernet, or LOOPBACK

Properties

Interface status

Interface

Input packets in error

Output packets in error

Thresholds

Percent of packet collision

Table 2-23 Network Interface resource model

T T T

File system resource model This resource model measures how efficiently the file systems are being used. The dependencies for this resource model are shown in Table 2-24.

Percentage of available I-nodes

Available space (KB)

Percentage of I-nodes used

Thresholds

Percentage of file system space used

Table 2-24 File system resource model

Indications

Fragmented file system FragmentedFileSystem Low space available LowKAvail

Low

Properties

T

F T

Low percentage of available I-nodes LowPercInodesAvail

T

Security resource model The Security resource model provides information about files and the users logged onto the system. It highlights the following items or changes that might indicate security breaches: 򐂰 򐂰 򐂰 򐂰

Property changes, such as the owner, group, or attributes, for certain files The number of logons onto the system by the same user A suspect superuser An account that is not valid for root

The dependencies for the resource model are shown in Table 2-25 on page 46 and Table 2-25 on page 46.

Chapter 2. Planning the implementation

45

High log-in number for user HighLoggingNumber

Duplicated

Null password

F

T

F

T T

Account not valid for root NotRegularRootAccount

F F

Custom logins

T

F

Suspect super group SuspectSuperGroup

F

F

T

F

T Ta

Null password PasswordNull Suspect superuser SuspectSuperUser

root

F

Superusers

root

F

Supergroups

Special groups

Defined users

Special users

Duplicated account DuplicatedAccount

UID 0

UID -1

Properties

Indications

Alternative groups

Thresholds

Instrumentation

Table 2-25 Security resource model and file monitoring

T

F

F

a. This indicator will also happen for a group with a null password, but only for HP-UX 10 and Solaris 2 systems.

46

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Alternative owners

File group

Files to be monitored

Alternative groups

File owner

File mode

Member

Matches

File Exists

Member

Matches

Matches

Thresholds

F

T

Properties Indications

Illegal owner IllegalOwner

F

Illegal group IllegalGroup

T F

T

Wrong file mode WrongMode

T

Nonexistent file FileNoteExisting

F

F

T

T F

Network RPC/NFS resource model The Network RPC/NFS resource model detects problems and monitors the performance of the RPC and NFS servers and clients. This resource model should only be distributed to Solaris machines. The dependencies for this resource model are shown in Table 2-26 on page 48.

Chapter 2. Planning the implementation

47

High NFS server read operations HighNFSSrvRead

T

High retransmitted calls HighPercRetrans

T

High network traffic NetworkBusy

T

T

High NFS buffer size HighNFSBufferSize

T

T

High NFS server get-attribute operations HighNFSSrvGetattr

T

High RPC bad calls HighPercRPCBadCalls

T

Slow network NetworkSlow

T

High NFS server readlink operations HighNFSSrvReadLink High timeout and badxids HighTimeoutsAnd_Badxids High timeout and badxids HighTimeoutsAnd_Badxids

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

T T T T

Percent of NFS svr write operations

Percent of client RPC calls in timeout

Percent of client RPC badxids

Percent of NFS svr readlink operations

Percent of bad RPC calls

Percent of NFS svr getattr operations

T

High duplicate RPC server calls HighPercDupReqs

48

Percent of Svr RPC dup requests

Indications

Percent of NFS srv read ops

Thresholds

Percent of Client RPC retranmissions

Table 2-26 Network RPC/NFS resource model (Sun Solaris specific)

High NFS server write operations HighNFSSrvWrites

Percent of NFS svr write operations

Percent of client RPC calls in timeout

Percent of client RPC badxids

Percent of NFS svr readlink operations

Percent of bad RPC calls

Percent of NFS svr getattr operations

Percent of Client RPC retranmissions

Percent of Svr RPC dup requests

Indications

Percent of NFS srv read ops

Thresholds

T

2.5 Installation and configuration planning IBM Tivoli Monitoring 5.1 can be installed on any ManagedNode within your Tivoli Managed Environment. There are product components that must be installed on the Tivoli server, gateways, and the endpoints you are going to manage. Before you begin to install the product, you must consider the minimum requirements and configuration issues, as described in the next few pages. Note that the requirement information provided here is only an overview to assist in your planning effort. For a more detailed description, you must refer to Chapter 4, “Installation and migration” on page 71, and also the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569, and IBM Tivoli Monitoring Release Notes Version 5.1.1, GI10-5797.

2.5.1 Hardware requirements There are IBM Tivoli Monitoring 5.1 components that are installed and run on the server, gateway, and endpoint. However, since the IBM Tivoli Monitoring 5.1 engine component is run on the endpoint, most of the computing power will be necessarily at the endpoint itself. The IBM Tivoli Monitoring 5.1 engine, along with the required resource models, are automatically installed the first time a IBM Tivoli Monitoring 5.1 profile is distributed to the endpoint. Since this can impact your network environment, you must determine if special planning is required before you start distributing profiles for the first time.

Chapter 2. Planning the implementation

49

Disk consumption The IBM Tivoli Monitoring 5.1 engine disk space requirements range from 8 MB to 16 MB depending on the endpoint platform type. This space includes the IBM Tivoli Monitoring 5.1 engine binaries, which are installed on the endpoint with the initial distribution of IBM Tivoli Monitoring 5.1 resource models. This does not include the space required for the Java Runtime Environment on the UNIX endpoint platforms. The average IBM Tivoli Monitoring 5.1 disk space requirements range from 8 KB to 13 KB per resource model, depending on the endpoint platform type.

Memory consumption Memory requirements for the IBM Tivoli Monitoring 5.1 endpoint processes are only slightly affected by increases in the number of resource models active on both Windows and UNIX platforms. The base memory required for the IBM Tivoli Monitoring 5.1 endpoint on the Windows platform averaged about 6 MB, while the IBM Tivoli Monitoring 5.1 endpoint on the UNIX platform required about 10 MB. Overall memory demand for a machine is increased by about another 3 MB on a Windows endpoint when processing 20 resource models simultaneously. On the UNIX platform, the memory requirement is increased by about 2 MB when processing 20 resource models.

Processor consumption Comparing the CPU utilization for the IBM Tivoli Monitoring 5.1 endpoint on the various platform types shows that for four resource models running at 60 second time intervals, the amount of CPU used is very similar and falls in the 2–3 percent range. A short interval time, such as 10 seconds, can have a dramatic impact on endpoint performance and should be used sparingly. As well, additional resource models will have an effect on CPU consumption. We advise testing your specific set of resource models running at your chosen time intervals on a system with hardware and software that is representative of your environment to get an accurate picture of CPU consumption.

2.5.2 Software requirements In order to use IBM Tivoli Monitoring 5.1 to effectively monitor your systems and resources, you need to verify that the Tivoli environment, including the Tivoli management region server, endpoint gateway, and the Tivoli Enterprise Console server, have the required patches, as described in 4.1.3, “Software requirements” on page 75.

50

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Windows NT The following are required for Windows NT: 򐂰 Windows Management Instrumentation (WMI) Core Version 1.5 http://www.microsoft.com/downloads/release.asp?ReleaseID=18491

򐂰 Microsoft Data Access Components (MDAC) Version 2.6 http://www.microsoft.com/downloads/release.asp?ReleaseID=30894

򐂰 Jet 4.0 Service Pack 3 http://www.microsoft.com/downloads/release.asp?ReleaseID=25482

Windows 2000 No prerequisites other than the OS.

UNIX Java Runtime Environment (JRE) 1.3.0 or 1.3.1 must be installed on the endpoint (endpoints running Solaris must have JRE 1.3.1-01): 򐂰 If you already have an appropriate copy of JRE installed on the target system, after installing IBM Tivoli Monitoring 5.1 you should use the provided task, DMLinkJre, to link the product to your existing JRE (see “Installing JRE” on page 99). Use the DMRemoveLinkJre task to remove the link to JRE. 򐂰 If a UNIX or Linux endpoint does not have an appropriate copy of JRE 1.3.0 installed, you can install it from the product CD. 򐂰 You may also use the -J option of the wdmdistrib command to push the JRE to the endpoint in conjunction with the profile distribution.

2.6 Planning for migration This section will discuss some of the key items you should focus on in preparation for migrating to IBM Tivoli Monitoring 5.1.

2.6.1 Think migration, not upgrade At the heart of IBM Tivoli Monitoring 5.1 is a fundamentally different product than the "classic" Distributed Monitoring products at Version 3.7 and prior. This difference means that moving to IBM Tivoli Monitoring 5.1 should be viewed as a migration , not simply an upgrade. Tools have been provided to help ease this migration (see 5.1, “Migration process” on page 112, for more information about

Chapter 2. Planning the implementation

51

these tools). This is an opportunity to rethink some of your monitoring architecture; your environment has matured, you and your team have more experience with monitoring, and the product has matured and likely provides features unavailable when you first deployed classic Distributed Monitoring.

2.6.2 Auditing your current environment Prior to using the Tivoli-provided migration tools, it would be a good idea to audit and clean up your environment. Things you’re looking for include: 򐂰 򐂰 򐂰 򐂰

Unused sentry profiles Disabled monitoring probes Asynchronous monitors User ID and group ID customizations

Unused sentry profiles Ask yourself some of the following questions: 򐂰 Are there any sentry profiles to which nothing is subscribed? 򐂰 Have you created monitoring profiles in anticipation of something that never came about? 򐂰 Did your original architecture include a large number of "empty containers" simply to make the environment seem consistent? 򐂰 Can you remove the monitoring profiles specific to SomeProduct Version 5.0 now that your company has completely rolled out SomeProduct Version 6.0? To help find sentry profiles that have no subscribers, the sample script in Example 2-1 can be modified to suit your environment. A sample of the output is provided in Example 2-2 on page 54. See Appendix A, “Additional material” on page 529, for instructions on how to access an electronic copy. Important: Some Tivoli-created profile managers have no subscribers, but should not be removed. In Example 2-2 on page 54 this would include Tivoli/Sentry Defaults-tividc11-region.

Example 2-1 find_no_subscribers.sh script #!/bin/sh # # Scan all Profile Managers containing Sentry Profiles # report those with no subscribers. # Jamie Carl - IBM/Tivoli Software Group # unsupport work #

52

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

SentryProfileList="inputfile.SentryProfile.ITM5.1.redbook" ProfileManagerList="inputfile.ProfileManager.ITM5.1.redbook" SortedProfileManagerList="inputfile.sorted.ProfileManager.ITM5.1.redbook" rm $SentryProfileList > /dev/null 2>&1 rm $ProfileManagerList > /dev/null 2>&1 rm $SortedProfileManagerList > /dev/null 2>&1 echo "" echo "Building list of Sentry Profiles. . . \c" wls /Library/SentryProfile > $SentryProfileList echo "OK" echo "" echo "Building list of Profile Managers containing those Sentry Profiles. . . \c" while read SentryProfileLabel do SentryProfileOID=`wlookup -r SentryProfile "$SentryProfileLabel"` ProfileManagerOID=`idlattr -tgv $SentryProfileOID profile_organizer Object` ProfileManagerLabel=`idlattr -tgv $ProfileManagerOID label string | sed s/\"//g` echo $ProfileManagerLabel >> $ProfileManagerList done < $SentryProfileList echo "OK" echo "" echo sort echo echo

"Sorting and filtering out duplicate Profile Managers. . .\c" -u $ProfileManagerList > $SortedProfileManagerList "OK" ""

while read ProfileManagerLabel do echo "Scanning ProfileManager:$ProfileManagerLabel for subscribers. . . \c" numSubscribers=`wgetsub @ProfileManager:"$ProfileManagerLabel" | wc -l` if [ "$numSubscribers" -eq "0" ] then echo "NONE" echo " NONE: $ProfileManagerLabel has no subscribers!" else echo "OK" fi done < $SortedProfileManagerList

Chapter 2. Planning the implementation

53

Example 2-2 find_no_subscribers.sh script output tividc11:/tivoli>./find_no_subscribers.sh Building list of Sentry Profiles. . . OK Building list of Profile Managers containing those Sentry Profiles. . . OK Sorting and filtering out duplicate Profile Managers. . .OK Scanning ProfileManager:Classic for subscribers. . . NONE NONE: Classic has no subscribers! Scanning ProfileManager:Tivoli/Sentry Defaults-tividc11-region for subscribers. . . NONE NONE: Tivoli/Sentry Defaults-tividc11-region has no subscribers! Scanning ProfileManager:async.pm for subscribers. . . OK Scanning ProfileManager:pf.tividc-11.dm for subscribers. . . NONE NONE: pf.tividc-11.dm has no subscribers! Scanning ProfileManager:spooler.pm for subscribers. . . OK

Disabled monitoring probes Monitoring probes are frequently disabled rather than removed from a monitoring profile. The migration script will report on all monitoring probes it finds, regardless of whether they are enabled or disabled. Now is the best time to remove unused and/or obsolete monitoring probes. To help find disabled monitors, the sample script in Example 2-3, can be modified to suit your environment. A sample of the output is provided in Example 2-4 on page 55. See Appendix A, “Additional material” on page 529, for instructions on how to access an electronic copy. Example 2-3 find_disabled_monitors.sh script #!/bin/sh # # Scan for disabled monitoring probes in all SentryProfiles # Jamie Carl - IBM/Tivoli Software Group # unsupport work # for SentryProfileLabel in `wls /Library/SentryProfile` do echo "SCANNING SentryProfile:$SentryProfileLabel for disabled monitors. . . \c" foundDisabled=`wlsmon -sa "$SentryProfileLabel" | grep "+d"` if [ "$foundDisabled" ] then echo "FOUND" SentryProfileOID=`wlookup -r SentryProfile "$SentryProfileLabel"` ProfileManagerOID=`idlattr -tgv $SentryProfileOID profile_organizer Object`

54

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

ProfileManagerLabel=`idlattr -tgv $ProfileManagerOID label string | sed s/\"//g` echo " FOUND: $SentryProfileLabel within $ProfileManagerLabel contains disabled monitoring probes!" echo " $SentryProfileLabel,$ProfileManagerLabel,$ProfileManagerOID" else echo "OK" fi done

Example 2-4 has the output. Example 2-4 find_disabled_monitors.sh script output tividc11:/tivoli>./find_disabled_monitors.sh SCANNING SentryProfile:TivoliSentryDefaults for disabled monitors. . . OK SCANNING SentryProfile:sentry.1 for disabled monitors. . . OK SCANNING SentryProfile:example5 for disabled monitors. . . OK SCANNING SentryProfile:example6 for disabled monitors. . . OK SCANNING SentryProfile:print.spooler.monitoring.prf for disabled monitors. . . FOUND FOUND: print.spooler.monitoring.prf within spooler.pm contains disabled monitoring probes! print.spooler.monitoring.prf,spooler.pm,1931340152.1.1100#TMF_CCMS::ProfileManager# SCANNING SentryProfile:async.prf for disabled monitors. . . OK

Asynchronous monitors IBM Tivoli Monitoring 5.1 currently has no specific migration steps for universal-asynchronous string and universal-asynchronous numeric monitors. Special effort will be necessary to migrate these monitors into your environment. Understand why these monitors are being used and consider what alternatives are available. Can new functionality within IBM Tivoli Monitoring 5.1 overcome your need for asynchronous monitors? To help find asynchronous monitors, the sample script in Example 2-5 can be modified to suit your environment. A sample of the output is provided in Example 2-6 on page 56. See Appendix A, “Additional material” on page 529, for instructions on how to access an electronic copy. Example 2-5 find_async_monitors.sh script #!/bin/sh # # Scan for Universal/Asyncronous String and Asyncronous Numeric # monitors in all SentryProfiles # Jamie Carl - IBM/Tivoli Software Group # unsupport work # for SentryProfileLabel in `wls /Library/SentryProfile`

Chapter 2. Planning the implementation

55

do echo "SCANNING SentryProfile:$SentryProfileLabel for async monitors . . . \c" foundAsync=`wlsmon -sa "$SentryProfileLabel" | grep "async"` if [ "$foundAsync" ] then echo "FOUND" SentryProfileOID=`wlookup -r SentryProfile "$SentryProfileLabel"` ProfileManagerOID=`idlattr -tgv $SentryProfileOID profile_organizer Object` ProfileManagerLabel=`idlattr -tgv $ProfileManagerOID label string | sed s/\"//g` echo " FOUND: $SentryProfileLabel within $ProfileManagerLabel contains async monitors!" else echo "OK" fi done

Example 2-6 shows the output. Example 2-6 find_async_monitors.sh script output tividc11:/tivoli>./find_async_monitors.sh SCANNING SentryProfile:TivoliSentryDefaults for async monitors . . . OK SCANNING SentryProfile:sentry.1 for async monitors . . . OK SCANNING SentryProfile:example5 for async monitors . . . OK SCANNING SentryProfile:example6 for async monitors . . . OK SCANNING SentryProfile:print.spooler.monitoring.prf for async monitors . . . OK SCANNING SentryProfile:async.prf for async monitors . . . FOUND FOUND: async.prf within async.pm contains async monitors!

User ID and group ID customizations Tivoli Distributed Monitoring (Classic Edition) 3.7 and prior had the ability to customize the user ID and group ID under which the monitoring probes within the Distributed Monitoring Profile where executed. (See Figure 2-3 on page 57.) If no customization was made to the user ID or group ID, the monitoring probes defaulted to run as nobody/nobody. In IBM Tivoli Monitoring 5.1, all monitoring and local program execution is run as the $root_user ID map (widmap list_entries root_user). If you specify a TME Task as a response action, this task can be configured to run under any account you wish.

56

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 2-3 Customized user ID in a classic Distributed Monitoring Profile

To help report user ID and group ID assignments, the sample script in Example 2-7 can be modified to suit your environment. A sample of the output is provided in Example 2-8 on page 58. See Appendix A, “Additional material” on page 529, for instructions on how to access an electronic copy. Example 2-7 find_profile_accounts.sh script #!/bin/sh # # Dump a listing of all Sentry Profiles and their userid/groupid. # # Jamie Carl - IBM/Tivoli Software Group # unsupport work # for SentryProfileLabel in `wls /Library/SentryProfile` do UserGroup=`wgetsntid "$SentryProfileLabel"` echo "$SentryProfileLabel,$UserGroup" done

Chapter 2. Planning the implementation

57

Example 2-8 find_profile_accounts.sh script output tividc11:/tivoli>./find_profile_accounts.sh TivoliSentryDefaults,user: nobody group: nobody sentry.1,user: nobody group: nobody example5,user: nobody group: nobody example6,user: nobody group: nobody print.spooler.monitoring.prf,user: ausres00 group: nobody async.prf,user: nobody group: nobody

2.6.3 Tivoli Monitoring versions that can coexist It is a supported function and feature of IBM Tivoli Monitoring 5.1 to coexist with Tivoli Distributed Monitoring (Classic Edition) 3.7. Obviously running two separate monitoring engines will result in higher resource consumption on your monitored systems, so this should be leveraged only when necessary (during your migration period, for example). It is not possible to run IBM Tivoli Monitoring 5.1 with prior releases of products that are resource model-based, such as Tivoli Distributed Monitoring (Advanced Edition) 4.1, Tivoli Distributed Monitoring for Windows 3.7, and Tivoli Manager for NT 3.6.2.

2.7 Summary As with any IT initiative, successful implementation of IBM Tivoli Monitoring 5.1 as the enterprise systems monitoring tool, requires careful planning and preparation. In your planning effort, keep in mind that the IBM Tivoli Monitoring 5.1 implementation should closely fit in with your overall systems management strategy. This strategy, regardless of the methodology used, must encompass the necessary organization and technology elements. As discussed in this chapter, planning for a IBM Tivoli Monitoring 5.1 implementation will involve the following: 򐂰 Understanding your current environment 򐂰 Identifying the requirements and the issues you are going to address 򐂰 Being aware of the minimum installation requirements 򐂰 Moving from Classic Distributed Monitoring to IBM Tivoli Monitoring 5.1 should be approached as a migration, not simply a product upgrade

58

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 While designing a general solution, concentrating on the most critical IT resources 򐂰 Enabling other systems management tools and process to make efficient use of the data generated by IBM Tivoli Monitoring 5.1

Chapter 2. Planning the implementation

59

60

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

3

Chapter 3.

Firewall considerations This chapter describes the flow of information between the various Tivoli components as they relate to IBM Tivoli Monitoring Version 5.1. This information will be useful to customers that need to implement this product in a firewall environment. In this chapter we discuss: 򐂰 򐂰 򐂰 򐂰

Tivoli Firewall Security Toolbox Tivoli Monitoring Web Health Console Distributing Resource Models Endpoint-generated TEC events

© Copyright IBM Corp. 2002

61

3.1 The Tivoli Firewall Security Toolbox For our testing, we have utilized the Tivoli Firewall Security Toolbox Version 1.2. This product is generally available for download on the Tivoli support Web site at: ftp://ftp.tivoli.com/support/patches/patches_1.2/1.2-TFS-0001/

This chapter will not go into extensive detail on how to install the Tivoli Firewall Security Toolbox; for further information on installation and configuration of this product, consult the product Release Notes (located within the .tar file from the FTP site mentioned above). The Tivoli Firewall Security Toolbox allows you to install and maintain your Tivoli gateway systems within your more secure LAN environment, rather than requiring gateways to be pushed outside the firewall and into the DMZ. The four components of this product include: 1. Endpoint Proxy - This component is utilized by the gateway, without requiring the gateway to explicitly know the component exists. The purpose of the Endpoint Proxy is to emulate an actual endpoint to the gateway. The Endpoint Proxy funnels requests for a specified endpoint through a single, configurable TCP/IP port and passes it down to its child, which may be a gateway proxy or relay. 2. Relay - This component is a non-essential piece of the Tivoli Firewall Security Toolbox. The Relay component can be placed between “layers” of firewalls, allowing endpoints to be manageable even if they are separated from their gateway by multiple firewalls. A Relay’s sole purpose is to pass information sent to it up or down the chain to an endpoint proxy, gateway proxy, or other relays. 3. Gateway proxy - This component is installed in the DMZ and serves to emulate a Tivoli gateway. Endpoints are configured to point to this system as their gateway. As with the endpoint proxy, the endpoint is not explicitly aware of the fact that this destination is not truly a gateway. 4. Event sink - This component also is installed in the DMZ and serves to emulate a Tivoli Enterprise Console (TEC) Server. Any non-TME (also known as non-secure) adapters in the zone served by this event sink are configured to point to this event sink system as their TEC Server. The event sink is the only component of the four that requires installation on a system with a Tivoli endpoint. When the event sink receives an event, the event is passed to the endpoint. The endpoint subsequently sends the event to the TEC Server through the Framework as a TME or secure event. Figure 3-1 on page 63 illustrates the flow of data within IBM Tivoli Monitoring 5.1 for the Web Health Console.

62

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

End User / Browser Client Random

HTTPS

ITSODEV4 146.84.31.19

HTTP 443 80 IBM HTTP Server Random 9080 Websphere Application Server Random

Web Health Console Server

94

TMR Server

Figure 3-1 Data flow from end user to TMR Server for Web Health Console

Figure 3-2 on page 64 and Figure 3-3 on page 65 (combined) illustrate the flow of data within IBM Tivoli Monitoring 5.1 utilizing the Tivoli Firewall Security Toolbox.

Chapter 3. Firewall considerations

63

Web Health Console Server

TMR Server 94 iom/bdt Port Range

Port Range iom/bdt

ITWIN1 146.84.31.3

94

port-range parm: 6000+

Endpoint Proxy Random 7070

Gateway 9494

TIVIDC11 9.3.240.68

Random

7071

ITWIN2 146.84.31.4

Random Relay

7070

Random

Gateway Proxy

Figure 3-2 Data flow from Web Application Server to gateway proxy

In Figure 3-2, you will notice a communication between the TMR Server and gateway simply marked iom/bdt. Depending on your TMR configuration, this conversation takes place differently. At a high level, four possible scenarios are: 1. No port-range restrictions, no single-port BDT The conversation is over random ports. 2. Port-range restrictions The conversation is restricted to a defined port range. 3. Single-port BDT enabled The conversation initiates from a random port and is directed to a single, defined port (default 9401).

64

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

4. Both port-range restrictions and single-port BDT enabled The conversation initiates from a defined port range and is directed toward the single, defined port (default 9401).

Relay

ITSO26 10.69.14.26 9444

Random

LCFD

Event Sink

7071

Gateway Proxy Random

9494 Range Random TME Event

TME Event

Non-TME Event

ITSO27 10.69.14.27

Non-TME Event Generator (eg. ITM engine, postemsg)

Random

9495

LCFD Downcall

Figure 3-3 Data flow from Relay to Tivoli Endpoint/LCF

In Figure 3-3, you will notice that all communication goes through a gateway proxy. This includes TME events, non-TME events, and all Framework communication.

3.2 Special considerations for firewall environments The following sections describe what special considerations must be made when using IBM Tivoli Monitoring 5.1 within a firewall environment.

Chapter 3. Firewall considerations

65

3.2.1 Web Health Console The Web Health Console is the only IBM Tivoli Monitoring 5.1 component that truly requires special attention, as the remaining components utilize the Tivoli Framework and/or the Tivoli Firewall Security Toolbox. The Web Health Console is an IBM WebSphere application hosted by an IBM HTTP Server. For more detailed information about the Web Health Console itself, see Chapter 8, “Web Health Console” on page 179. As shown in Figure 3-1 on page 63, accessing the Web Health Console from an end user’s Web browser requires either: 򐂰 Port 80 (standard HTTP) 򐂰 Optionally port 443 (standard secure HTTP) The Web Health Console server must have connectivity to the Tivoli Management Region (TMR) Server on port 94. All of the information passed to the end user’s Web browser is through the IBM HTTP Server, at no time does the end user system connect directly to other Tivoli components, such as the TMR Server, gateway, or endpoint.

3.2.2 Distributing Resource Models Distributing Resource Models to endpoints utilizes the Multiplexed Distribution Version 2 (MDIST2) functionality of the Framework. No special configuration should be necessary to facilitate the distribution of Resource Models. It should be noted, however, that when the flow of data passes through components of the Tivoli Firewall Security Toolbox, there will be some performance loss. The loss in performance is influenced by factors such as the speed of firewall and the number of Relay components over which the data must hop before reaching its intended recipient; the more Relays the slower the distribution.

3.2.3 Endpoint-generated TEC events Endpoints can be sent using one of two methods, TME event or non-TME event.

TME events If you have configured your Tmw2kProfile to use TME Delivery (see Figure 3-4 on page 67 for an example) it can be configured in a standard fashion, pointing to the appropriate eventServer object. The events will flow via the Framework from the Endpoint to the TEC Server, leveraging the Tivoli Firewall Security Toolbox components along the way.

66

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 3-4 Properties of a profile configured for TME (Secure) Delivery

Non-TME events If you have configured your Tmw2kProfile to use non-TME Delivery (see Figure 3-5 on page 68 for an example), it must be configured to send events to an accessible system running the Event Sink component of the Tivoli Firewall Security Toolbox (see item 4 on page 62). In our environment, ITSO26 (10.69.14.26) has the event Sink component installed and listening on port 9444. This system is not actually a TEC Server, but the non-TME delivery method believes it is. When the event Sink receives the event, it will hand it over to the local Tivoli Management Agent (LCFD), which in turn sends the event as a TME/secure event via the Framework.

Chapter 3. Firewall considerations

67

Figure 3-5 Properties of a profile configured for non-TME (Unsecure) Delivery

68

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Part 2

Part

2

Installation and migration In this part we describe the process for installing IBM Tivoli Monitoring 5.1. This includes migrating from Tivoli Web Component Manager and past Tivoli monitoring products such as: 򐂰 򐂰 򐂰 򐂰

Tivoli Manager for NT 3.6.2 Tivoli Distributed Monitoring for Windows 3.7 Tivoli Distributed Monitoring (Classic Edition) 3.7 Tivoli Distributed Monitoring (Advanced Edition) 4.1

Migration of your custom monitors is an important step when migrating to IBM Tivoli Monitoring 5.1.

© Copyright IBM Corp. 2002

69

70

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

4

Chapter 4.

Installation and migration This chapter describes the installation and migration of the IBM Tivoli Monitoring 5.1 product. In addition, this chapter gives some guidelines for migration from Classic Edition and Advanced Edition. The following topics are discussed in this chapter: 򐂰 Section 4.1, “Installation and migration guidelines” on page 72 򐂰 Section 4.2, “Installation and upgrade” on page 80 򐂰 Section 4.3, “Uninstalling the product” on page 102 򐂰 Section 4.4, “Installing TBSM IBM Tivoli Monitoring 5.1” on page 105 򐂰 Section 4.5, “Installing Web Health Console” on page 105 򐂰 Section 4.6, “Installing IBM Tivoli Monitoring Historical Data Gathering” on page 110

© Copyright IBM Corp. 2002

71

4.1 Installation and migration guidelines Is this section we discuss installation and migration of IBM Tivoli Monitoring 5.1. Before we discuss the steps to build our environments we will describe the environments we are building.

4.1.1 Project environments During this project, we also have two remote residents. In the following sections we are explaining the local lab configuration at the ITSO, as well as the remote Ghufran and Jason environments.

ITSO environment In our ITSO Austin environment we have built two TMR environments, one for a new install of IBM Tivoli Monitoring 5.1 and the other for an upgrade.

Region tividc11-region - IBM Tivoli Monitoring 5.1 new installation We have one Tivoli management region server, and one gateway at each side of the network. Table 4-1 shows the configuration of the lab equipment. Table 4-1 Lab configuration

72

Host name

TME function

OS level

TIVIDC11

Tivoli management region server/gateway

AIX Version 4.3.3

TIVIDC12

TEC Server/RIM host

AIX Version 4.3.3

ITWIN1

Gateway

Windows NT Server 4.0 Sp6a

TIVIDC13

ManagedNode/RIM host/ TEDW Server

Windows NT Server 4.0 Sp6a

TIVIDC14

Endpoint

Windows 2000 Advanced Server Sp1

TIVIDC15

Endpoint

Windows 2000 Advanced Server Sp1

ITSO21

Endpoint

Sun Solaris 5.7

ITSO27

Endpoint

Windows 2000 Server Sp1

ITSODEV1

Endpoint

AIX 5.1

ITSODEV3

Endpoint

AIX 4.3.3

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Host name

TME function

OS level

ITSODEV4

Web Server

AIX 4.3.3

ITSOTIV1

Endpoint

Windows 2000 Professional SP 1

RHLINUX7

Endpoint

Red Hat Linux 7.1

VMLINUX5

Endpoint

SuSE Linux 7.0 (z/OS) Kernel 2.2.16

Figure 4-1 illustrates the logical layout of tividc11-region.

TMR Server / Gateway AIX 4.3.3 - tividc11

Endpoint W2K ADV tividc14

Endpoint W2K Pro itsotiv1

Endpoint W2K ADV tividc15

Endpoint Solaris 7 itso21

TEC Server AIX 4.3.3 - tividc12

Gateway Windows NT Server - itwin1

Endpoint AIX 5.1 itsodev1

TEDW Server Windows NT Server tividc13

Web Health Console Apache WebServer AIX 4.3.3 - itsodev4

Endpoint Endpoint Red Hat Linux 7.1 SuSE Linux 7.0 (z/OS) vmlinux5 rhlinux71

Endpoint AIX 4.3.3 itsodev3

Figure 4-1 Lab environment tividc11-region

Region itsodev-region - IBM Tivoli Monitoring 5.1 upgrade In itsodev-region, we have one Tivoli management region server with the gateway functionality, as shown in Table 4-2 on page 74.

Chapter 4. Installation and migration

73

Table 4-2 Itsodev region Host name

TME function

OS level

ITSODEV2

Tivoli management region server/gateway

AIX 4.3.3

ITSODEV4

Endpoint

AIX 4.3.3

ITW2K

Endpoint

Windows 2000 Professional SP 1

ITWIN1

Endpoint

Windows NT Server 4.0 Sp6a

Remote Ghufran region In region G, we have one Tivoli management region server with the gateway functionality, one TEC Server and one TBSM Database Server, and the Application Server and Console Server, as shown in Table 4-3. Table 4-3 Ghufran region Host name

TME function

OS level

GEESUN

Tivoli management region server/TEC Server/ gateway/endpoint

Solaris 8

TURBO

Endpoint

Windows 2000 Professional Sp1

B-PC1

TBSM Database Server, Application Server, and Console Server

Windows 2000 Server Sp1

Remote Jason region In region J, we have one Tivoli management region server, TEC Server, and gateway, as shown in Table 4-4. Table 4-4 Jason environment

74

Host name

TME function

OS level

CERTTMR

Tivoli management region server/TEC Server

AIX 4.33

CERTGATE

Gateway

Windows 2000 Server

CERTWEB

Web server

Windows 2000 Server

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Note: Since the itsodev-region is only used in the upgrade section, when we refer to ITSO environment it means tividc11-region.

4.1.2 Framework requirements IBM Tivoli Monitoring 5.1 requires Tivoli Management Framework (TMF) Version 3.7B or later. If you are going to be using the product across a firewall, the minimum version of TMF is 3.7.1; refer to IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569. If you are running TMF Version 3.7B, then you must install the 3.7-TMF-0010 Framework patch to enable the full support of Windows 2000. To enable Linux support, install patches 3.6.1-TMF-0034, 3.6.1-TMF-0062, 3.7-TMF-0021, 3.7.1-TMF-0009, and 3.7.1-TMF-0037. Refer to the Tivoli Framework 3.7.1 Installation Guide, GC32-0395, for further information on installing patches. In the ITSO environment, we have TMF Version 3.7.1 with patches 3.6.1-TMF-0034, 3.6.1-TMF-0062, 3.7-TMF-0021, 3.7.1-TMF-0009, 3.7.1-TMF-0037, 3.7.1-TMF-0058, 3.7.1-TMF-0059, 3.7.1-TMF-0060, and 3.7.1-TMF-0067 installed on the Tivoli management region servers and ManagedNodes/gateways.

4.1.3 Software requirements There are additional software components that need to be installed in the environment for IBM Tivoli Monitoring 5.1 to function correctly at both the gateway level and the endpoint. Table 4-5 lists the different monitoring functions and the prerequisite software that is required. Table 4-5 Prerequisite software Function

Required software

Endpoints running on UNIX or Linux

Java Runtime Environment 1.3.0. JRE 1.3.1-01 must be installed on Solaris endpoints.

Endpoints running on Windows NT 4.0

WMI must be installed on the Windows NT. The minimum version is 1.1; however, 1.5 is recommended.

Chapter 4. Installation and migration

75

Function

Required software

Data Logging facility on Windows NT endpoints

The ODBC driver for MS Access 2000 must be installed on the endpoint. If the endpoint does not have MS Access 2000 installed, run the mdac_typ.exe file that is provided with MS Data Access Components Version 2.1 at: http://www.microsoft.com/data/

Sending event to TEC

TEC Server Version 3.7 plus patch 3.7-TEC-0004. If you want to send secure events, the TEC Adapter Configuration Facility must be installed on the gateways.

Sending events to Tivoli Business Systems Manager

IBM Tivoli Monitoring 5.1 must be installed on the gateway. Java Runtime Environment 1.3.0 must be installed on the gateway. The Tivoli Business Systems Manager patches 1.5-BSM-0010 and 1.5-BSM-0016 should be installed on the TBSM server. The adapter cannot be installed on HP-UX gateways.

Use of Gathering Historical Data on Tier1 Tivoli management region server

Patch 3.7-TMF-0035 must be installed if the Tier1 Tivoli management region server is connected to Tier2 endpoints.

Using Software Installation Services (SIS) for installing the product components

SIS client and SIS depot Version 3.7 with the following patches: 3.7-SISCLNT-0003 3.7-SISDEPOT-0003.

Systems using the Web Health Console product component

Netscape 6.x or Internet Explorer 6.x must be installed on the system.

Note: The JRE version on the endpoints do not necessarily have to be the JRE Version 1.3.0 provided by IBM.

4.1.4 Migration The following sections review migration.

76

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Tivoli Distributed Monitoring (Classic Edition) family IBM Tivoli Monitoring 5.1 allows you to monitor availability and performance status of resources on your systems to identify bottlenecks and potential resource problems, based on a set of resource models designed to detect runtime bottlenecks and other potential problems and to automatically recover from critical situations, eliminating the need for system administrators to manually scan through extensive performance data. Tivoli Distributed Monitoring (Classic Edition) 3.7, basically verifies the retrieved values from scripts or commands, matching those values with thresholds, and finally, based on user’s customization, triggers some response (Tivoli Enterprise Console events, e-mail, notices, and so on). Tivoli Distributed Monitoring (Classic Edition) 3.7 leaves to you the correlation and analysis of the problem’s root cause. The IBM Tivoli Monitoring 5.1 product bundle provides IBM Tivoli Monitoring 5.1 and Tivoli Distributed Monitoring (Classic Edition) 3.7 coexistence, so you can use both and plan their migration. You should take the following two aspects into consideration: Coexistence

The two family products can coexist, as they have two different implementations at the server, gateways, and endpoints. The installation of IBM Tivoli Monitoring 5.1 does not change the Tivoli Distributed Monitoring (Classic Edition) 3.7 environment or configuration, even when working in compatibility mode.

Migration

To make the migration from Tivoli Distributed Monitoring (Classic Edition) 3.7 to IBM Tivoli Monitoring 5.1 as easy as possible and to save your investment on custom scripts and home-developed monitoring collections, IBM Tivoli Monitoring 5.1 runs in compatibility mode, the new working mode that allows IBM Tivoli Monitoring 5.1 users to use Tivoli Distributed Monitoring (Classic Edition) 3.7 monitor collections and custom scripts within resource models.

For more information about migration and compatibility mode check Chapter 5, “Migrating Classic Edition monitors” on page 111.

Tivoli Distributed Monitoring (Advanced Edition) family The Distributed Monitoring (Advanced Edition) must be upgraded to IBM Tivoli Monitoring 5.1. Table 4-6 on page 78 shows the different versions of Distributed Monitoring (Advanced Edition).

Chapter 4. Installation and migration

77

Table 4-6 Advanced Edition version upgrade path Current product

Tivoli Manager for NT 3.6.2 (see note below)

Upgrade path

Tivoli Distributed Monitoring for Windows 3.7 patch 003

Tivoli Distributed Monitoring for Windows 3.7 patch 003

Tivoli Distributed Monitoring (Advanced Edition) 4.1

IBM Tivoli Monitoring 5.1

Tivoli Distributed Monitoring (Advanced Edition) 4.1

IBM Tivoli Monitoring 5.1

Tivoli Distributed Monitoring (Advanced Edition) 4.1

IBM Tivoli Monitoring 5.1

Products within the Distributed Monitoring (Advanced Edition) family must be upgraded to Version 5.1. None of the products of this family can coexist in the same Tivoli management region. Important: In our upgraded environment from Tivoli Manager for NT 3.6.2 we found two errors conditions during the upgrade from Tivoli Distributed Monitoring (Advanced Edition) 4.1 to IBM Tivoli Monitoring 5.1. The first error was a missing library, libatrc.a; the second error was an oserv general failure. Due to these problems, if you have a Tivoli Manager for NT 3.6.2 environment or an upgraded environment from Tivoli Manager for NT 3.6.2, we suggest that you uninstall the product, then install IBM Tivoli Monitoring 5.1 as a new product. Before you act on our recommendation we suggest that you talk to Tivoli Support to determine the valid migration path that is now available at the time of your migration. Since writing the book, development has produced a patch 4.1-DMA-0006 that reports to fix the problem we saw.

Compatibility between versions Table 4-7 on page 79 shows the backward compatibility between the baroc files.

78

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Table 4-7 Baroc files compatibility IBM Tivoli Monitoring 5.1 BAROC files

Tivoli Distributed Monitoring (Advanced Edition) 4.1 BAROC files

Tivoli Distributed Monitoring for Windows 3.7 patch 003 BAROC files

IBM Tivoli Monitoring 5.1

Compatible

Tivoli Distributed Monitoring (Advanced Edition) 4.1

Compatible

Compatible

Compatible

Tivoli Distributed Monitoring for Windows 3.7 patch 003

Compatible

Compatible

Compatible

The BAROC file available with IBM Tivoli Monitoring 5.1 can also be used with Tivoli Distributed Monitoring (Advanced Edition) 4.1 or with Tivoli Distributed Monitoring for Windows 3.7 patch 003. The BAROC file available with Tivoli Distributed Monitoring (Advanced Edition) 4.1 can also be used with Tivoli Distributed Monitoring for Windows 3.7 patch 003. The BAROC file available with Tivoli Distributed Monitoring for Windows 3.7 patch 003 can also be used with Tivoli Distributed Monitoring (Advanced Edition) 4.1. Table 4-8 shows the backward compatibility between the resource models. Table 4-8 Resource models compatibility

IBM Tivoli Monitoring 5.1

Resource models created in IBM Tivoli Monitoring 5.1

Resource models created in Tivoli Distributed Monitoring (Advanced Edition) 4.1

Resource models created in Tivoli Distributed Monitoring for Windows 3.7

Compatible

Compatible

Compatible

Chapter 4. Installation and migration

79

Resource models created in IBM Tivoli Monitoring 5.1

Tivoli Distributed Monitoring (Advanced Edition) 4.1

Resource models created in Tivoli Distributed Monitoring (Advanced Edition) 4.1

Resource models created in Tivoli Distributed Monitoring for Windows 3.7

Compatible

Compatible

Tivoli Distributed Monitoring for Windows 3.7

Compatible

Relation between Resource models and Workbench Windows and UNIX resource models created using IBM Tivoli Monitoring 5.1 Workbench can only be used with IBM Tivoli Monitoring 5.1. Windows and UNIX resource models created using Tivoli Distributed Monitoring (Advanced Edition) 4.1 Workbench can be used also with IBM Tivoli Monitoring 5.1. UNIX resource models need to be rebuilt using IBM Tivoli Monitoring 5.1 Workbench. For further information see 4.2.2, “Upgrade to IBM Tivoli Monitoring 5.1” on page 87. Windows resource models created using Tivoli Distributed Monitoring for Windows 3.7 Workbench can be used also with IBM Tivoli Monitoring 5.1, or with Tivoli Distributed Monitoring (Advanced Edition) 4.1.

4.2 Installation and upgrade In the following sections we describe the installation and upgrade of IBM Tivoli Monitoring 5.1 and also the installation of JRE on endpoints.

4.2.1 Fresh Installation of IBM Tivoli Monitoring 5.1 IBM Tivoli Monitoring 5.1 can be installed using various methods. It can be installed using the desktop, command line, or Tivoli Software Installation Services (SIS). We used the desktop in our environment. If you would prefer to use the command line or SIS, refer to the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569, for more information.

80

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The following steps were done to install the IBM Tivoli Monitoring 5.1 product in ITSO environment: 򐂰 Back up the object database and also the Tivoli file system. This can be done through the Tivoli desktop by selecting Desktop -> Backup or by issuing the wbkupdb command. It is suggested that you back up the entire Tivoli file system of the node that the product will be installed on. 򐂰 Install the IBM Tivoli Monitoring 5.1 product on the Tivoli management region server and gateways. Tip: If any problems occurs during the installation process, they will be registered in the file /tmp/tivoli.cinstall (UNIX) or %DBDIR%/tmp/tivoli.cinstall (Windows). In our ITSO environment we installed the product doing the following steps: 1. From the Tivoli desktop, select Desktop -> Install -> Install Product. 2. The Install Product dialog box shows three products that are available to install, as shown in Figure 4-2 on page 82.

Chapter 4. Installation and migration

81

Figure 4-2 Products to install

3. Select IBM Tivoli Monitoring 5.1.0, then select your Tivoli management region server and the gateways that you want to install IBM Tivoli Monitoring 5.1 on. Select Install and follow the normal installation dialog boxes. Tip: If you are installing or upgrading IBM Tivoli Monitoring 5.1 on a Windows TMR Server environment, you must do the additional procedures below to create the IBM Tivoli Monitoring 5.1 Task Libraries.

82

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Additional procedures for Windows TMR Environment In the Windows TMR box, set the Tivoli environment variables and enter in bash mode. Run the script after_install_win.sh; the script contents can be viewed in Example 4-1. Example 4-1 Contents of script after_install_win.sh IRO=`wlookup InterRegion` IROname=`idlattr -t -g $IRO name string` IROname=`eval echo $IROname` DefaultMw2kRegionName=”TivoliDefaultMw2kRegion-$IROname" wtll -r -p $DefaultMw2kRegionName -P $BINDIR/tools/cat $BINDIR/TME/Tmw2k/tmnt.tll wtll -r -p $DefaultMw2kRegionName -P $BINDIR/tools/cat $BINDIR/TME/Tmw2k/tmnt_util.tll

This completes the installation of the base IBM Tivoli Monitoring 5.1 code. During the install, a default policy region is created in the top level policy region, as shown in Figure 4-3 on page 84. The policy region is called TivoliDefaultMw2kRegion and is appended by the Tivoli management region region name. To access this policy region, select Desktop -> TMR Connections -> Top Level Policy Region.

Chapter 4. Installation and migration

83

Figure 4-3 IBM Tivoli Monitoring 5.1 policy region

The policy region in our ITSO environment is called TivoliDefaultMw2kRegiontividc11-region. This policy region contains a profile manager and two task libraries, as shown in Figure 4-4.

Figure 4-4 Profile manager and task library

84

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The Task Library Tivoli Distributed Monitoring (Advanced Edition) Tasks window contains tasks, as shown in Figure 4-5.

Figure 4-5 Tivoli Distributed Monitoring (Advanced Edition) Tasks window

The tasks are described below. DMCollectEpEnv

This task collects information about the environment at the endpoint. The data collected is written to a file using the Execute Task dialog (Save to File option). This task does not accept arguments.

DMCollectEpLog

This task collects (in a tar file created at the endpoint) all the endpoint logs and information about the size and dates of the binaries as well as the current and universal time the logs were created. It collects logs from $LCF_DATDIR. This task accepts the name of the tar file as an argument.

DMCollectMnLog

This task collects (in a tar file created at the ManagedNode in the $DBDIR directory) all the ManagedNode logs and traces including event logs for Windows platforms. This task accepts the name of the tar file as an argument.

Chapter 4. Installation and migration

85

DMEndpointUninstall

This task removes all the IBM Tivoli Monitoring 5.1 files that were downloaded to the endpoint with the Tmw2k profile.

DMFixCliCmdsOn41MNs This task fixes problems related to commands wdmcmd and wdmlseng issued from a server with IBM Tivoli Monitoring 5.1 through a ManagedNode with Tivoli Distributed Monitoring (Advanced Edition) 4.1 fail-on-endpoints with the previous code. DMLinkJre

This task is used for linking the IBM Tivoli Monitoring 5.1 product on the endpoint to the JRE runtime directory. This task is detailed further in 4.2.4, “Installing JRE” on page 99.

DMRebootUninstall

This task ensures that IBM Tivoli Monitoring 5.1 will not be restarted at reboot time on the endpoints unless you distribute Tmw2kProfile again.

DMRemoveLinkJre

This task is used for removing the link between the IBM Tivoli Monitoring 5.1 product and the JRE runtime directory on the endpoint.

For more information about the tasks DMCollectEpEnv, DMCollectEpLog, and DMCollectMnLog, refer to 14.5.3, “Serviceability tasks” on page 499.

Figure 4-6 IBM Tivoli Monitoring Utility Tasks window

86

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The Task Library: IBM Tivoli Monitoring Utility Tasks window contains the following tasks, as shown in Figure 4-6 on page 86: dm_mn_send_email

This task is used for sending an e-mail in response an event. It requires SMTP configuration. For further information look at Appendix C “Configuring for SMTP E-Mail” of the Tivoli Framework 3.7.1 Installation Guide, GC32-0395.

dm_mn_send_notice

This task is used for sending a notice in response to an event.

For more information about the tasks dm_mn_send_email and dm_mn_send_notice, look at 9.1.1, “Creating and customizing profiles” on page 206.

4.2.2 Upgrade to IBM Tivoli Monitoring 5.1 You can only upgrade from Tivoli Distributed Monitoring (Advanced Edition) 4.1 to IBM Tivoli Monitoring 5.1. For previous versions you must upgrade to Tivoli Distributed Monitoring (Advanced Edition) 4.1 first. To upgrade you can use the Tivoli desktop, selecting Install -> Install Patch -> IBM Tivoli Monitoring Upgrade, Version 4.1 (DMA) to 5.1.0 or using the command line, wpatch. To upgrade, follow the procedure below: 1. Use the wdmcmd –stop -e endpoint_label command to stop the IBM Tivoli Monitoring 5.1 processes on all endpoints. 2. Use the wdmmn -stop all command to stop the IBM Tivoli Monitoring 5.1 processes on all ManagedNodes. This command stops the process tmnt_hb_eng and tmnt_task_eng. 3. Upgrade the server component at the Tivoli management region server, then upgrade the gateways. Note: All existing resource models in the Tivoli management region server will be upgraded to be compatible with the new software. 4. For UNIX and Linux endpoints, redistribute all profiles. This redistribution will automatically install the upgraded IBM Tivoli Monitoring 5.1 endpoint component and will start the resource models. 5. For Windows endpoints, the engine binaries will be upgraded by issuing the wdmcmd –restart -e endpoint_label command. After you have stopped the engine, wait a minimum of 15 minutes before issuing the command.

Chapter 4. Installation and migration

87

If you have the Tivoli Business Systems Manager Adapter or the Tivoli Decision Support components installed, then install the appropriate upgrade. Note: If you have the TDS Configuration component already installed, the appropriate upgrade is called IBM Tivoli Monitoring – Gathering Historical Data Component,Upgrade 4.1 (DMA) to 5.1.0. We did the upgrade procedures in itsodev-region. Example 4-2 is based on this region. This region initially had the following Tivoli management products installed: 򐂰 Framework 3.7.1 - 3.7.1-TMF-0067 򐂰 Tivoli Distributed Monitoring (Advanced Edition) 4.1, upgraded from Tivoli Distributed Monitoring for Windows 3.7 Based on 4.1.4, “Migration” on page 76, we migrate this environment to IBM Tivoli Monitoring 5.1. In addition, we have added two custom resource models in Tivoli Distributed Monitoring (Advanced Edition) 4.1. Example 4-2 Upgrading to IBM Tivoli Monitoring 5.1 # Adding custom resource models in Tivoli Distributed Monitoring (Advanced Edition) 4.1 itsodev2:/rm_custom>wdmrm -add DMXUsrCpu.tar Tivoli Distributed Monitoring - Adding new resource model Copying DMXUsrCpu.cat msgfile ... Copying DMXUsrCpu.cat zipfile ...

Tivoli Distributed Monitoring - Resource model utility Parsing configuration file DMXUsrCpu.conf ... Configuration file successfully parsed. Checking for event redefinition... Starting resource DMXUsrCpu registration ... the resource DMXUsrCpu has been successfully stored. Registration completed. Installation completed. itsodev2:/rm_custom>wdmrm -add TMW_UsrPhysicalDiskModel.tar Tivoli Distributed Monitoring - Adding new resource model Copying TMW_UsrPhysicalDiskModel.cat msgfile ... Copying TMW_UsrPhysicalDiskModel.cat zipfile ...

88

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Tivoli Distributed Monitoring - Resource model utility Parsing configuration file TMW_UsrPhysicalDiskModel.conf ... Configuration file successfully parsed. Checking for event redefinition... Starting resource TMW_UsrPhysica DiskModel registration ... the resource TMW_UsrPhysicalDiskModel has been successfully stored. Registration completed. Installation completed. itsodev2:/rm_custom>wdmrm -list Resource -> DMXUsrCpu NLS name : CPU product_id : none major_version : 1 minor_version : 0 platform : aix4-r1\hpux10\linux-ix86\linux-s390\solaris2 message catalog : DMXUsrCpu zip file : DMXUsrCpu.zip Resource -> TMW_UsrPhysicalDiskModel NLS name : Physical Disk product_id : none major_version : 1 minor_version : 0 platform : w32-ix86 message catalog : TMW_UsrPhysicalDiskModel zip file : TMW_UsrPhysicalDiskModel.zip # Checking running engine itsodev2:/>wdmlseng -e itsodev4 -verbose Forwarding the request to the engine...

The following profiles are running: unix#itsodev-region DMXFileSystem: Running LowPercInodesAvail 100 % FragmentedFileSystem 100 % LowKAvail 100 % tmw2k.custom_unix.pf#itsodev-region DMXUsrCpu: Running UsrHigh_SysCPUUsage 100 %

Chapter 4. Installation and migration

89

UsrLow_IdleCPUUsage 100 % itsodev2:/>wdmlseng -e itwin1 -verbose Forwarding the request to the engine... The following profiles are running: Tmw2k.custom_win.pf#itsodev-region TMW_UsrPhysicalDiskModel :Running TMW_UsrPhysicalPossibleFrag TMW_UsrHighPhysicalDiskWriteBytesSec TMW_UsrHighPhysicalPercentDiskTime TMW_UsrHighPhysicalDiskXferRate TMW_UsrSlowPhysicalDrive TMW_UsrHighPhysicalDiskReadBytesSec

100 100 100 100 100 100

% % % % % %

# Stopping the endpoint / managed node engine itsodev2:/>wdmcmd -stop -e itwin1 itsodev4 itsotiv-a Stopping engine on endpoint 1765307597.4.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1765307597.4.517+#TMF_Endpoint::Endpoint# Stopping engine on endpoint 1765307597.8.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1765307597.8.517+#TMF_Endpoint::Endpoint# Stopping engine on endpoint 1765307597.6.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1765307597.6.517+#TMF_Endpoint::Endpoint# itsodev2:/>wdmmn -stop all Stopping task_engines on managed nodes..... Stopped task_engine on Managed Node:1765307597.1.348 #TMF_ManagedNode: :Managed_Node# # Upgrading using command wpatch itsodev2:/>wpatch -c /itm_upg -i DM51UPG.IND itsodev2 Checking patch dependencies... Product TMNT_3.6.2 is already installed as needed. Dependency check completed. Inspecting node itsodev2... Installing Patch: IBM Tivoli Monitoring Upgrade, Version 4.1 (DM Advanced Edition) to 5.1.0

Unless you cancel, the following operations will be executed: For the machines in the independent class: hosts: itsodev2 itsodev2 already has the Message Catalogs installed (from itsodev2). need to copy the LCF (generic) to: itsodev2:/Tivoli/bin/lcf_bundle

90

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

need to copy the LCFNEW (generic) to: itsodev2:/Tivoli/bin/lcf_bundle.40 For the machines in the aix4-r1 class: hosts: itsodev2 itsodev2 already has the Binaries installed (from itsodev2). need to copy the ALIDB (aix4-r1) to: itsodev2:/Tivoli/itsodev2.db need to copy the MAN (aix4-r1) to: itsodev2:/Tivoli/man/aix4-r1 need to copy the LIB (aix4-r1) to: itsodev2:/Tivoli/lib/aix4-r1 Continue([y]/n)?y Creating patch installation description object...Created. Executing queued operation(s) Distributing machine independent LCF Imagen (old version) --> itsodev2 ...........Patch install completed successfully. Completed. Distributing architecture specific Server Database --> itsodev2 .................................................Patch install completed successfully. Completed. Distributing architecture specific Man Pages --> itsodev2 . Completed. Distributing architecture specific Libraries --> itsodev2 . Completed. Distributing machine independent LCF Images (new version) --> itsodev2 .....................Patch install completed successfully. Completed. Registering patch installation attributes...Registered. Finished patch installation.

As we have added custom resource models in Tivoli Distributed Monitoring (Advanced Edition) 4.1, now we need to upgrade the custom resource models to be compatible with the IBM Tivoli Monitoring 5.1 resource models. The following example is based on the UNIX custom resource model DMXUsrCpu.

Chapter 4. Installation and migration

91

We have done the following steps using IBM Tivoli Monitoring 5.1 Workbench: 1. Open the IBM Tivoli Monitoring 5.1 resource model that corresponds to the resource model that you modified, and extract all the dependencies belonging to the resource model. In our case, we have customized the DMXCpu resource model, so we extracted the dependencies from the DMXCpu resource model, as shown in Figure 4-7 on page 93. Tip: We create a folder called DMXCpu_dep, and subfolders for each dependency because each dependency has specific binaries.

92

05/02/2002 05/02/2002 05/02/2002 05/02/2002 05/02/2002

12:10p 12:10p 12:11p 12:11p 12:11p





aix4-r1 All hpux10 linux-ix86 linux-s390

05/02/2002

12:11p

solaris2

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 4-7 Extracting dependencies from resource model DMXCPU

2. Open the customized resource model and remove all the dependencies. We opened the custom resource model DMXUsrCpu, then removed all its dependencies, as shown in Figure 4-8 on page 94.

Chapter 4. Installation and migration

93

Figure 4-8 Removing dependencies from DMXUsrCpu

3. Save the custom resource model without dependencies. 4. Add the dependencies in the custom resource model by importing the dependencies that you extracted (Figure 4-9 on page 95).

94

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 4-9 Adding dependencies in DMXUsrCpu

5. Save the custom resource model with the dependencies. 6. Regenerate the resource model tar file (Figure 4-10 on page 96).

Chapter 4. Installation and migration

95

Figure 4-10 Building the DMXUsrCpu with the updated dependencies

7. Uninstall the old resource model and install the new tar file in the Tivoli management region, using the commands shown in Example 4-3. Example 4-3 Removing and adding a custom resource model # Removing custom Resource Model itsodev2:/>wdmrm -remove DMXUsrCpu Tivoli Distributed Monitoring - Removing resource model

96

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Removing resource model DMXUsrCpu ... Tivoli Distributed Monitoring - Resource model utility Starting removing of the resource DMXUsrCpu ... deleting resource DMXUsrCpu ... the resource DMXUsrCpu has been successfully deleted. cleaning of profiles ... deleting entry from profile tmw2k.custom_unix.pf... Removing completed. Removing files DMXUsrCpu.cat and DMXUsrCpu.zip ... Deinstallation completed. # Adding custom resource model itsodev2:/rm_custom>wdmrm -add DMXUsrCpu.tar Tivoli Distributed Monitoring - Adding new resource model Copying DMXUsrCpu.cat msgfile ... Copying DMXUsrCpu.cat zipfile ...

Tivoli Distributed Monitoring - Resource model utility Parsing configuration file DMXUsrCpu.conf ... Configuration file successfully parsed. Checking for event redefinition... Starting resource DMXUsrCpu registration ... the resource DMXUsrCpu has been successfully stored. Registration completed. Installation completed.

8. Add the updated resource model to the profile it belonged to, as shown in Figure 4-11 on page 98.

Chapter 4. Installation and migration

97

Figure 4-11 Adding the custom resource model to the profile

4.2.3 Installing prerequisite software on endpoints IBM Tivoli Monitoring 5.1 requires various software components to be installed in the environment. If you plan to monitor any UNIX servers, that is, AIX, Solaris, HP-UX, or Linux, Java needs to be installed on the UNIX endpoint; refer to Table 4-5 on page 75 for the software requirements. On Windows, the WMI, which is based on CIM standards, is the provider agent. On UNIX and Linux platforms, the information provider agent is incorporated in the product based on CIM specifications, which is TouchPoint. So, when we distribute a IBM Tivoli Monitoring 5.1 profile to a UNIX endpoint, this collection agent is downloaded to the endpoint. The agent, however, needs to know where the JRE is located on the endpoint. To achieve this, we link the endpoint to the top level directory of the JRE. In the following sections we detail the installation of Java and the other prerequisite software in our ITSO environment. Note: In our ITSO lab environment, we installed IBM JRE Version 1.3.0 on the UNIX endpoints.

98

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

If you are going to use any disk or network card related resource models on Windows servers, you need to do the following: 1. Run diskperf -y. 2. Enable the SNMP protocol.

4.2.4 Installing JRE You can install JRE using three available installation methods: 򐂰 JRE 1.3.0 is available on the product CD for installation using the Tivoli Software Installation Service (SIS). 򐂰 JRE 1.3.0 is also available on the product CD in compressed format, for manual installation or installation with the wdmdistrib –J command. 򐂰 On UNIX/Linux endpoints, if the endpoint already has an appropriate version of JRE installed, you need only link the product component to the existing JRE using a task provided with the product.

Installation using wdmdistrib We have installed JRE using the wdmdistrib command in the endpoint itso21. Example 4-4 shows the command syntax. Example 4-4 JRE installation using wdmdistrib tividc11:/>wdmdistrib -p linux -J /cdrom/tools/jre vmlinux5 AMW0162I - Operation successfully submitted. Distribution ID is 1931340152.34

Where: -p = Tmw2kProfile name -J = Path to the JRE directory on the Tools CD The JRE distribution through wdmdistrib can also be executed on Windows endpoint targets. Note: When you use this installation method, it extracts all the JRE code under the path $LCF_BINDIR/../JRE/DMAE on the endpoint target. Using this installation method, we did not have to run the task DMLinkJre. For more information about profile distribution, refer to 9.1.2, “Profile distribution” on page 220.

Chapter 4. Installation and migration

99

Manual installation Following is the procedure for manual installation. 1. Copy the jre13.tar.gz file from the directory on the Tools CD that applies to the operating system where JRE is to run, to the directory where you want to install JRE: – AIX: JRE/aix4-r1. – Linux: JRE/linux-ix86. – Linux S/390: JRE/linux-s390. – Solaris: JRE/solaris2. – HP-UX: Refer to the installation instructions, which can be found at HP’s Web site (http://www.hp.com), on the page entitled HP-UX Runtime Environment for the Java 2 Platform, Version 1.3.0. – Windows: JRE/w32-ix86. 2. From the directory where you copied the jre13.tar.gz file, issue the command: gzip –dc jre13.tar.gz | tar xvf –

3. If JRE is manually installed on a UNIX/Linux endpoint, you need to run the DMLinkJre task.

Linking JRE through task DMLinkJre On UNIX and Linux platforms, the information collection agent is incorporated in the product, based on CIM specifications. So, when we distribute a IBM Tivoli Monitoring 5.1 profile to a UNIX endpoint, this collection agent is downloaded to the endpoint. The agent, however, needs to know where the JRE is located on the endpoint. To achieve this, we link the endpoint to the top level directory of the JRE. Since we manually installed JRE on the itsodev1, we had to execute the task on this target. 1. Select Desktop -> TMR Connections -> Top Level Policy Region. 2. Open the TivoliDefaultMw2kRegion-tividc11-region policy region and then open the Tivoli Distributed Monitoring (Advanced Edition) Tasks Task Library and execute the DMLinkJre task by right-clicking the icon and selecting Execute Task. This will display the DMLinkJre task options panel, as shown in Figure 4-12 on page 101. From this panel, select the endpoint that the task will be executed on and optionally select Display on Desktop from the Output Destination box.

100

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 4-12 Task options panel

a. Select Execute. b. The next dialog box that is displayed is the Configure Task Arguments box, as shown in Figure 4-13 on page 102. There is one parameter that needs to be entered here, the fully-qualified path to where you have installed the JRE. In our environment, we installed the JRE in /usr/java130. 3. Select Set & Execute to run the task.

Chapter 4. Installation and migration

101

Figure 4-13 DMLinkJre configure task arguments

Note: To verify that the link has been created, go to the endpoint and look at the lcf directory /opt/Tivoli/lcf/bin/aix4-r1/JRE/. There should be a link (DMAE -> /usr/java130) created.

4.3 Uninstalling the product To uninstall the product you should follow this sequence: 1. Uninstall the endpoint components. 2. Uninstall from the server and gateways.

4.3.1 Uninstalling IBM Tivoli Monitoring 5.1 from endpoints There are two ways to remove the IBM Tivoli Monitoring 5.1 product from an endpoint. The following sections explain both methods.

Shell script dm_endpoint_uninstall IBM Tivoli Monitoring 5.1 provides a script to remove the binaries from endpoints. The script is named dm_endpoint_uninstall.sh and is stored in $BINDIR/TME/Tmw2k.

102

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The dm_endpoint_uninstall.sh script performs the following steps: 򐂰 Stops the Tmw2k application, if it is running. 򐂰 Removes all the IBM Tivoli Monitoring 5.1 files that were downloaded to endpoints with the Tmw2k profile from those endpoints. 򐂰 Removes the boot_method tmnt_boot from the endpoints. 򐂰 Where appropriate, removes the application keys from the Windows registry. To uninstall the endpoint components, you should use the script with the following parameter: dm_endpoint_uninstall.sh endpoint_label1 endpoint_label2

...

Where endpoint_label is the label of the endpoint. If more than one endpoint label is specified, they should be separated by a space. In our ITSO environment we uninstalled the product from the endpoint itso21. Example 4-2 shows the script stout. Example 4-5 Uninstalling using the script dm_endpoint_uninstall tividc11:/tivoli/bin/aix4-r1/TME/Tmw2k>./dm_endpoint_uninstall.sh itso21 [ itso21 ] Removing endpoint itso21 from the Endpoint's cache on the Managed Node... Removing boot method for endpoint itso21 ... ############################################################################ Task Name: DMRebootUninstall Task Endpoint: itso21 (Endpoint) Return Code: 1 ------Standard Output-----------Standard Error Output-----############################################################################ Removing engine code for endpoint itso21 ... ############################################################################ Task Name: DMEndpointUninstall Task Endpoint: itso21 (Endpoint) Return Code: 0 ------Standard Output-----Stopping Distributed Monitoring (Advanced Edition) engine ... Removing out-of-cache Distributed Monitoring (Advanced Edition) engine files. Removing out-of-cache Distributed Monitoring (Advanced Edition) engine log an race files... Removing in-cache Distributed Monitoring (Advanced Edition) edition files ... Removing Distributed Monitoring (Advanced Edition) message catalogs ... ------Standard Error Output-----############################################################################

Chapter 4. Installation and migration

103

The following commands can be used to check if the uninstallation was successful: wep boot_method list "" `wlookup -r Endpoint endpoint_label` wdmlseng -e endpoint_label -verbose

The script dm_endpoint_uninstall runs both tasks DMEndpointUninstall and DMRebootUninstall on the target endpoint. Note: The uninstallation script does not remove the system data source tmw2k created on Windows endpoints. This is a known problem and it is registered as defect 27472.

Uninstallation tasks You can use the task DMEndpointUninstall to remove all the IBM Tivoli Monitoring 5.1 files that were downloaded to endpoints. If you decide to use this task, you must also execute the task DMRebootUninstall, otherwise the boot_method tmnt_boot will not be removed and the process Tmw2k will try to initialize during the next endpoint restart. Important: We suggest the use of the script dm_endpoint_uninstall.sh to uninstall IBM Tivoli Monitoring 5.1 components from endpoints.

4.3.2 Uninstalling IBM Tivoli Monitoring 5.1 from the server/gateway To uninstall IBM Tivoli Monitoring 5.1 from servers or gateways, you can use the following command: wuninst tagname destination_target –rmfiles

Where tagname is one of the registered product tags for IBM Tivoli Monitoring 5.1 that is provided by Tivoli. 򐂰 DM_Advanced_Edition_TDS is used to indicate Tivoli Decision Support for Server Performance Prediction. 򐂰 DM_Advanced_Edition_TBSMA is used to indicate Tivoli Business Systems Manager Adapter. 򐂰 TMNT_3.6.2 is used to indicate IBM Tivoli Monitoring.

destination_target is the gateway or server from which you want to remove the product. If you specify a server, the product is uninstalled from all its gateways.

104

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

For more information about command line syntax and usage of the wuninst command, refer to the Tivoli Framework 3.7.1 Reference Manual, SC31-8434, and the Tivoli Management Framework Release Notes, which comes with the product. Tip: If you have IBM Tivoli Monitoring 5.1 installed on a ManagedNode (that is, an endpoint with the same label) the uninstall could fail. To bypass this issue, you should stop and start the endpoint before running the uninstall command.

4.4 Installing TBSM IBM Tivoli Monitoring 5.1 For the complete steps of installation and integration, refer to Chapter 12, “Tivoli Business Systems Manager integration” on page 407.

4.5 Installing Web Health Console This section explain the installation procedures for Web Health Console.

Web Health Console software requirements The Web Health Console runs on Netscape 6.2 (or later) and Internet Explorer 6.x. The Web Health Console uses WebSphere Application Server, Advanced Edition Single Server 4.0.2, which is installed as part of the Web Health Console install. Refer to the following Web address for the software prerequisites of WebSphere Application Server Advanced Edition Single Server 4.0.2: http://www-3.ibm.com/software/webservers/appserv/doc/v40/prereqs/aes_v402.htm

The Web Health Console installation requires at least 450 MB of temporary space.

Installation on Windows environment To install the Web Health Console on Windows, you should follow these steps: 1. Using Web Health Console disk 2, double-click the setupwin32.exe file. If the temporary directory does not have enough space, you could get an error. To avoid this, enter the following command: setupwin32 -is:tempdir TMPdir

Where TMPdir is the name of the temporary directory.

Chapter 4. Installation and migration

105

2. Follow the directions presented in the install dialogs. In particular, provide the directory name for the location on which you wish to install the Web Health Console Server. The directory name must contain no spaces. Provide the user name under which the Web Server will run. This user must have Act As Operating System access. Provide the password for the user. To install in silent mode using the command line, you must provide the following arguments: -silent -P base_install.installLocation=DirectoryName -W user_input.user=User Name -W user_input.password=Password

You can also include the above arguments in a file and pass the file to the launcher using the options switch -options options file.

Installation on UNIX environment To install the Web Health Console on UNIX: 1. Using Web Health Console disk 1, execute one of the following files depending on the UNIX platform you are running: – – – – –

setupaix.bin on AIX setuphp1020.bin on HP-UX10.2 setuphp11x.bin on HP-UX11.x setupsolarisSparc.bin on Sun Solaris setuplinux.bin on Linux

Note: If the temporary directory does not have enough space, you could get an error. To avoid this, enter the command (corresponding to your UNIX platform): setupaix -is:tempdir TMPdir

Where TMPdir is the name of the temporary directory. 2. Follow the directions presented in the install dialogs. In particular, provide the directory name for the location on which you wish to install the Web Health Console Server. To install in silent mode using the command line, you must provide the following arguments to the setup command indicated before: -silent -P base_install.installLocation=.Directory Name....

You can also include the above arguments into a file and pass the file to the launcher using the options switch -options options file.

106

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Note: The UNIX installation will install the IBM HTTP Server into a standard location regardless of the directory that you specify for the install. The rest of the installation will go to the directory that you have specified. The standard locations are: 򐂰 򐂰 򐂰 򐂰

AIX: /usr/HTTPServer Sun Solaris: /opt/IBMHTTPD Linux: /opt/IBMHTTPServer HP: /opt/HTTPServer

Troubleshooting the installation To troubleshoot the Web Health Console installation process, refer to IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569. In our environment, we installed Web Health Console on a Windows machine. Figure 4-14 shows the initial installation process.

Figure 4-14 Initial Web Health Console installation

Figure 4-15 on page 108 shows the license agreement required to install the product.

Chapter 4. Installation and migration

107

Figure 4-15 Web Health Console license agreement

Figure 4-16 shows the directory where Web Health Console will be installed.

Figure 4-16 Directory

Figure 4-17 on page 109 shows the required user and password window.

108

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 4-17 User and password settings

Figure 4-18 shows the products to be installed.

Figure 4-18 Web Health Console products to be installed

Figure 4-19 on page 110 shows the final installation process window.

Chapter 4. Installation and migration

109

Figure 4-19 Web Health Console final installation

4.6 Installing IBM Tivoli Monitoring Historical Data Gathering For the complete steps of installation and configuration, refer to 13.3.2, “Installing IBM Tivoli Monitoring Version 5.1 Data Gathering” on page 435.

110

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

5

Chapter 5.

Migrating Classic Edition monitors This chapter discusses and provides information for upgrading an already existing Distributed Monitor for Windows NT Version 3.7 environment to IBM Tivoli Monitoring 5.1. This chapter provides examples of converting Distributed Monitoring 3.7 Classic monitors using resource models that are based on the new IBM Tivoli Monitoring 5.1. This chapter covers the following: 򐂰 򐂰 򐂰 򐂰

Section 5.1, “Migration process” on page 112 Section 5.2, “Classic Edition monitor conversion example” on page 119 Section 5.3, “Custom numeric monitor conversion example” on page 137 Section 5.4, “Converting a CSL-based monitor” on page 147

© Copyright IBM Corp. 2002

111

5.1 Migration process IBM Tivoli Monitoring 5.1 provides a set of tools to help facilitate Tivoli Distributed Monitoring (Classic Edition) 3.7 users to migrate their monitoring solutions into a the IBM Tivoli Monitoring 5.1 environment. There is no one tool to completely automate the migration process because, the Tivoli Distributed Monitoring (Classic Edition) 3.7 and IBM Tivoli Monitoring 5.1 are totally different products. This chapter will describe some of the facilities that can be used to migrate Tivoli Distributed Monitoring (Classic Edition) 3.7 monitors to the new IBM Tivoli Monitoring 5.1 product. Note: Many of the principles regarding the Workbench in this chapter were discussed in detail in the Workbench chapter. A review of Chapter 10, “Workbench” on page 245, is recommended before reading this chapter.

5.1.1 Migration overview Figure 5-1 on page 113 is an overview of the migration process of converting Tivoli Distributed Monitoring (Classic Edition) 3.7 monitors to IBM Tivoli Monitoring 5.1 resource models. The following is a description of the six steps depicted in Figure 5-1 on page 113 as a possible migration strategy: 򐂰 Stage 1: This is how we start with a common configuration. This simple example assumes two probes (P1 and P2). 򐂰 Stage 2: Deploying IBM Tivoli Monitoring 5.1 to the server does not affect the custom probes. IBM Tivoli Monitoring 5.1 can run in parallel with Tivoli Distributed Monitoring (Classic Edition) 3.7. 򐂰 Stage 3: Deploy the resource model Engine (RME) to an Endpoint; still no changes to the Tivoli Distributed Monitoring (Classic Edition) 3.7 monitoring environment. 򐂰 Stage 4: Use the Workbench to migrate probe P1 to a Compatibility mode monitor (CP1) and deploy as a resource model. 򐂰 Stage 5: Migrate all probes to Compatibility mode resource models. 򐂰 Stage 6: Migrate Compatibility probes to CIM-based providers, undeploy the probes and remove the Tivoli Distributed Monitoring (Classic Edition) 3.7 engine.

112

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Custom Probes Migration Compatibility mode is a transitional function DM 3.7

ITM 5.1

Sentry Engine

Sentry Engine

P2 P1

P2 P1 Stage 2

P1

P1

Stage 1 ITM 5.1 Sentry Engine P1

P2 P1

Stage 4

RME

RM RM CP1

ITM 5.1 Sentry Engine P1

RME

P2 P1

Stage 3

ITM 5.1

ITM 5.1

RME

RME

RM

Stage 5

RM

RM

RM

Stage 6 Provider

CP1

Figure 5-1 Migration from Tivoli Distributed Monitoring (Classic Edition) 3.7

5.1.2 Custom monitors overview Before we begin describing the migration process we will provide a short overview of the type of custom monitors that are in the Tivoli Distributed Monitoring (Classic Edition) 3.7 product. Tivoli Distributed Monitoring (Classic Edition) 3.7 provides a wide range of monitors that are delivered with the product, such as monitors for operating system parameters, network parameters, and application parameters. In a number of cases, however, customers have extended the monitoring collections by creating custom monitors or creating custom monitor collections. Some examples are: 򐂰 Monitoring a device for which no monitor or monitoring collection exists 򐂰 Monitoring special operating system or network characteristics that are not covered by the standard monitors shipped with Tivoli Distributed Monitoring (Classic Edition) 3.7

Chapter 5. Migrating Classic Edition monitors

113

򐂰 Monitoring an operating system platform that is not directly supported by Tivoli Distributed Monitoring (Classic Edition) 3.7 򐂰 Running a probe that runs longer than sixty seconds 򐂰 Monitoring a list of resources (for example, services or file systems) There are basically two ways to create custom monitors in the Tivoli Distributed Monitoring (Classic Edition) 3.7 product, which are: 1. Creating custom numeric or string scripts as part of the Unix_Sentry or universal monitor collections. 2. Create Collection Specification Language (CSL) code and implement a new monitor collection using the mcsl command.

Creating custom numeric or string scripts The easiest way to create custom monitors is to use the numeric script and string script monitors from the Unix_Sentry or Universal monitoring collections. In these collections there are two specific monitors that are supplied to create custom monitors. The two monitors are the numeric script and the string script monitors. These monitors are stubs for customer-supplied scripts or programs. The actual code for the stub monitors are supplied as an argument to the monitor. This can be seen in Figure 5-2 on page 115.

114

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-2 Unix_Sentry numeric script monitor

In this example we are using a bourne shell script indicated by performance.sh; however, we could have used any scripting language that is supported by the endpoint (for example, Perl, bourne shell, korn shell, BAT file, and so on). A compiled program could also be supplied as a custom script. The difference between a numeric script and a string script is in the value the scripts return upon execution. The numeric script will return a numeric value that can be interpreted by the Tivoli Distributed Monitoring (Classic Edition) 3.7 engine. The string script will return a string that can be interpreted by the Tivoli Distributed Monitoring (Classic Edition) 3.7 engine.

Chapter 5. Migrating Classic Edition monitors

115

There are some limitations using the numeric and string scripts: 򐂰 Monitor arguments cannot be supplied through the Tivoli Distributed Monitoring (Classic Edition) 3.7 profile GUI like other shipped monitors. However, arguments can be passed along with the program name argument. 򐂰 The numeric and string script monitors will be from the Unix_Sentry or the universal collections and will all have the same event class for every event sent to TEC. 򐂰 The numeric and string script monitors are not extensible for different platforms. If one script needs to run differently on specific platforms, all of the code difference will have to be handled in the script. Section 5.3, “Custom numeric monitor conversion example” on page 137, is an example of a custom numeric script that will be converted into the new IBM Tivoli Monitoring 5.1 resource model format.

Monitoring Capability Specification Language (MCSL) MCSL addresses the limitations listed above. MCSL allows us to do the following: 򐂰 򐂰 򐂰 򐂰

Create new monitoring collections and monitors. Furnish monitors with a GUI to enter parameters. Automatically deploy monitoring code. Easily install monitoring collections in other TMRs. Note: The Tivoli Distributed Monitoring (Classic Edition) 3.7 Reference manual lists the mcsl command as the Monitoring Capability Specification Language. Most people refer to MCSL as the Monitoring Collection Specification Language and CSL as the Collection Specification Language. CSL is the raw source code supplied by the developer to create a MCSL collection.

In order to use MCSL we need to install the Application Development Environment (ADE), which is part of the Tivoli Framework. With MCSL we can still write our actual monitoring code in any of the languages that are supported by the numeric and string script monitors. Monitors can be written in Perl, korn shell, bourne shell, or C. MCSL is like C code and is primarily used to create a wrapper for our monitor source code, and it also makes it available to the Tivoli Distributed Monitoring (Classic Edition) 3.7 product. Tivoli Distributed Monitoring (Classic Edition) 3.7 provides an mcsl command that can be used to compile the CSL code, and that also installs the new monitor collections in the TMR. Figure 5-3 on page 117 is an example of a monitor collection and a specific monitor created using MCSL.

116

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Collections

Monitors

Arguments Figure 5-3 MCSL-created monitor collection and monitor

Section 5.4, “Converting a CSL-based monitor” on page 147, is an example of a CSL source monitor that will be converted into the new IBM Tivoli Monitoring 5.1 resource model format. For more information on creating MCSL/CSL monitors and collections see the following resources: 򐂰 Creating Custom Monitors for Tivoli Distributed Monitoring, SG24-5211 򐂰 Migrating from Systems Monitor for AIX to TME 10 Distributed Monitoring, SG24-4936 򐂰 http://www.orb-data.com/TipsOverview.html

Chapter 5. Migrating Classic Edition monitors

117

5.1.3 Sentry profile analyzer The sentry profile analyzer script analyzes the contents of all the sentry profiles and, based on a mapping table provided with IBM Tivoli Monitoring 5.1, produces a report suggesting how the monitors can be replaced with resource models or how new resource models can be created to collect the same data. The installation of the IBM Tivoli Monitoring 5.1 product puts the script, dmae_sentryanalyser.sh, in the $BINDIR/TME/Tmw2k/Migration_helper directory. The installation also places the mapping table file, monitors_rm_table.txt, in the same directory. The analysis is based on the mapping table. The launch of the script dmae_sentryanalyse.sh produces a report file dmae_analyser_report.txt. The report file suggests how the monitors can be replaced with resource models or how new resource models can be created to collect the same data. The mapping table defines metric values returned by each monitor provided by Tivoli Distributed Monitoring (Classic Edition) 3.7. The metric is compared to the mapping table and a report is generated if a metric is collected by a IBM Tivoli Monitoring 5.1 resource model (for example, the monitor AvailBytes of the NT_Memory monitoring collection is collected by the Memory resource model). In some cases a metric may be made available by a CIM provider, independently of whether it is a property of a CIM class. For example, the PgRdPerSec metric is made available by the PerfProv WMI provider, but there is no CIM class available on Windows 2000 for that property. The mapping table defines: 򐂰 The metrics values returned by each monitor provided by Tivoli Distributed Monitoring (Classic Edition) 3.7 collected by a IBM Tivoli Monitoring 5.1 resource model. 򐂰 Which CIM class has, as a property, the metric defined for a Tivoli Distributed Monitoring (Classic Edition) 3.7 monitor. 򐂰 The metrics defined in a CIM class, and if there is a IBM Tivoli Monitoring 5.1 MOF file defined. 򐂰 Any metrics that can be made available by a CIM provider. The sentry profile analyzer script must be run from the Tivoli environment on the TMR Server or ManagedNode. After sourcing the Tivoli environment, change to the proper directory before running the analyzer script. On the bash shell, type: cd $BINDIR/TME/Tmw2k/Migration_helper dmae_sentryanalyser.sh -p

118

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Based on the content of the mapping table described above, the sentry profile analyser analyzes all sentry profiles present in the Tivoli management region and produces a report that suggests the way to proceed in the migration process. Optionally, using the -p option, it can also create IBM Tivoli Monitoring 5.1 profiles with the resource models covering as much as possible the resources monitored by the Tivoli Distributed Monitoring (Classic Edition) 3.7 monitors. The script runs as follows: 򐂰 It verifies whether the metrics of the monitors set in a sentry profile are collected by any IBM Tivoli Monitoring 5.1 resource model. If they are, the script generates a section in the report describing which monitor can be replaced with which resource model. Optionally, a Tmw2kProfile with the same name as that containing the monitor will be created. 򐂰 If no resource models that collect a specific metric are found in the mapping table then the script tries to identify whether a CIM class exists that provides that value. 򐂰 Sometimes there might be a case where the script finds a metric that is not implemented by any CIM class, but that is provided by a provider such as PerfProv. Microsoft provides, with WMI core, the WMI performance counter provider that can be used to define CIM classes representing the objects and their counters as they appear in the Windows performance monitor. If the monitor examined cannot be remapped in any CIM class without the creation of a new provider or if the monitor is a custom script, the script suggests using the Compatibility Mode in conjunction with the Wizard-driven process and choosing the Tivoli Distributed Monitoring (Classic Edition) 3.7 classic monitoring collection as the data source. Appendix G in the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569, gives a completed overview of migrating Classic Edition monitors to the new IBM Tivoli Monitoring 5.1 resource model based monitors.

5.2 Classic Edition monitor conversion example In the first example we are going to demonstrate how to convert a Tivoli Distributed Monitoring (Classic Edition) 3.7 monitor that is defined in a shipped monitor collection to a IBM Tivoli Monitoring 5.1 resource model based monitor. In this example we are going to use the diskioratek monitor in the Unix_Sentry monitor collection. This monitor reports the total disk I/O written to and read from the specified disk during the previous five seconds. The total I/O for the specified disk is reported in kilobytes/second.

Chapter 5. Migrating Classic Edition monitors

119

The first step we are going to perform is to look at the monitor in more detail. The mcsl command can be used to display information about monitor collections and give details about a specific monitor. First we need to find the name of the monitor collection. The mcsl -c will list all of the installed monitor collections in a TMR. We know that the diskioratek is a monitor in the UNIX monitor collection in this example, so we entered the following command to find the exact name of the collection: mcsl -c | grep Unix

The output of this command can be seen in Figure 5-4.

Figure 5-4 MCSL -c example

Next we want to see exactly what type of monitor the diskioratek monitor is. We can do this by using the mcsl -p command to list all of the monitors in a specific collection. The following command will list information on the diskioratek monitor: mcsl -p Unix_Sentry | grep diskiorate

The output of this command can be seen in Figure 5-5.

Figure 5-5 Example mcsl -p command

120

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Finally, if we want to actually see what code gets executed by this monitor, we can use the mcsl -q command. The mcsl -q command dumps out the internal source for a monitor collection in unformated output. We used the following command in this example: mcsl -q Unix_Sentry > /tmp/Unix_Sentry.mcsl

Once all the source is dumped to a file we can use a command editor like vi on UNIX or Wordpad on Windows to edit the source. In the editor we can do a find on the diskioratek monitor name in the output source and try to find the actual code associated with the monitor. As we can see in Figure 5-6 on page 122, the format is not desirable for reading.

Chapter 5. Migrating Classic Edition monitors

121

Figure 5-6 Unix_Sentry/diskioratek MCSl output

122

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Using this method can sometimes be effective for monitors that only execute one command. For more complicated monitors there is a tool on the Web called owgetmon.pl. Owgetmonpl is a Perl script that can be found at http://www.orb-data.com/TipsOverview.html. The script can be run as follows: ./owgetmon.pl -f %c/%m/%i

The owgetmon.pl is a Perl script that will write out the source code of a monitor. By default it will build menu lists of all of the monitor collections followed by each monitor in a selected collection. The %c/%m/%i are flags that tell the script how to create the output. The script will create the file in a directory name of the collection (%c), sub-directory of the monitor name (%m), and the file name of the interp type selected (%i). An example of the output produced for the diskioratek monitor can be seen in Figure 5-7.

Figure 5-7 Example owgetmon.pl output

Note: A quick way to find out if a Classic Edition monitor is already supported by a IBM Tivoli Monitoring 5.1 resource model is to open up the monitors_rm_table.txt file and do a find for a specific monitor. This is the file that is used by the sentryanalyzer script as an input file. The sentryanalyzer script does a lookup against this file for the monitor name in question. When the monitor name in the file is found we can read the line from left to right. The first entry after the monitor name is the support resource model of “-”. The next column is the MOF file of “-”. The next column is the resource class or provider that supports this monitor. We can see in Figure 5-8 on page 124 that none of the shipped resource models support the diskioratek monitor.

Chapter 5. Migrating Classic Edition monitors

123

Figure 5-8 Editing the monitors_rm_table.txt file to find a monitor

Creating a resource model from a MCSL collection In order to change the diskioratek monitor into a resource model based monitor we will have to use the IBM Tivoli Monitoring 5.1 Workbench resource model wizard. The resource model wizard will prompt us with a set of windows that will create a Compatibility Model monitor. For more information about using the resource model wizard please see 10.3.2, “ITSO_LogicalDiskModel example” on page 301. Once inside the Workbench we are going to select File -> New and choose Java Script Resource Model (for this example). Next we will select Distributed Monitoring (Classic Edition) Collection as the data source type (see Figure 5-9 on page 125).

124

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-9 Converting MCSL-based monitor collections

In Figure 5-9 we have selected the Distributed Monitoring (Classic Edition) Collection data source option. This option is used if we are converting from a MCSL-dumped collection like the one we are using in this example. We could have also used this option to convert a user-developed CSL monitor, as we will see later in 5.4, “Converting a CSL-based monitor” on page 147.

Chapter 5. Migrating Classic Edition monitors

125

The next window provided by the wizard will ask for the type of collection we are converting. In this example we are converting a dumped MCSL collection (that is, the Unix_Sentry collection). We select the Dumped Monitor Collection option for the MCSL dumped file. This can be seen in Figure 5-10. We also used the browser on this screen to select the dumped MCSL output file (that is, Unix_Sentry.mcsl). Notice that the Format of the file to import button is Dumped Monitoring Collection in this example. This option is used for dumped MCSL files.

Figure 5-10 Selecting the Dumped Monitoring Collection option

After we select the Next button the Wizard will format all of the monitors in the Unix_Sentry collection and display them as seen in Figure 5-11 on page 127.

126

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-11 Listing all of the monitors in the Unix_Sentry monitor collection

Next we are going to select the diskioratek monitor in the list. If the monitor was created with arguments, the wizard will display an argument screen. The diskioratek monitor accepts the disk name as an argument. This can be seen in Figure 5-12 on page 128. Note: Notice we have entered hdisk0 as the argument in this example. How did we know if that was a valid argument? One way would be to open an existing Tivoli Distributed Monitoring (Classic Edition) 3.7 profile that has the diskioratek monitor and look at the supplied arguments. However, if this is a monitor that is being created we might not have an existing monitor to look at. We could look at the command in the monitor code (MCSL), as seen in the previous example, and run the command, in this case iostat. We ran the iostat to see that hdisk0 was a valid argument for this monitor.

Chapter 5. Migrating Classic Edition monitors

127

Figure 5-12 Selecting the Monitor Arguments

Tip: The wizard will not allow us to select more than one monitor of the same type. For example, we cannot select the diskioratek for hdisk0 and hdisk1 from the wizard. However, like all of the previous examples, it is very easy to modify a resource model after the wizard completes. In this example we could add multiple instances to the resource model elements after the wizard completes. The next window presented by the wizard is the Event Triggering Conditions window. This is the window that defines the event and threshold elements for the resource model. In this example we are going to select the diskioratek and set a threshold (for testing >1). This can be seen in Figure 5-13 on page 129.

128

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-13 Selecting event and threshold elements

Next the wizard will prompt for logging elements. This can be seen in Figure 5-14.

Figure 5-14 Selecting logging elements

We then select the Finish button and get prompted with the default cycle time. In this example we selected 120 seconds as the cycle time.

Chapter 5. Migrating Classic Edition monitors

129

Finally we can see in Figure 5-15 the fully generated resource model based on the diskioratek monitor. Notice we have renamed the resource model and the event elements.

Figure 5-15 Wizard-generated resource model for diskioratek after modifications

We also changed the general setting, as can be seen in Figure 5-16 on page 131.

130

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-16 General Settings window

Notice in Figure 5-15 on page 130 the dynamic model element CIM classes do not have any instances defined under them, nor do the DM classic probes. This is because this not a CIM-based resource model. It is a compatibility mode resource model. Instead of calling a CIM provider, this resource mode is going to invoke a compatibility mode method associated with the DM classic probe definition. We can see the actual code that will get invoked for this example by selecting the Unix_Sentry.diskioratek element in the pane. Figure 5-17 on page 132 is an example of the diskioratek probe display.

Chapter 5. Migrating Classic Edition monitors

131

Figure 5-17 Diskioratek Source Details window

Finally let us look at the VisitTree code in Example 5-2 on page 133 that was generated by the wizard to get a better understanding of how compatibility mode resource models work. Note: In previous examples using the resource model wizard, the generated VisitTree code had to be modified. In this chapter none of the code generated by the Workbench wizard had to be modified. All of the migration example VisitTree code worked out of the box without any modifications. First let us look at something new in this VisitTree example. The wizard generated variable definitions in the beginning of the VisitTree routine. These variables are not in any of the specific functions (that is, SetDefaultConfiguration, Init, or VisitTree). These variables, because they are not defined in a specific

132

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

function, are considered global variables. The compatibility mode resource models use global variables to store pointers to mapping tables. These mapping tables will be persistent between VisitTree calls and are used to save the previous monitor values. Example 5-1 shows the global variables that were used for this resource model. Example 5-1 Global variables //''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''''''''''''' // IBM Tivoli Monitoring // Decision Tree script // // This file has been generated by IBM Tivoli Monitoring Workbench // // 05/27/2002 13:19:57 //''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''' var hTablediskioratek; var flagNotFirstRun = 0;

Example 5-2 is the VisitTree code for this resource model. Example 5-2 Diskioratek VisitTree function 1 function VisitTree(Svc) 2 { 3 var curdiskioratek; 4 5 6 var hashKey; 7 var found; 8 9 var Olddiskioratek; 10 11 var hPropTable; 12 var ProbeArgCount; 13 var ProbeArgIdx; 14 var ParamCount; 15 var ParamIdx; 16 var Different; 17 18 hPropTable = Svc.CreateMap(); 19 20 //Implementation for the monitor diskioratek 21 ProbeArgCount = Svc.GetStrParameterCount(“Probe_arg_for_diskioratek”);

Chapter 5. Migrating Classic Edition monitors

133

22 for (ProbeArgIdx = 0; ProbeArgIdx < ProbeArgCount; ProbeArgIdx++) { 23 24 Svc.RemoveMapAll(hPropTable); 25 hashKey = “diskioratek” + Svc.GetStrParameter(“Probe_arg_for_diskioratek”,ProbeArgIdx); 26 curdiskioratek = Svc.CallDMNumProbe(“Unix_Sentry.diskioratek”, Svc.GetStrParameter(“Probe_arg_for_diskioratek”,ProbeArgIdx)); 27 Svc.SetMapNumElement(hPropTable,"diskioratek",curdiskioratek); 28 found = Svc.ExistsMapElement(hTablediskioratek, hashKey); 29 if (found) 30 Olddiskioratek = Svc.GetMapNumValue(hTablediskioratek, hashKey); 31 else 32 Olddiskioratek = 0; 33 34 Svc.SetMapNumElement(hPropTable,"Olddiskioratek",Olddiskioratek); 35 Svc.SetMapStrElement(hPropTable,"collection","Unix_Sentry"); 36 Svc.SetMapStrElement(hPropTable,"probe",“diskioratek”); 37 38 Svc.SetMapStrElement(hPropTable,"probe_arg",Svc.GetStrParameter("Probe_arg_for_diskioratek",Pro beArgIdx)); 39 Svc.SetMapStrElement(hPropTable,"prev_value", Olddiskioratek); 40 Svc.SetMapStrElement(hPropTable,"value", curdiskioratek); 41 Svc.SetMapStrElement(hPropTable,"relation_delta", Olddiskioratek - curdiskioratek ); 42 if (curdiskioratek > Svc.GetThreshold("Thr_diskioratek_gt") ) { 43 Svc.SetMapNumElement(hPropTable,"UpperBound",Svc.GetThreshold("Thr_diskioratek_gt")); 44 Svc.SendEventEx ("ITSO_Unix_Sentry_diskioratek_too_high",hPropTable); 45 } 46 47 Svc.LogInstEx ("Unix_Sentry_Availability","diskioratek", hPropTable); 48 49 Svc.SetMapNumElement(hTablediskioratek, hashKey, curdiskioratek); 50 } 51 52 Svc.DestroyMap(hPropTable); 53 54 return (0); 55 56 }

134

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Let us examine the code in Example 5-2 on page 133. 򐂰 Line 21 This is the method call to get the count of values in the parameter list from the parameter element. When a monitor is selected from the wizard it will create a unique parameter element for each monitor that is selected. In the Example 5-2 on page 133 we selected diskioratek. Here we could have added another value in the Probe_arg_for_diskioratek parameter element for multiple instances of the same monitor. The VisitTree code generated by the wizard will handle multiple values in the parameter element. Note: If different monitors are selected in the wizard, the generated VisitTree code will create multiple parameter elements and create a separate loop for each monitor.

򐂰 Line 22 This is the loop for each value in the parameter element, (that is., in this case the monitor). In this example there is only one value in the parameter element. We can see in the VisitTree function that the wizard-generated code will handle multiple instances of the same monitor. The wizard unfortunately will not generate multiple instances of the same monitor. 򐂰 Line 25 This line is used to generate a hash key. The hash key is a concatenation of monitor name and monitor argument. This will be used later to retrieve the previous value for this monitor instance. The compatibility mode monitors are designed to work similarly to the Classic Edition monitors in that they will store the previous value and calculate a delta value on each iteration. In the case of the resource model an iteration is every VistTree cycle time. 򐂰 Line 26 This line calls the compatibility mode method, Svc.CallDMNumProbe. This method will invoke the specific monitor code listed in Example 5-17 on page 132. The second argument in the method call is another method call to the Svc.GetStrParameter method to get the argument from the parameter element. In this example this will be the hdisk0 we supplied to the wizard in Example 5-2 on page 133. The Svc.CallDMNumProbe method call will return only one line of standard output.

Chapter 5. Migrating Classic Edition monitors

135

Tip: This Svc.CallDMNumProbe method runs the probe exactly the same way Tivoli Distributed Monitoring (Classic Edition) 3.7 monitors run. The probe will only run for 60 seconds, and any output prefixed with the # character will not be used in the evaluation. At the current time the #-prefixed output is not used. This may be supported in future releases of IBM Tivoli Monitoring 5.1. 򐂰 Line 28 This line retrieves the previous value stored for this monitor. The hTablediskioratek is the global variable used to store the previous value of the monitor. 򐂰 Line 28–32 These lines check for the existence of the previous values of the diskioratek monitor from the global mapping table. 򐂰 Line 34–40 These are the method calls to set the mapping tables used by the Svc.SendEventEx and Svc.LogInstEx methods. 򐂰 Line 41 This line calculates the delta value and stores it in the mapping table. 򐂰 Line 42 This is a comparison to see if the value returned from Svc.CallDMNumProbe exceeds the threshold element. 򐂰 Line 43 This line sets the threshold value as an element in the mapping table so that it shows as part of the events and the log. 򐂰 Line 44 This is the call to generate an indication. 򐂰 Line 47 This is a call to log all of the properties. In the wizard-generated code all elements are logged on every iteration of VisitTree. If we wanted to change this we could move this line to the inside of the previous loop. 򐂰 Line 49 This line saves the current value from the probe in the global mapping table. This value will be used on the next iteration of VisitTree as the previous value.

136

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

5.3 Custom numeric monitor conversion example In the next example we are going to demonstrate how to convert a Tivoli Distributed Monitoring (Classic Edition) 3.7 custom numeric monitor into a resource model based monitor. We are going to take a custom monitor that was defined as a custom numeric monitor in the universal collection and convert it to a resource model using the Workbench resource model wizard. The custom numeric monitor that we have developed is called processor.sh. The processor.sh is a bourne shell script that invokes the UNIX sar command on AIX to gather CPU metrics. Example 5-3 is an example of the processor.sh code that was used. Example 5-3 Processor.sh script #!/usr/bin/sh # # process.sh - This script takes three arguments: # # $1 = type of cpu utilization (-u,-s,-w,-i) # $2 = processor number (e.g., 0 ) Default=ALL # $3 = interval number for sar averaging Default=5 interval=5 if [ "$#" -eq 0 ] ; then echo "Usage: preocessor.sh " exit 1 else type=$1 fi if [ "$2" -eq "" ] ; then processor=ALL else processor=$2 fi if [ "$3" -ne "" ] ; then interval=$3 fi cmd="sar -u -P $processor $interval" case "-u" "-s" "-w" "-i" * )

$type in ) output=`$cmd | awk 'NR==5 {print $3}'`;; ) output=`$cmd | awk 'NR==5 {print $4}'`;; ) output=`$cmd | awk 'NR==5 {print $5}'`;; ) output=`$cmd | awk 'NR==5 {print $6}'`;; output=`$cmd | awk 'NR==5 {print $3}'`;;

Chapter 5. Migrating Classic Edition monitors

137

esac echo $output

As we can see from Example 5-3 on page 137, processor.sh takes three arguments and is flexible in checking for longer intervals or for totals of all processors.

Creating a resource model from a custom monitor In order to change the processor.sh custom monitor into a resource model based monitor we will have to use the IBM Tivoli Monitoring 5.1 Workbench resource model wizard. The resource model wizard will prompt us with a set of Windows that will create a Compatibility Mode monitor. For more information about using the resource model wizard please see 10.3.1, “ITSO_PhysicalDiskModel example” on page 282. Once inside the Workbench we are going to select File -> New and choose Java Script Resource Model. Next we will select Custom Script as the data source type, as can be seen in Figure 5-18 on page 139. In this example we need to select Custom Script as the data source when we are building a resource model from a custom numeric or string script.

138

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-18 Creating custom script resource models

The next window in the wizard will ask for the location of the source script to import from. In this case we supplied the source location of the processor.sh script. Figure 5-19 on page 140 is an example of this screen. Note: There is a little glitch in the Import window in the wizard. In order to make the Next button available we had to make an update to the shell input field. All we did was overtype the processor.sh value and press Enter. By doing this the Next button became available.

Chapter 5. Migrating Classic Edition monitors

139

Figure 5-19 Import the Custom Script window

Next the wizard will prompt for the event and threshold elements. In this example there is only one property and it is the script result. This is the value that will be returned by the processor.sh script. An example of this can be seen in Figure 5-20 on page 141.

140

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-20 Setting the event and threshold elements

Next the wizard will prompt for logging elements. This can be seen in Figure 5-21.

Figure 5-21 Setting the logging elements

Finally, we can see in Figure 5-22 on page 142 the fully generated resource model based on the processor.sh custom numeric script monitor. Notice we have renamed the resource model and the event elements.

Chapter 5. Migrating Classic Edition monitors

141

Figure 5-22 ITSO_Processor.sh resource model

We also changed the general setting, as can be seen in Figure 5-23 on page 143. Note: Notice in Figure 5-22 that there are no elements under the dynamic model CIM classes or under DM classic probes. The ITSO_Processor.sh resource model is neither a CIM-based model nor considered a classic edition probe; it is a custom script resource model. The custom script monitors are just converted into Svc.Shell method calls.

142

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-23 ITSO_Processor.sh General Settings window

Finally, let us look at the VisitTree code that was generated by the wizard to get a good understanding of how custom script resource models run. This resource model wizard for custom scripts will also generate global variables as can be seen in Example 5-4. The custom script resource models use global variables to store pointers to mapping tables. These mapping tables will be persistent between VisitTree calls and are used to save the previous script results. Example 5-4 shows the global variables that were used for this resource model. Example 5-4 ITSO_Processor.sh global variables //''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''''''''''''' // IBM Tivoli Monitoring // Decision Tree script // // This file has been generated by IBM Tivoli Monitoring Workbench // // 05/28/2002 15:15:10 //''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''''''''' var hTableScriptResult; var flagNotFirstRun = 0;

Chapter 5. Migrating Classic Edition monitors

143

Example 5-5 is the VisitTree code for this resource model. Example 5-5 ITSO_Processor_sh VistTree code 1 function VisitTree(Svc) 2 { 3 var curScriptResult; 4 5 6 var hashKey; 7 var found; 8 9 var OldScriptResult; 10 11 var hPropTable; 12 var ProbeArgCount; 13 var ProbeArgIdx; 14 var ParamCount; 15 var ParamIdx; 16 var Different; 17 18 hPropTable = Svc.CreateMap(); 19 20 //Implementation for the monitor ScriptResult 21 Svc.RemoveMapAll(hPropTable); 22 hashKey = "ScriptResult"; 23 curScriptResult = Svc.Shell ("processor.sh"); 24 Svc.SetMapNumElement(hPropTable,"ScriptResult",curScriptResult); 25 found = Svc.ExistsMapElement(hTableScriptResult, hashKey); 26 if (found) 27 OldScriptResult = Svc.GetMapNumValue(hTableScriptResult, hashKey); 28 else 29 OldScriptResult = 0; 30 31 Svc.SetMapNumElement(hPropTable,"OldScriptResult",OldScriptResult); 32 Svc.SetMapStrElement(hPropTable,"prev_value", OldScriptResult); 33 Svc.SetMapStrElement(hPropTable,"value", curScriptResult); 34 Svc.SetMapStrElement(hPropTable,"relation_delta", OldScriptResult - curScriptResult); 35 if (curScriptResult > Svc.GetThreshold("Thr_ScriptResult_gt") ) { 36 Svc.SetMapNumElement(hPropTable,"UpperBound",Svc.GetThreshold("Thr_ScriptResult_gt")); 37 Svc.SendEventEx ("ITSO_processor_sh_ScriptResult_too_high",hPropTable); 38 } 39 40 Svc.SetMapStrElement(hPropTable,"key", "@"); 41 Svc.LogInstEx ("processor_sh_Availability","ScriptResult", hPropTable); 42 43 Svc.SetMapNumElement(hTableScriptResult, hashKey, curScriptResult); 44 45 Svc.DestroyMap(hPropTable);

144

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

46 47

return (0);

Let us examine the code in Figure 5-5 on page 144. 򐂰 Line 22 This line is used to generate a hash key. The hash key in this example is just the literal ScriptResult. This will be used later to retrieve the previous value for this monitor instance. The custom script monitors are designed to work similarly to the Classic Edition monitors in that they will store the previous value and calculate a delta value on each iteration. In the case of this resource model an iteration is every VistTree cycle time. 򐂰 Line 23 This line is the call for the custom script Svc.Shell method. The Svc.Shell method will invoke a shell script. We saw an example of this in 10.3.4, “ITSO_FileSystem Example” on page 353. The resource model wizard will generate code to call the script as a shell environment. The wizard will also add the script automatically as a dependency element, as can be seen in Figure 5-22 on page 142. The Svc.Shell method will only return one line of standard output just like the Svc.CallDMNumProbe. Tip: This Svc.Shell method runs the script exactly the same way Tivoli Distributed Monitoring (Classic Edition) 3.7 monitors run. The script will only run for 60 seconds, and any output prefixed with the # character will not be used in the evaluation. At the current time the #-prefixed output is not used. This may be supported in a future release of IBM Tivoli Monitoring 5.1.

Chapter 5. Migrating Classic Edition monitors

145

Note: Behavior of Svc.Shell method within IBM Tivoli Monitoring Version 5.1. There is a difference in behavior between the UNIX and Windows platforms in regards to the Svc.Shell method when we tested it. 򐂰 Windows – Any process launched via this method has a 60-second time limit. If the process does not terminate it is killed. – If the launched process results in a non-zero return code or empty output, the resource model is stopped and put into an error status. This will be changed in a future release or patch to behave like the UNIX platform. 򐂰 UNIX – Processes launched via this method have no time limit. This will be changed in a future release or patch to behave like the Windows platform. A hung or looping process could cause the entire resource model to hang. Do not create resource models on UNIX with the assumption that processes can run over 60 seconds. – If the launched process results in a non-zero return code or empty output, the resource model continues running. It is our understanding that a future release or patch will allow for a configured time limit on both platforms. 򐂰 Line 25 This line retrieves the previous value store for this monitor. hTableScriptResult is the global variable used to store the previous value of the monitor. 򐂰 Line 26–29 These lines check for the existence of the previous values of the processor.sh script monitor from the global mapping table. 򐂰 Line 31–33 These are the method calls to set the mapping tables used by the Svc.SendEventEx and Svc.LogInstEx methods. 򐂰 Line 34 This line calculates the delta value and stores it in the mapping table.

146

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Line 35 This is a comparison to see if the value returned from Svc.Shell exceeds the threshold element.

5.4 Converting a CSL-based monitor In the last example we are going to convert a custom-developed monitor using MCSL/CSL code. First we are going to show how the processor.sh custom script can be turned into a CSL-based monitor, and then we will convert it to a resource-based monitor. Figure 5-6 is an example of the CSL code we used to turn processor.sh into a CSL-based monitor. Example 5-6 Processor.csl #include "Sentry2_0.dsl" Collection "ITSO_Processor" { CodeID = "$id:processor.csl,v 1.0$"; Version = "1.0"; Require = ">2.0.2"; EventBaseClass = "ITSO_Processor"; NoticeGroup = "Sentry"; HelpMessage = (__no_catalog__,"This collection contains custom monitors",1); #include "operators.csl" #include "choicelists.csl" #include "formats.csl" ChoiceList ProcessorCPUOption { ButtonLabel=(__no_catalog__,"CPU Output Option",1); { { (__no_catalog__,"User",1) "-u"} { (__no_catalog__,"System",1) "-s" } { (__no_catalog__,"Wait io",1) "-w"} { (__no_catalog__,"Idle",1) "-i" } }; }; Monitor processor Numeric Group numeric { Description = (__no_catalog__,"CPU Processor percent used",1); HelpMessage = (__no_catalog__,"CPU Processor percent used Select processor (0-4)",1);

Chapter 5. Migrating Classic Edition monitors

147

Argument (__no_catalog__,"CPU Option",1) RestrictedChoice "ProcessorCPUOption" DefaultValue "-u"; Argument (__no_catalog__,"Processor",1) DefaultValue "0"; Argument (__no_catalog__,"Interval",1) DefaultValue "5"; Implementation (aix4-r1) Shell ("/bin/sh", "-c", Command, "processor") Import "/RMFiles/ITSO/processor.csl/processor.sh"; }; }

Figure 5-24 on page 149 is an example of the Tivoli Distributed Monitoring (Classic Edition) 3.7 Add Monitor window for the MCSL-generated code in Example 5-6 on page 147.

148

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-24 MCSL-generated monitor collection

Creating a resource model from a CSL file In order to change the processor.csl custom monitor into a resource model based monitor we will have to use the IBM Tivoli Monitoring 5.1 Workbench resource model wizard. The resource model wizard will prompt us with a set of Windows that will create a Compatibility Mode monitor. For more information about using the resource model wizard please see 10.3.1, “ITSO_PhysicalDiskModel example” on page 282.

Chapter 5. Migrating Classic Edition monitors

149

Once inside the Workbench, select File -> New and choose Java Script Resource Model for this example. Next we select Distributed Monitoring (Classic Edition) Collection as the data source type, as can be seen in Figure 5-25. In this example we need to select Distributed Monitoring (Classic Edition) Collection as the data source.

Figure 5-25 Converting CSL monitors using the resource model wizard

The next window in the wizard will ask for the location of the source CSL code to import from. In this case we supplied the source location of the processor.csl file. Figure 5-26 on page 151 is an example of this screen. Notice the Format of the file to import button in this example is CSL. This option is used when the monitor is in source CSL format.

150

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-26 Importing the processor.csl file

When we select the CSL file, the wizard will prompt us with a Preprocessing settings window. These are the settings that will be used on the mcsl command. In most cases the defaults will work. Figure 5-27 on page 152 is an example of the Preprocessing settings window.

Chapter 5. Migrating Classic Edition monitors

151

Figure 5-27 Preprocessor settings window

Next the wizard will prompt us for the monitor sources. In this example there is only one monitor source and it is the processor script. Since the CSL was based on a numeric monitor, the processor monitor will have the type numeric. An example of this can be seen in Figure 5-28.

Figure 5-28 Select the Monitoring Sources window

If the monitor was created with arguments, the wizard will display an argument screen. The processor monitor accepts the three arguments included in the script. This can be seen in Figure 5-29 on page 153.

152

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-29 Processor monitor arguments

Next the wizard will prompt us for the event and threshold elements. In this example there is only one property and it is the processor script. This is the value that will be retuned by the processor monitor created by the CSL. An example of this can be seen in Figure 5-30 on page 154.

Chapter 5. Migrating Classic Edition monitors

153

Figure 5-30 Specify the Event Triggering Conditions window

Next the wizard will prompt for logging elements. This can be seen in Figure 5-31.

Figure 5-31 Selecting log elements

154

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Finally we can see in Figure 5-32 the fully generated resource model based off the processor monitor. Notice we have renamed the resource model and the event elements.

Figure 5-32 ITSO_Processor

We have also changed the general settings, as can be seen in Figure 5-33 on page 156.

Chapter 5. Migrating Classic Edition monitors

155

Figure 5-33 ITSO_Processor General Settings window

Notice in Figure 5-32 on page 155, in the Dynamic Model element, the DM Classic Probe has ITSO_Processor.processor as a sub-element. This is the probe that was created by the source CSL. If we display the probe element we will see the source CSL code. This can be seen in Figure 5-34 on page 157.

156

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 5-34 Monitor Source Details window

The actual probe code is store in the ITSO_Processor.dmjsws project file and can be seen using WinZip. To use WinZip, the.zip file has to be extracted and the file has to be renamed with the .tar.gz extension before we can open it. An example can be seen in Figure 5-35 on page 158.

Chapter 5. Migrating Classic Edition monitors

157

Figure 5-35 The extracted tar file - Probe file

Example 5-7 is an example of the VisitTree code generated for this example. Example 5-7 ITSO_Processor VisitTree function function VisitTree(Svc) { var curprocessor;

var hashKey; var found; var Oldprocessor; var var var var var var

hPropTable; ProbeArgCount; ProbeArgIdx; ParamCount; ParamIdx; Different;

hPropTable = Svc.CreateMap(); //Implementation for the monitor processor ProbeArgCount = Svc.GetStrParameterCount("Probe_arg_for_processor"); for ( ProbeArgIdx = 0; ProbeArgIdx < ProbeArgCount; ProbeArgIdx++) { Svc.RemoveMapAll(hPropTable); hashKey = "processor" + Svc.GetStrParameter("Probe_arg_for_processor",ProbeArgIdx); curprocessor = Svc.CallDMNumProbe("ITSO_Processor.processor", Svc.GetStrParameter("Probe_arg_for_processor",ProbeArgIdx));

158

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Svc.SetMapNumElement(hPropTable,"processor",curprocessor); found = Svc.ExistsMapElement(hTableprocessor, hashKey); if (found) Oldprocessor = Svc.GetMapNumValue(hTableprocessor, hashKey); else Oldprocessor = 0; Svc.SetMapNumElement(hPropTable,"Oldprocessor",Oldprocessor); Svc.SetMapStrElement(hPropTable,"collection","ITSO_Processor"); Svc.SetMapStrElement(hPropTable,"probe","processor"); Svc.SetMapStrElement(hPropTable,"probe_arg",Svc.GetStrParameter("Probe_arg_for_processor",Probe ArgIdx)); Svc.SetMapStrElement(hPropTable,"prev_value", Oldprocessor); Svc.SetMapStrElement(hPropTable,"value", curprocessor); Svc.SetMapStrElement(hPropTable,"relation_delta", Oldprocessor - curprocessor ); if (curprocessor > Svc.GetThreshold("Thr_processor_gt") ) { Svc.SetMapNumElement(hPropTable,"UpperBound",Svc.GetThreshold("Thr_processor_gt")); Svc.SendEventEx ("ITSO_Processor_processor_too_high",hPropTable); } Svc.LogInstEx ("ITSO_Processor_Availability","processor", hPropTable); Svc.SetMapNumElement(hTableprocessor, hashKey, curprocessor); } Svc.DestroyMap(hPropTable); return (0); }

The VisitTree function for this example is basically identical to the example in Figure 5-2 on page 133. The only difference is the generated names in the code. The code structure is exactly the same. See Figure 5-2 on page 133 for an overview of the code logic.

Chapter 5. Migrating Classic Edition monitors

159

160

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

6

Chapter 6.

Migrating from Tivoli Web Component Manager This chapter discusses the migration process and tools provided with IBM Tivoli Monitoring Version 5.1. These processes and tools are very well documented in the chapter on migration considerations in the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569. Here we hope to fill in a couple of the blanks that we found when following the process in the user’s guide.

© Copyright IBM Corp. 2002

161

6.1 Tools The primary tool in the Tivoli Web Component Manager migration is the TIMS2XML Java-based application. This application extracts the information in the TWCM monitors running on your Web servers and formats the data into an XML file. You may think that this is not a migration. However, these tools assist in the migration by extracting the data from the monitor and putting it into a format that allows you to easily recreate it in an IBM Tivoli Monitoring Version 5.1 resource model. The TIMS2XML tool is located on your Tools CD under the directory named TWCM, and must be installed on your Tivoli Internet Management Server (TIMS). The installation process simply copies the folder from the CD to your TIMS server. The configuration process is more in-depth. You must configure TIMS2XML before first use. To configure it, locate the file named launch.bat (Windows NT/2000) or launch.sh (UNIX) and open the file in a text editor. This file is located in the directory that you copied over from the Tools CD. Example 6-1 shows our launch.bat. Example 6-1 launch.bat @echo off rem *** Path to the location of JDK binaries dir set JDK_BINDIR=C:\Java13\bin rem set JDK_BINDIR= rem *** Path to the TIMS lib dir set TIMS_LIBDIR=D:\Tivoli\Internet\ManagementServer\TIMS\lib rem set TIMS_LIBDIR= rem *** Path to the JDBC classes dir set JDBC_PATH=D:\Oracle\Ora81\jdbc\lib\classes111.zip rem set JDBC_PATH= rem *** Path to the Migration.jar file set MIGRATION_PATH=.. rem set MIGRATION_PATH= rem ############## END of User-configurable variables ############# set OLD_PATH=%PATH% set OLD_CLASSPATH=%CLASSPATH% set PATH=%JDK_BINDIR%;%PATH%

162

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

set CLASSPATH=%MIGRATION_PATH%\migration.jar;%TIMS_LIBDIR%\fw-common.jar;%TIMS_LIBD IR%\fw-svr.jar;%TIMS_LIBDIR%\properties.jar;%TIMS_LIBDIR%;%TIMS_LIBDIR%\propert ies;%JDBC_PATH% java com.tivoli.wcm.migration.Tims2XML > TIMS_tasks.xml set PATH=%OLD_PATH% set CLASSPATH=%OLD_CLASSPATH% set set set set set

OLD_PATH= OLD_CLASSPATH= JDK_BINDIR= TIMS_LIBDIR= JDBC_PATH=

The classpath variables must be set correctly or the tool will fail. Example 6-2 is an example of properly set classpath variables. Example 6-2 Classpath variables set TIMS_HOME=D:\Tivoli\Internet\ManagementServer\TIMS set CLASSPATH=%TIMS_HOME%\lib\fw-common.jar set CLASSPATH=%CLASSPATH%;%TIMS_HOME%\lib\fw-svr.jar set CLASSPATH=%CLASSPATH%;%TIMS_HOME%\lib\properties.jar set CLASSPATH=%CLASSPATH%;%TIMS_HOME%\lib set CLASSPATH=%CLASSPATH%;%TIMS_HOME%\lib\properties set CLASSPATH=%CLASSPATH%;D:\Oracle\Ora81\jdbc\lib\classes111.zip set CLASSPATH=%CLASSPATH%;.

Once the above parameters are specified you can begin the migration process. To execute TIMS2XML run the launch.bat or launch.sh file. By default the output will go into a file called TIMS_tasks.xml in your working directory. You can then view this file with a browser; Internet Explorer views these files very well. Open the XML file in a browser; each monitor is broken down by the title of Task Name. The mapping of the TWCM tasks into IBM Tivoli Monitoring Version 5.1 resources is laid out in the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569, in the Migrating from Tivoli Web Component Manager section.

Chapter 6. Migrating from Tivoli Web Component Manager

163

164

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Part 3

Part

3

Configuration In this part we describe the process for configuring IBM Tivoli Monitoring 5.1.

© Copyright IBM Corp. 2002

165

166

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

7

Chapter 7.

Heartbeat IBM Tivoli Monitoring 5.1 includes a heartbeat function, which monitors the basic signs of life at endpoints attached to the gateway at which it is enabled. One of the problems that customers faced with older versions of Distributed Monitoring was that if the engine on the endpoint failed, they had no way of knowing this. The heartbeat function regularly monitors the endpoints, checking if they are running correctly. Events may be sent to the Tivoli Business Systems Manager (provided that the Tivoli Business Systems Manager Adapter component is installed at the gateway), the Tivoli Enterprise Console (TME/secure events only), or a Tivoli Notice Group (Tivoli Distributed Monitoring Advanced Edition). The function can register the following statuses for an endpoint in its cache. They are divided into two groups, depending on whether an information or an error event is sent to the monitors: 򐂰 Statuses for which an information event is sent: – Alive – Heartbeat has been stopped 򐂰 Statuses for which an error event is sent: – Resource model in error status – IBM Tivoli Monitoring 5.1 engine has been stopped – Endpoint not available

© Copyright IBM Corp. 2002

167

The following topics are discussed is this section: 򐂰 Heartbeat overview 򐂰 Installing and configuring the heartbeat 򐂰 Controlling heartbeat

168

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

7.1 Heartbeat overview The heartbeat function is enabled on the ManagedNode/gateway of the endpoints that you want to monitor. The heartbeat monitoring engine has the ability to forward events to the Tivoli Enterprise Console and/or Tivoli Business Systems Manager and send notices to the Tivoli Notice Groups. There are three activities that make up the heartbeat function: 򐂰 Endpoint registration 򐂰 Heartbeat monitoring 򐂰 Viewing endpoint cache

7.1.1 Endpoint registration When an IBM Tivoli Monitoring 5.1 profile is pushed to an endpoint for the first time or the IBM Tivoli Monitoring 5.1 engine is restarted on an endpoint, the information in the endpoint cache is updated when the gateway receives a message from the endpoint informing the gateway that its IBM Tivoli Monitoring 5.1 engine has been started. Figure 7-1 illustrates the flow of data.

Figure 7-1 Heartbeat endpoint registration data flow

Chapter 7. Heartbeat

169

Once the profile has been distributed to the endpoint or the IBM Tivoli Monitoring 5.1 engine has been restarted on the endpoint, it sends an upcall (a method called register_endpoint) to the gateway, which then registers the endpoint in the endpoint cache. Additionally, while the engine is active, it sends this register_endpoint upcall to its gateway every 18 minutes. This behavior is in place to help track endpoint migration between gateways. As of this release, this upcall cannot be configured for a different time interval, nor can it be disabled. Also note that this upcall occurs regardless of whether the heartbeat is turned on or turned off.

7.1.2 Heartbeat monitoring The heartbeat status check of endpoints is initiated from the gateway. Note that because of this, the granularity of control is at a per-gateway level. Either all registered endpoints on a given gateway are eligible for a heartbeat (the heartbeat is turned on for that gateway) or none of them are (the heartbeat is turned off on the gateway). Figure 7-2 shows the flow of data during the heartbeat monitoring process.

Figure 7-2 Heartbeat monitoring data flow

170

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The gateway issues periodic heartbeat requests (downcalls) to all attached endpoints. The data that is returned is then stored in the endpoint cache and events are sent to the configured event targets.

7.1.3 Viewing the endpoint cache When the heartbeat monitor detects problems with the resource model, IBM Tivoli Monitoring 5.1 engine, or endpoint, it will send events to Tivoli Enterprise Console, Tivoli Business Systems Manager, or the Tivoli notice groups. In addition, the heartbeat information in the endpoint cache can also be viewed using the wdmmngcache command. Figure 7-3 shows the flow of data when the command is invoked.

Figure 7-3 Endpoint cache retrieval data flow

Chapter 7. Heartbeat

171

7.2 Installing and configuring heartbeat If you are using TEC, make sure you have installed and loaded the necessary BAROC files and rule sets to handle heartbeat-generated events. In order to install these files run the script $BINDIR/TMNT_TEC/dmae_tec_inst.sh on your TEC Server. Alternatively, you can manually apply the files $BINDIR/TMNT_TEC/hb_events.baroc and hb_events.rls to your TEC Server. The heartbeat function forms an integral part of IBM Tivoli Monitoring 5.1 and is already installed when IBM Tivoli Monitoring 5.1 is installed on a gateway. The heartbeat engine runs as the process tmnt_hb_eng. There are a number of commands that are used to configure and control the heartbeat function: 򐂰 wdmconfig: This command lets you amend various aspects of the configuration of the IBM Tivoli Monitoring 5.1 components at a gateway. The file is located in the $BINDIR\bin directory. Some of the parameters that are specific to the heartbeat are: – – – – –

heartbeat.send_events_to_tbsm (default=false) heartbeat.send_events_to_tec (default=false) heartbeat.tec_server (default=[null]) heartbeat.send_events_to_notice (default=true) heartbeat.reboot_engine_if_down (default=false)

These five parameters are needed to configure the heartbeat. Refer to the IBM Tivoli Monitoring Version 5.1 User’s Guide, SH19-4569, for more information. 򐂰 wdmheartbeat: This command stops or starts the heartbeat function on gateways. It also configures the frequency and queries the status of the heartbeat processor. 򐂰 wdmmn: This command stops and starts the following processes on the gateway: – Task engine – Tivoli Business Systems Manager adapter – Heartbeat engine (stop only) Note: The heartbeat engine can be stopped, but not restarted, with the wdmmn command. After stopping the engine it is necessary to use the wdmheartbeat command to reconfigure the frequency and restart the heartbeat engine.

򐂰 wdmmngcache: This command can perform a number of functions, one of which is to retrieve the endpoint cache from a gateway.

172

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 7-4 illustrates the flow of data when heartbeat controlling commands are issued.

Figure 7-4 Heartbeat configuration and control data flow

The following steps were carried out in our environment to set up the heartbeat function. 1. To configure the heartbeat events to be sent to the Tivoli Enterprise Console server and the notice group as well as enabling auto-recovery if an engine is discovered to be down, we issued the command: wdmconfig -m all -D heartbeat.send_events_to_tec=true -D heartbeat.tec_server=EventServer -D heartbeat.send_events_to_notice=true -D heartbeat.reboot_engine_if_down=true

The above parameters are updated in the $DBDIR/dmml/.config file (%DBDIR%\dmml for Windows systems).

Chapter 7. Heartbeat

173

2. Configure the frequency (300 seconds) and start the monitoring of endpoints: wdmheartbeat -m all -s 300

This starts the heartbeat engine on the gateway. To check, issue ps -ef | grep tmnt_hb_eng on UNIX or ntprocinfo | grep tmnt_hb_eng on Windows. The frequency is also updated in the $DBDIR/dmml/.config file (%DBDIR%\dmml for Windows systems). This completes the setup of the environment. To view the status of the endpoint cache on the gateway, we issued the command: wdmmngcache -m all -l -v

This retrieves the endpoint cache and the status of each endpoint, resource model, and Tivoli Monitoring engine status, as shown in Example 7-1. Example 7-1 Retrieving heartbeat status tividc11:/> wdmmngcache -m all -l -v Processing ManagedNode tividc12... Processing ManagedNode tividc11... Processing ManagedNode itwin1... Endpoint | HB status | TBSM status -----------------------------------------+----------------------+-------------------itso21 Alive Not discovered itso27 Alive Not discovered itsodev1 Unreachable Not discovered tividc12 RMsInError Not discovered tividc15 Alive Not discovered linux390 Alive Not discovered itsodev3 Alive Not discovered

To view whether the heartbeat engine is stopped or started, as well as the current heartbeat frequency, you can issue the wdmheartbeat command, as shown in Example 7-2 on page 170. Example 7-2 Retrieving the heartbeat engine status and interval tividc11:/> wdmheartbeat -m all -q Processing ManagedNode tividc12... HeartBeat processor status: STARTED, time interval: 300 Processing ManagedNode tividc11... HeartBeat processor status: STARTED, time interval: 300 Processing ManagedNode itwin1... HeartBeat processor status: STARTED, time interval: 300

174

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

7.3 TEC event classes and rule sets The following section describes the default TEC event classes and rule set behavior for the heartbeat function. It also includes some items that may be of concern, and customizations you may wish to make. The following TEC event classes can be generated by the heartbeat function: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

HeartBeat_Off HeartBeat_EndpointUnreachable HeartBeat_DMAgentDown HeartBeat_DMAgentAlive HeartBeat_ResourceModelsInError HeartBeat_DMAgentRestarted HeartBeat_EndpointMigrated

In all of the following examples, the host name slot is the endpoint label being monitored, while the adapter_host slot is the ManagedNode/gateway that generated the given event.

7.3.1 HeartBeat_Off This event indicates that the heartbeat has been turned off. See Example 7-3. Example 7-3 Heartbeat_Off example HeartBeat_Off; severity='HARMLESS'; origin='9.3.240.72'; hostname='tividc15'; adapter_host='tividc11'; msg='The heart-beat checking has turned off.';

7.3.2 HeartBeat_EndpointUnreachable This event indicates that the heartbeat discovered an endpoint that is not communicating via the Framework. See Example 7-4. Example 7-4 HeartBeat_EndpointUnreachable example HeartBeat_EndpointUnreachable; severity='CRITICAL'; origin='9.3.240.72'; hostname='tividc15'; adapter_host='tividc11'; msg='Tivoli lcf endpoint is unreachable.';

Chapter 7. Heartbeat

175

7.3.3 HeartBeat_DMAgentDown This event indicates that the endpoint can be reached via the Framework; however, the Tivoli Monitoring engine is stopped or not functioning. See Example 7-5. Note that the heartbeat function can be configured to attempt an automated recovery of the engine by specifying heartbeat.reboot_engine_if_down=true within the wdmconfig command. Example 7-5 HeartBeat_DMAgentDown example HeartBeat_DMAgentDown; severity='CRITICAL'; origin='9.3.240.72'; hostname='tividc15'; adapter_host='tividc11'; msg='The Tivoli Monitoring engine was stopped or it is not working properly.';

7.3.4 HeartBeat_DMAgentAlive This event indicates that the Tivoli Monitoring engine on the specified endpoint is functioning normally. See Example 7-6. Example 7-6 HeartBeat_DMAgentAlive example HeartBeat_DMAgentAlive; severity='HARMLESS'; origin='9.3.240.71'; hostname='tividc14'; adapter_host='tividc11'; msg='The tivoli Monitoring engine is alive.';

7.3.5 HeartBeat_ResourceModelsInError This event indicates that one or more resource models are not functioning properly on the specified endpoint. A list of failing resource models, as well as the error state, are provided within the msg field. Example 7-7 was generated by distributing a logical disk resource model to a Windows 2000 Server that did not have the disk performance counters enabled (diskperf -y), resulting in a missing prereq failure. Example 7-7 HeartBeat_ResourceModelsInError example HeartBeat_ResourceModelsInError; severity='CRITICAL'; origin='9.3.240.71'; hostname='tividc14'; adapter_host='tividc11';

176

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

msg='The following resource models are not running properly: test3#tividc11-region@TMW_LogicalDisk@Missed Prereq'; rms_in_error=['test3#tividc11-region@TMW_LogicalDisk@Missed Prereq'];

7.3.6 HeartBeat_DMAgentRestarted This event indicates that the heartbeat discovered a failed Tivoli Monitoring engine and was able to successfully restart the engine. See Example 7-8. In order for this event to be generated, the heartbeat must be configured to attempt recovery by specifying heartbeat.reboot_engine_if_down=true within the wdmconfig command. Example 7-8 HeartBeat_DMAgent_Restarted example HeartBeat_DMAgentRestarted; severity=''; origin='9.3.240.72'; hostname='tividc15'; adapter_host='tividc11'; msg='The Tivoli Monitoring engine was restarted.';

Important: In the version of code being tested, this event was missing the severity slot value. As a result, it will not be visible on the TEC Console and causes the following error within TEC (seen by issuing a wtdumprl command on the TEC Server): PARSING_FAILED~'Line 2: Cannot parse value: does not match enum definition'

Internal defect #26280 was opened in regards to this issue. No workaround has been found at the time of writing.

7.3.7 HeartBeat_EndpointMigrated This event indicates that the specified endpoint has migrated to another gateway. See Example 7-9. Example 7-9 HeartBeat_EndpointMigrated example HeartBeat_EndpointMigrated; severity='HARMLESS'; origin='9.3.240.71'; hostname='tividc14'; adapter_host='tividc11'; msg='Tivoli lcf endpoint migrated to another gateway.';

Chapter 7. Heartbeat

177

7.3.8 Heartbeat-related TEC Rules The rule set (hb_events.rls) provided by Tivoli consists of a single, rather simplistic, rule that closes all current events for a given endpoint upon the arrival of a new event. This rule set ensures that only the most current event for a given endpoint is visible on the TEC Console. Many customers will want to do a more sophisticated correlation of events. Some of our suggested improvements include: 򐂰 HeartBeat_DMAgentAlive events should be used exclusively for correlation. Closing or reducing the severity of currently open events stating the endpoint is down would be the most common correlation. It is typically not a best practice to send “everything is fine” events to the TEC Console. 򐂰 HeartBeat_EndpointMigrated events should be considered for correlation only, unless the migration of endpoints is considered an event worthy of action in your environment.

178

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

8

Chapter 8.

Web Health Console This chapter discusses the IBM Tivoli Monitoring 5.1 Web Health Console. We give an overview of the Web Health Console and describe the functionality. The following topics are discussed: 򐂰 򐂰 򐂰 򐂰

8.1, “Overview of the Web Health Console” on page 180 8.2, “Web Health Console components” on page 180 8.3, “Defining health” on page 182 8.4, “Using the Web Health Console” on page 184

© Copyright IBM Corp. 2002

179

8.1 Overview of the Web Health Console The Web Health Console is a Web-based graphical interface that displays the state information about the deployed resource models and aggregated health information on endpoints. You can use the Web Health Console to monitor real-time data or historical data. The distributed profile must be configured to collect historical data for that data to be accessible by the Health Console. The Health Console follows a Model-View_controller architecture for retrieval and showing of performance data. Each request is executed in a different thread. The exchange of information among the main objects inside the Health Console is realized through an event-driven architecture. The data that is gathered using either the real-time or historical methods is retrieved using XML. The console displays activity of the Analyzer and Aggregator components of the new agent technology. The console operates independently of the TME environment but is still validated and authenticated through the TME. You do not need to have an endpoint on the machine that you are running the Health Console from. When you start the Health Console, you are prompted to log in to the Tivoli management region server or ManagedNode using a valid Tivoli user ID. The Health Console feature has been available since Tivoli Distributed Monitoring for Windows 3.7, when it was developed as a stand-alone Java Client. The Java Health Console for Tivoli Distributed Monitoring (Advanced Edition) 4.1 works with IBM Tivoli Monitoring 5.1. In IBM Tivoli Monitoring 5.1, the Health Console is now Web-based only, and utilizes the following components, which are installed automatically during the installation of the Web-based Health Console, as discussed in 4.5, “Installing Web Health Console” on page 105: 򐂰 WebSphere Application Server, Advanced Edition Single Server 4.0.2 򐂰 IBM HTTP Server

8.2 Web Health Console components The Web Health Console consists of the following components: 򐂰 Client browser console interface 򐂰 WebSphere-based Web Health Console component 򐂰 IBM Tivoli Monitoring TMR monitoring components The high-level data flow of the Web Health Console is shown in Figure 8-1 on page 181.

180

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

HTTP/HTTPS HTTP JCF/CORBA

IBM HTTP Server Client Browsers

Tivoli Managed Region

WebSphere Application Server with Web Health Console Application

Figure 8-1 High-level data flow of the Web Health Console

The Web Health Console has the following features: 򐂰 Very easy startup for end users – Simple one Web site access. – No installation for individual users (no space needed). – A lower powered end-user machine can be used for the Web Health Console. – Low data transfer: Only data that is absolutely necessary is transferred to the Web Health Console browser. The Web Health Console server offloads a significant amount of data processing. 򐂰 Single-point upgrade Upgrades are made only at the Web Health Console server. 򐂰 Uses standard IBM Web technology – The WebSphere Application Server is the underlying application technology. – Allows remote Web-based management of the Web Health Console server.

Chapter 8. Web Health Console

181

8.3 Defining health In the context of the Web Health Console, the health of a resource is defined as a numerical value between 0 and 100. A value of zero means that an event has been generated and a value of 100 indicates that the monitored resource is normal. Intermediate values indicate that one or more indications have been generated. The value is calculated by the number of occurrences (indications) and holes (the lack of indications) in the preceding cycle time. An occurrence is the term used to refer to a cycle during which an indication occurs for a given resource model. Hole is the term used to refer to a cycle during which an indication does not occur. If we define an event as three occurrences and two holes, then the maximum cycles required to generate an event would be seven. For every cycle, if a threshold is exceeded, then the count of occurrences would increase. If a threshold is not exceeded, then the count of occurrences would not increase. The important thing to remember is that if the event is defined with two holes, then if we receive three consecutive holes, the count of occurrences is reset to 0. If we receive two or less holes the occurrences are considered consecutive. Table 8-1 illustrates this situation. Table 8-1 Defining health Sequence

Count of occurrences

100

1 occurrence

1001

2 occurrences

10010

2 occurrences

100100

2 occurrences

1001000

0 occurrences, as more than two consecutive holes have occurred

1001001

3 occurrences

In the above examples the maximum number of cycles that would be required to generate an event would be seven: 1 0 0 1 0 0 1

The health values are calculated by dividing 100 by the number of occurrences required to generate an event. So in this case, the health value for one occurrence is 100/3=33.3 Table 8-2 on page 183 now includes the health for each cycle.

182

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Table 8-2 Health values Sequence

Count of occurrences

Health

100

1 occurrence

66

1001

2 occurrences

33

10010

2 occurrences

33

100100

2 occurrences

33

1001000

0 occurrences, as more than two consecutive holes have occurred

100

1001001

3 occurrences, an indication is generated

0

The Web Health Console obtains events and indications from endpoints. The Web Health Console displays the health of each potential problem as a numeric value between 100 (perfect health) and zero (with zero meaning that the conditions for the corresponding event are met). Intermediate values show the percentage of occurrences currently registered with respect to the total number of occurrences needed to trigger an event. Table 8-3 gives the health percentage changes in steps of 25 percent because four occurrences were required to trigger an event. If the indication required five occurrences, the health percentage would have changed by steps of 20 percent. Resource health is determined at the indication level and passed up to the endpoint. The lowest health of any indication in a resource model is shown as the health of that resource model, and the lowest health of any resource model installed on an endpoint is shown as the health of that endpoint. For example, if one indication on one resource model that is installed on an endpoint has a health of zero, the health of the endpoint is shown as zero. The required occurrences, cycle times, thresholds, and parameters for indications are defined when the resource model is created in the IBM Tivoli Monitoring Workbench. Table 8-3 Health determination example Cycle

1

2

3

4

5

CPU percent

55

73

54

63

68

Occurrences or holes

H

O

H

O

O

Occurrence count

0

1

1

2

3

Health percent

100

75

75

50

25

Chapter 8. Web Health Console

183

8.4 Using the Web Health Console The relationship between the various screens in the Web Health Console is shown in Example 8-2. The Web Health Console is particularly useful for monitoring and diagnosing problems on those endpoints for which a Tivoli Enterprise Console event has already been sent. Both historical and real-time data can be analyzed to further diagnose potential problems. We describe how to view online data as well as historical data in the following sections. Before we discuss viewing data, we first need to set up the Health Console with the endpoints from our environment.

Login

Resource Model List View

Endpoint List View

Endpoint List With Resource Models

Endpoint View

Resource Model History

Resource Model Indications

Graph View

Figure 8-2 Web Health Console screens

184

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

8.4.1 Logging on to the Web Health Console You can connect the Web Health Console to any Tivoli management region server or ManagedNode and configure it to monitor any or all of the endpoints that are found in that region (assuming that all gateways are interconnected). To connect to the Web Health Console you need access to the server on which the Web Health Console server is installed and the IBM Tivoli Managed Region on which you want to monitor health. All user management and security is handled through the IBM Tivoli management environment. This includes creating users and passwords as well as assigning authority. The Web Health Console is started by launching Internet Explorer or Netscape, and browsing to http://server_name/dmwhc, where server_name is the host name of the server on which you have installed the Web Health Console Server. When you browse to this URL, you are required to enter a valid Tivoli administrator ID to log in to the Health Console. You will be prompted to enter a user ID, password, and the host name of your Tivoli Management Region server or ManagedNode, as shown in Figure 8-3 on page 186. The user ID you specify does not require any special Tivoli role to successfully log on to the Web Health Console. However, to operate the Web Health Console, the user requires TMR roles of user or admin, as discussed in this section later on.

Chapter 8. Web Health Console

185

Figure 8-3 Web Health Console login panel

When you are successfully logged in the first time, you are presented with the Preferences view, as shown in Figure 8-4 on page 187. At this point, you are required to import endpoints that you need to monitor. The Web Health Console can only monitor endpoints that are defined in the Tivoli management region that it is connected to.

186

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-4 Web Health Console Preferences -> Endpoint view

To populate the endpoint list: 1. Type the name of an endpoint filter in the Filter field. For example, type b* to display all endpoints in the Tivoli-managed region with names that start with b and that have resource models installed. The filtering capability support is limited to the (*) regular expression character. 2. Click Go. 3. Click the endpoint to select it from the Available endpoints list. 4. Use Control+click or Shift+click to select multiple endpoints. 5. Click Add. 6. Repeat these steps to add more endpoints to the list.

Chapter 8. Web Health Console

187

Note: You can apply a new filter at any time to update the Available endpoints list without affecting the Selected endpoints list. You can move endpoints from the Selected endpoints list to the Available endpoints list by using the Remove button. 7. Click Save to save the Selected endpoint list or Cancel to cancel these changes. If the Tivoli administrator has the admin TMR role, then she will be able to select endpoints, and in this case the user TMR role is not required If the Tivoli administrator you are logged in as does not have the user TMR role, and also does not have the admin TMR role, then you will not be able to connect to any endpoints, as shown in Figure 8-5 on page 189

188

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-5 Tivoli Administrator with no user Tivoli role

When you log on subsequently, the endpoint list is automatically loaded. As shown in Figure 8-6 on page 190, the General option on the Preferences page can be used to configure the Refresh Interval, Default View, and Optimize Cache settings, which will be used by the Web Health Console.

Chapter 8. Web Health Console

189

Figure 8-6

Web Health Console Preferences General view

As shown in Figure 8-7 on page 191, when you are sucessfully logged in the first time, you are presented with the Chart view, which specifies the type that is displayed when you click Graph in the Endpoint view.

190

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-7 Web Health Console Preferences Chart view

8.4.2 The Web Health Console main view When you start the Web Health Console, the top portion of every view of the Web Health Console, except the Login view, contains a common menu bar, as shown in Figure 8-8 on page 192. This navigation bar has fly-over help associated with each button.

Chapter 8. Web Health Console

191

Figure 8-8 Web Health Console common menu bar

8.4.3 The Endpoint List View The Endpoint List View, as shown in Figure 8-9 on page 193, shows the current health of all the endpoints specified in the Endpoints tab of the Preferences view. (See Figure 8-4 on page 187 for more information on preferences.)

192

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-9 Endpoint List View

The Web Health Console sorts the endpoints by health order, with the lowest health displayed first. For example, an endpoint with a health percentage of 0 is listed before an endpoint with a health percentage of 70. If there is a problem contacting an endpoint, the Web Health Console displays a message indicating the problem. The Endpoint List View table provides information on the status of the specific endpoint being monitored (Running or Engine could not be reached) and the lowest health of all the resource models installed on the endpoint. For example, if the endpoint has two resource models installed and one is at 50 percent health and the other is at 90 percent health, this column displays 50. The health is displayed as an exact health percentage and as an icon representation of possible states of alert.

Chapter 8. Web Health Console

193

The sample icons for the endpoint health are shown in Figure 8-10. Reading the icons in Figure 8-10 from left to right, the first icon shows the health of all of the resource models installed on the endpoint at 100 percent. The second icon shows the health of at least one of the resource models installed on the endpoint as less than 100 percent but greater than 0. The third icon shows the health of at least one of the resource models installed on the endpoint at 0.

Figure 8-10 Icons showing the Endpoint Health

If the IBM Tivoli Monitoring 5.1 engine is switched off on a particular endpoint, then the Web Health Console will report this. To start or stop the engine, the user who is logged in must have the TMR admin role. Otherwise a message will be displayed as shown in Figure 8-11.

Figure 8-11 Web Health Console showing that the engine is unreachable

194

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

We can then drill down to get further information on which resource models are running on the endpoint shown by clicking the endpoint name, as shown in Figure 8-12.

Figure 8-12 Web Health Console and the endpoint resource model information

Chapter 8. Web Health Console

195

To display the health of a specific endpoint, or group of endpoints, type the appropriate information in the Filter field and click Submit. For example, type custxyz-* to display all endpoints in the Selected endpoints list with names that start with custxyz- and that have resource models installed. The filtering capability support is limited to the (*) regular expression character. To start or stop the IBM Tivoli Monitoring 5.1 engine, the Select an action drop-down list can be used, as shown in Figure 8-13.

Figure 8-13 Starting or stopping the IBM Tivoli Monitoring 5.1 engine

To successfully stop or start the engine, the administrator must have the admin, super, or senior role.

196

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

8.4.4 The Resource Model List View The Resource Model List View, as shown in Figure 8-14, shows all of the resource models installed on the endpoints specified in the Endpoint List page of the Preferences view. (See Figure 8-6 on page 190 for more information on preferences.) The Web Health Console sorts the resource models by health order, with the lowest health displayed first. For example, a resource model with a health percentage of 30 is listed before a resource model with a health percentage of 70.

Figure 8-14 Resource Model List View

We can drill down a particular resource model and get a list of endpoints on which the model is running.

Chapter 8. Web Health Console

197

As shown in Figure 8-15, a resource model can be stopped, started, and even the profile can be removed from the selected endpoints.

Figure 8-15 Managing the resource model on an endpoint

198

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

If an indication has been generated, than a online graph can be displayed with the data relevant to the indication by selecting the indication, as shown in Figure 8-16.

Figure 8-16 Viewing online data for an indication

The resulting graph displayed when the Graph button is clicked is shown in Figure 8-17 on page 200. This graph will be continuously updated.

Chapter 8. Web Health Console

199

Figure 8-17 Online data for indications generated by a resource model

By clicking the Historical Data radio button in the Resource Models frame, as shown in Figure 8-16 on page 199, you will be able to view the historical data generated by this resource model on the endpoint. The bottom portion of the screen will change, allowing you to choose which data you are interested in, as shown in Figure 8-18 on page 201.

200

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-18 Historical data options

Chapter 8. Web Health Console

201

Using the historical data selection controls, you select instances and metrics from the selected resource model to create a chart of recent historical data from the endpoint. To create historical data, logging must be enabled for the resource model. With logged data, you can use the historical data graph to identify specific instances of resource problems over the past one, six, twelve, or twenty-four hours. To create a historical data graph as shown in Figure 8-19 on page 203, use the following steps: 1. Select a resource model from the Resource Model list in the upper frame. 2. Click the Historical Data radio button to display the Historical Data selection information in the lower frame. 3. Click the Resource drop-down list and select a resource. The Resource drop-down list is the only option that is active when the frame opens. 4. Click the Contexts drop-down list and select a context. Each context identifies a logical grouping of problems related to the specified resource. 5. Select one or more instances from the Instances list. These identify specific instances of the selected indication. 6. Select one or more metric from the Metrics list. These are the metrics used to measure the selected indication. 7. Click Graph.

202

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 8-19 Sample historical data graph

Chapter 8. Web Health Console

203

The historical data graphs can be displayed in various formats, as shown in Figure 8-20.

Figure 8-20 Available graph layouts

204

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

9

Chapter 9.

Configure and maintain IBM Tivoli Monitoring Version 5.1 In this chapter we describe how to configure and use IBM Tivoli Monitoring Version 5.1 and provide some hints to maintain your IBM Tivoli Monitoring Version 5.1 environment operation and health. The following topics are covered: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

9.1, “Configuring IBM Tivoli Monitoring Version 5.1” on page 206 9.1.1, “Creating and customizing profiles” on page 206 9.1.2, “Profile distribution” on page 220 9.2, “Maintaining your environment” on page 229 9.2.1, “IBM Tivoli Monitoring Version 5.1 command line” on page 230 9.2.2, “Heartbeat maintenance” on page 237 9.2.3, “Directories” on page 238 9.2.4, “Logs and files” on page 240

© Copyright IBM Corp. 2002

205

9.1 Configuring IBM Tivoli Monitoring Version 5.1 In this section we discuss the configuration and usage of the IBM Tivoli Monitoring Version 5.1 product.

9.1.1 Creating and customizing profiles This section discusses the creation and customization of profiles in IBM Tivoli Monitoring Version 5.1. We assume that you understand Framework concepts such as profile, dataless and database profile managers, policy regions, and so on. Refer to Chapter 3 of the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569, for an overview of the above concepts. We will not discuss naming conventions and standards for policy regions, profiles, and profile managers. We also demonstrate most of the concepts using the IBM Tivoli Monitoring Version 5.1 profile GUI. Once you understand the GUI and the various parameters that can be modified, it will then be easier to use the command line option to automate bulk configurations of profiles for your environment.

Setting managed resources Each policy region maintains a list of managed resource types that are defined for that specific policy region. You must add ProfileManager and Tmw2kProfile as managed resources before you can create these entities and work with them. The following steps were performed to set managed resources in our environment: 1. From within the policy region pr.tividc-11.itm click Properties -> Managed Resources from the menu. 2. The Set Managed Resources dialog is displayed. Check that the Tmw2kProfile and profile manager resources are displayed in the Current Resources list. If not, move them from the Available Resources list to the Current Resources list, as shown in Figure 9-1 on page 207. 3. Click Set & Close. For more information about managed resources, see the Tivoli Framework Version 3.7.1 User's Guide, GC31-8433.

206

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 9-1 Setting managed resources

Creating profiles The following steps were carried out to create the IBM Tivoli Monitoring Version 5.1 profiles: 1. From within the policy region pr.tividc-11.itm select Create -> Profile Manager. 2. The Profile Manager dialog is displayed. Enter a name and make sure the Dataless Endpoint Mode check box is selected, as shown in Figure 9-2 on page 208. 3. Select Create & Close.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

207

Figure 9-2 Creating profile manager

4. Double-click the profile manager to open it and select Create Profile. 5. From the Create New Profile dialog, select Tmw2kProfile and give it a name, as shown in Figure 9-3 on page 209. 6. Select Create & Close.

208

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 9-3 Profile creation

Adding resource models You can add resource models to a profile in two ways: 򐂰 Default (Add with Defaults button) 򐂰 Customized (Add button)

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

209

To add a customized resource model to our IBM Tivoli Monitoring Version 5.1 profile, we performed the following steps: 1. Double-click the IBM Tivoli Monitoring Version 5.1 profile. This will open up the empty IBM Tivoli Monitoring Version 5.1 profile configuration panel, as shown in Figure 9-4.

Figure 9-4 Empty Tmw2kProfile

2. Select Add. This will bring up the Add Resource Models to Profile dialog box, as shown in Figure 9-5 on page 211. 3. Select the category and the resource model from the drop-down menus. We selected UNIX-Linux as the category and Memory as the resource model.

210

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

You can now customize all the out-of-the-box parameters that are set, based on your monitoring requirements for your environment.

Figure 9-5 Add/configure resource model

Customizing the thresholds and cycle time When we add the resource model, default cycle time, threshold name, and default threshold, values for the resource model are displayed. The cycle time is the regular interval at which a resource model is required to collect data. This value can be changed. The threshold and their values can also be modified. Threshold values are compared to system resource values that are provided by the CIM-compliant data providers. The cycle time and the thresholds would both be used by the analyzer to determine how often to run through the decision tree and to what values the system resource data should be compared against.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

211

򐂰 To change the cycle time, fill in a new value in the Cycle Time box. When you are finished with all the customization, and select Add & Close, the new cycle time will be added to the profile. 򐂰 To change the thresholds, select the threshold. Change the value of the threshold and click Apply, as shown in Figure 9-6. As mentioned earlier, an indication is made up of a number of thresholds. So before modifying the default threshold values, it is important to understand the relation between the thresholds and indication.

Default Cycle Time

Default Threshold

Figure 9-6 Threshold and cycle time

Customizing the indications The second component that can be modified is the indications. Select the Indications button; this will bring up the Indications and Actions dialog box, as shown in Figure 9-7 on page 213.

212

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Available indications in the Resource Model Unix -> Memory Action List, Built-in Tasks or Framework Tasks

Figure 9-7 Configuring indications

We can configure the following: 򐂰 For each indication, we can configure the number of occurrences and holes that have to be met before an event is generated. 򐂰 We can configure where events are to be sent to, that is, Tivoli Enterprise Console and/or Tivoli Business Systems Manager. We can also change severity levels of the events. 򐂰 Built-in tasks are tasks that the certain resource models are configured to execute when an event is generated. We can enable and disable these tasks from here. 򐂰 If you have a requirement to run an external script, you can create a task to run the script and then add it to the resource model to be called when an event is generated.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

213

Built-in tasks IBM Tivoli Monitoring Version 5.1 provides, in the IBM Tivoli Monitoring Utility Tasks library, two new responses that can be used as actions when configuring the action list. 򐂰 dm_mn_send_email requires the parameter e-mail address; multiple e-mail addresses should be separated by commas, as shown in Figure 9-8.

Figure 9-8 Task dm_mn_send_email

򐂰 dm_mn_send_notice requires the parameters Notice Group and Priority, as shown in Figure 9-9 on page 215.

214

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 9-9 Task dm_mn_send_notices

Note: Both tasks are executed on the ManagedNode/gateway where the target endpoint is logged on. You can check the task execution on the ManagedNode using the odstat command.

Customizing the scheduler The third component is the scheduler. By default, the scheduler is configured to collect monitoring at all times. If you have a requirement to monitor certain types of devices during predefined time periods, you can configure the scheduler to monitor a certain endpoint based on your requirements (see Figure 9-10 on page 216). To schedule monitoring: 1. Select the Scheduler button from the Add Resource Model panel. 2. Uncheck the Always check box and enter your scheduling requirements. 3. Select Modify & Close.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

215

For setting the data collection period. By default, all resource models are set to always collect data.

The rules for the different schedule requirements

Figure 9-10 Scheduler customization

Customizing the data logging The fourth component that can be configured is the logging. What logging allows is to capture performance data and store it locally on the endpoint. This data can then be viewed by the Web Health Console and it can be stored in a central RDBMS for further analysis. By default, no performance data is logged. The data that is logged can be kept for up to 24 hours. To log data: 1. Click the Logging button from the Add Resource model panel, as shown in Figure 9-5 on page 211. 2. From the Logging dialog box, shown in Figure 9-11 on page 217, select the Enable Data Logging check box.

216

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

This will store the raw data for the resource models on the endpoint. To aggregate the data, select the Aggregate Data check box and provide the aggregation period. You also need to specify how long to keep the historical data before it is deleted from the database. The database is local to the endpoint. This data can be rolled up to a central RDBMS and, using Tivoli Enterprise Data Warehouse, you can report on the data. Data can be just resource properties or the result of a calculation. This check box enables the resource model to effectively log the desired data.

Data to be logged is cached and aggregated at fixed intervals you define (Aggregation Period). Then only the aggregated values are written in the database.

To specify the period for which data is to be stored in the database

Figure 9-11 Data logging panel

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

217

Note: On Windows endpoints, the data logging records information in the file $LCF_DATDIR/LCFNEW/Tmw2k/db/tmw2kdb.mdb. To provide ODBC access to this file, the system data source name tmw2k is created. On UNIX endpoints, the data logging records information in the file $LCF_DATDIR/LCFNEW/Tmw2k/Unix/data/logger/dblogger/datafile. The data files are created when there is a need to log data.

Customizing parameters Parameters are configurable for certain type of resource models. If we look at the UNIX Processes resource model, it contains processes to monitor. By default, it monitors the syslogd and the lcfd processes. If we require monitoring of additional processes, we can add them here, as shown in Figure 9-12 on page 219. To add other processes: 1. 2. 3. 4. 5.

218

Select the category UNIX-Linux and Processes. Click Parameters on the Profile configuration panel. This will bring up the Parameters Configuration dialog box. Select a valid process to be monitored and click the Add button. Select Apply Changes & Close.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 9-12 Changing parameters of resource model

Customizing the profile From the Profile Configuration dialog menu bar, select Edit -> Properties, which will open a dialog box, as shown in Figure 9-13 on page 220. This option configures whether events will be sent to Tivoli Enterprise Console and/or Tivoli Business Systems Manager. You can choose whether the events are to be secure or non-secure. If events are to be forwarded to Tivoli Enterprise Console using a secure connection, you also need to specify the Tivoli Enterprise Console server. If you are using non-secure, then you need to specify the IP address and port of the Tivoli Enterprise Console server.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

219

Figure 9-13 Profile properties

9.1.2 Profile distribution This section covers profile distribution.

MDist2 support IBM Tivoli Monitoring Version 5.1 uses Tivoli Management Framework Multiplexed Distribution service (MDist2) to perform profile distribution. It makes full use of the following MDist2 functions:

220

Asynchronous delivery

MDist2 uses an asynchronous interface to applications, which means that when IBM Tivoli Monitoring Version 5.1 submits a distribution request, it immediately gets back a distribution identifier and confirmation that the distribution is in progress.

Assured delivery

The distribution of IBM Tivoli Monitoring Version 5.1 profiles is assured even when there are network interruptions, power-off on the machines, or disconnected endpoints.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Check-point and restart

A distribution data stream that has been interrupted can be resumed by IBM Tivoli Monitoring Version 5.1 from the last successful check-point. This means that it is not necessary to resend all of the profile data when the distribution is resumed by mdist2, but the IBM Tivoli Monitoring Version 5.1 receiver method requests only the data that had not yet been sent when the interruption occurred.

Data depoting

MDist2 allows the storage of distribution segments at a depot close to the endpoint, and for the distribution to be submitted to the endpoints from that depot.

Note: Depots preloading and loading is done automatically only on first profile distribution. Once the distribution is completed, all the dependencies are on the depots, which means that the next distribution will download the dependencies from depots and not from the TMR Server.

MDist2 setup To set up and configure MDist2, refer to the Tivoli Framework Version 3.7.1 Installation Guide, GC32-0395, for further information. We have configured MDist2 on the host tividc12 using a DB2 database, with the following steps: 1. Create a DB2 database called mdist2. 2. Create the RIM object: wcrtrim -v DB2 -h tividc12 -d mdist2 -u db2inst1 -H /usr/lpp/db2_06_01 -s tcpip -i mdist mdist2

3. Executed the scripts located in $BINDIR/TME/MDIST2/sql/mdist_db2*.sql.

Profile distribution using MDist2 IBM Tivoli Monitoring Version 5.1 only supports profile distribution to ProfileManager, TMA endpoint, Application Proxy, and Application Services. The following options can be configured in a profile distribution. In the Profile Manager dialog box, open a profile to be distributed. Click Profile -> Distribute. The IBM Tivoli Monitoring Version 5.1 Profile dialog opens, as shown in Figure 9-14 on page 223.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

221

The Distribute Profile dialog box opens: Next level of subscribers Distributes the profile only to the subscribers named in the Distribute To these Subscribers scrolling list of the Distribute Profile dialog box. This selection distributes the profile only to the subscribers of the profile manager. It does not distribute to lower-level subscribers. If a profile manager with subscribers resides at the next lower level, you may need to perform the distribution process from profile managers at more than one level to reach all the profile endpoints. All levels of subscribers Distributes the profile to all subscribers in the hierarchy. An example follows to illustrate the difference between distributing to the two levels of subscribers. You have a profile hierarchy in which a dataless profile manager is subscribed to a profile manager, and the dataless profile manager has an endpoint subscribed. If you distribute to the next level of subscribers, the profile is distributed only to the dataless profile manager. If you distribute to all levels of subscribers, the profile is distributed to the dataless profile manager, and to the endpoint. Select this option if you want to distribute a profile in which your endpoint is the only subscriber. Select the Distribution Will option Make subscribers profile an EXACT COPY of this profile. This overwrites the subscribers profile with an exact copy of the profile being distributed. You must always use the Make exact copy option. Select subscribers to receive the profile by choosing them from the Don’t Distribute To These Subscribers scrolling list and moving them to the Distribute To These Subscribers scrolling list, as in Figure 9-14 on page 223. You can also configure the Distribution Defaults option, then every distribution of this profile will be done with the default settings that you provided. Note: In Distributed Monitoring (Classic Edition), the EXACT COPY option distributes all the SentryProfiles within the same ProfileManager. Using IBM Tivoli Monitoring Version 5.1 we could not see this behavior; only the selected Tmw2kProfile is pushed to the distribution target.

222

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 9-14 Profile distribution options

There are three possible ways to perform profile distribution in IBM Tivoli Monitoring Version 5.1, as follow: 򐂰 GUI distribution a. Drag and drop the profile in the target subscriber. b. In the Profile Manager dialog box, open the profile to be distributed. The IBM Tivoli Monitoring Version 5.1 Profile dialog box opens.Click Profile -> Distribute. 򐂰 CLI wdistrib You can use the Framework CLI wdistrib to distribute a IBM Tivoli Monitoring Version 5.1 Profile. wdistrib -l over_all @Tmw2kProfile:pf.unix.itm @Endpoint:vmlinux5

Note: Using the GUI distribution and wdistrib command, the Depot storage status is permanent.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

223

򐂰 New CLI wdmdistrib IBM Tivoli Monitoring Version 5.1 provides a new command based on MDist2, with a richer set of options, to distribute an IBM Tivoli Monitoring Version 5.1 profile to subscribers. In our environment, we used the GUI and the wdmdistrib to push IBM Tivoli Monitoring Version 5.1 profiles. In the following example we explore the use of wdmdistrib and MDist2. In Example 9-1 we distributed the Tmw2kProfile pf.unix.itm to the endpoint itso21. Its gateway/repeater is itwin1, using the command wdmdistrib with the flag -d, to disable the depot storage. Example 9-1 Profile distribution without depot storage # Viewing depot status tividc11:/>wdepot itwin1 describe Depot Location = c:/tivoli/itwin1.db/tmp/depot/ Depot Size = 512000 (KB) Temporary Storage = 0 (KB) Permanent Storage = 0 (KB) Total Storage = 0 (KB) Free Space = 512000 (KB) # Distributing the profile using wdmdistrib tividc11:/>wdmdistrib -p pf.unix.itm -e -w -i -d -J /cdrom/tools/jre itso21 AMW0162I - Operation successfully submitted. Distribution ID is 1931340152.53 # Viewing depot entries tividc11:/>wdepot itwin1 list Name Version DMXMemory@aix4-r1 1.0 15:04:09 DMXMemory@hpux10 1.0 15:04:09 DMXMemory@linux-ix86 1.0 15:04:0 9 DMXMemory@linux-s390 1.0 15:04:0 9 DMXMemory@solaris2 1.0 15:04:09 DMXCpu@aix4-r1 1.0 15:04:09 DMXCpu@hpux10 1.0 15:04:09

224

Type T T

Size (KB) Update time 8 (100%) 2002/04/29 7 (100%) 2002/04/29

T

7 (100%) 2002/04/29

T

7 (100%) 2002/04/29

T

7 (100%) 2002/04/29

T

8 (100%) 2002/04/29

T

6 (100%) 2002/04/29

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

DMXCpu@linux-ix86 15:04:09 DMXCpu@linux-s390 15:04:09 DMXCpu@solaris2 15:04:09 JRE@w32-ix86 15:04:09 JRE@solaris2 15:04:40 JRE@aix4-r1 15:05:11 JRE@linux-ix86 15:05:44 JRE@linux-s390 15:06:18

1.0

T

7 (100%) 2002/04/29

1.0

T

7 (100%) 2002/04/29

1.0

T

7 (100%) 2002/04/29

0.0

T

15401 (100%) 2002/04/29

0.0

T

14929 (100%) 2002/04/29

0.0

T

15821 (100%) 2002/04/29

0.0

T

16862 (100%) 2002/04/29

0.0

T

25392 ( 60%) 2002/04/29

# Viewing depot status, note that the Temporary Storage is in use tividc11:/>wdepot itwin1 describe Depot Location = c:/tivoli/itwin1.db/tmp/depot/ Depot Size = 512000 (KB) Temporary Storage = 104795 (KB) Permanent Storage = 0 (KB) Total Storage = 104795 (KB) Free Space = 407205 (KB) # Viewing the distribution status tividc11:/>wmdist -l -v -i 1931340152.53 Distribution ID: 1931340152.53 Name: pf.unix.itm(install) Owner: [email protected] Size: 104780 Source Application: Distributed Monitoring Source Node: 1931340152.1.579 Start Time: 2002.04.29 15:11:55 Finish Time: Expire Time: 2002.05.02 15:11:53 Update Time: 2002.04.29 15:11:56 Targets: 1 Complete: 0(0%) Waiting: 1 Paused: 0 Ready: 0 Unavailable: 0 Receiving: 0 Interrupted: 0 Sending: 0

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

225

Successful: 0 Failed: 0 Canceled: 0 Rejected: 0 Expired: 0 Stored: 0 tividc11:/>wmdist -e 1931340152.53 -t itso21 Name Status Start Time itso21 SUCCESSFUL 2002.04.29 15:07:45

End Time 2002.04.29 15:07:47

tividc11:/>wdepot itwin1 describe Depot Location = c:/tivoli/itwin1.db/tmp/depot/ Depot Size = 512000 (KB) Temporary Storage = 0 (KB) Permanent Storage = 0 (KB) Total Storage = 0 (KB) Free Space = 512000 (KB)

The depot is only used for temporary storage. You can see this with the command wdepot; the storage type flag is T (temporary). In this way the dependencies are downloaded for every distribution on the depots and, once the distribution finishes, are deleted from depot. In Example 9-2 we have distributed the same profile to the same target, without the flag -d, so depot storage is used. Example 9-2 Profile distribution with depot storage tividc11:/>wdmdistrib -p pf.unix.itm -e -w -i -J /backup/itm/tools/jre itso21 # Viewing depot entries, Permanent Storage Type is in use

tividc11:/>wdepot itwin1 list Name Version DMXMemory@aix4-r1 1.0 15:17:36 DMXMemory@hpux10 1.0 15:17:36 DMXMemory@linux-ix86 1.0 15:17:3 6 DMXMemory@linux-s390 1.0 15:17:3 6 DMXMemory@solaris2 1.0 15:17:36 DMXCpu@aix4-r1 1.0 15:17:36

226

Type P P

Size (KB) Update time 8 (100%) 2002/04/29 7 (100%) 2002/04/29

P

7 (100%) 2002/04/29

P

7 (100%) 2002/04/29

P

7 (100%) 2002/04/29

P

8 (100%) 2002/04/29

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

DMXCpu@hpux10 15:17:36 DMXCpu@linux-ix86 15:17:36 DMXCpu@linux-s390 15:17:36 DMXCpu@solaris2 15:17:36 JRE@w32-ix86 15:17:36 JRE@solaris2 15:18:07 JRE@aix4-r1 15:18:37 JRE@linux-ix86 15:19:10 JRE@linux-s390 15:19:44

1.0

P

6 (100%) 2002/04/29

1.0

P

7 (100%) 2002/04/29

1.0

P

7 (100%) 2002/04/29

1.0

P

7 (100%) 2002/04/29

0.0

P

15401 (100%) 2002/04/29

0.0

P

14929 (100%) 2002/04/29

0.0

P

15821 (100%) 2002/04/29

0.0

P

16862 (100%) 2002/04/29

0.0

P

41696 (100%) 2002/04/29

# Viewing depot status, note that the Permanent Storage is use tividc11:/>wdepot itwin1 describe Depot Location = c:/tivoli/itwin1.db/tmp/depot/ Depot Size = 512000 (KB) Temporary Storage = 0 (KB) Permanent Storage = 104795 (KB) Total Storage = 104795 (KB) Free Space = 407205 (KB)

The data was stored in the repeater depot, so further distribution for targets in the same repeater will download the dependencies from the depot, instead of the TMR Server. You can use the following flags with wdmdistrib: wdmdistrib -p [-D ...] [-e][-w][-i] [-J ][-d][-R][-l] [-s ] [subscriber...]

For further reference to wdmdistrib usage check IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569.

Rerunning the failed distributions When a distribution fails, IBM Tivoli Monitoring Version 5.1 creates a profile manager containing the endpoint subscribers that failed. To see the profile managers, using the desktop go to the policy region where the original profile manager exists and select View -> Refresh.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

227

The failed profile manager name is derived as follows: OriginalProfileName _Push_Failed_Bad_Interpreter

Error message AMW089E might be displayed at this point, indicating that the resource model type is not compatible with the endpoint operating system. For example, you might have distributed a Windows resource model to a UNIX endpoint, or vice versa. If the distribution fails because of any other error, then the profile manager name is derived as follows: OriginalProfileName _Distribution_Failed

Provided that the profile manager that you used for the original distribution was created without checking the Dataless endpoint mode option, you can use these profile managers to redistribute the profile to the failed endpoints when you have fixed the problem that caused the original failure. To do this, just subscribe the profile managers that contain the failed endpoints to the profile manager that contained the original profile. You can then distribute the original profile to the failed endpoints by selecting these profile managers as the target for the distribution. The profile managers can also be edited to delete an endpoint from a group of failed endpoints before retrying the distribution.

Profile removal We observed some issues when deleting or unsubscribing endpoints from the profile manager that contains Tmw2kProfiles. After a Tmw2kProfile deletion, the profiles copies distributed to the endpoints still remain. Unsubscribing a endpoint, with the option Delete all profile copies, from the profile manager keeps the profiles copies on the endpoint. This is a known problem and is registered as defect 27469. To remove the Tmw2kProfile copy distributed to a endpoint, you must use the command wdmeng, as shown in Example 9-3. Example 9-3 Removing a Tmw2kProfile copy tividc11:/>wdmeng -e itso21 -p pf.unix.itm#tividc11-region -delete Forwarding the request to the engine... The operation has been successfully completed.

Important: To remove the Tmw2kProfile copies from the endpoint, you must use the command wdmeng.

228

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

TMF MDist2 commands The TMF commands wmdist and wdepot provide many options to check and manage distribution packages. The following options can be useful to see IBM Tivoli Monitoring Version 5.1 profile distribution status: 1. List distributions: wmdist –l [-I dist_id] [-a] [-v]

-I: Specific distribution ID. -a: Shows only active distributions. -v: Verbose output. 2. Endpoint distribution status: wmdist –e dist_id [-t endpoint_id]

-t: Status for specific endpoint. 3. Cancel distributions: wmdist –c dist_id [endpoint…]

4. Pause distributions: wmdist –p dist_id [endpoint…]

5. Resume distributions: wmdist –r dist_id [endpoint…]

6. List repeater depot: wdepot repeater_name list -l

7. View repeater depot statistics: wdepot repeater_name describe

For more information about MDist2 commands, refer to Tivoli Framework Version 3.7.1 Installation Guide, GC32-0395.

9.2 Maintaining your environment In this section we describe the new command line, directory structure, and log files to be monitored. The maintenance of the Framework environment is not described in this chapter; however, you can find a good reference in Maintaining Your Tivoli Environment, SG24-5013.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

229

9.2.1 IBM Tivoli Monitoring Version 5.1 command line If you are a user of Tivoli Distributed Monitoring for Windows, you will be familiar with the old commands issued with that version, which had the prefix wtmnt. In IBM Tivoli Monitoring Version 5.1, all commands have the prefix wdm. The following steps have been taken to assist the transition from the old command prefixes to the new prefixes: 򐂰 Five of the old command names have been included with this release of the product as aliases for the new equivalent commands. This gives you time to migrate your scripts. 򐂰 Three of the old commands (wtmntaddrm, wtmntdefrm, wtmntrmrm) have been merged into one new command (wdmrm). The old commands have been made available with this release, with their old unchanged syntax and options. This gives you time to migrate your scripts. 򐂰 Eight new commands have been added, for which aliases with the old prefix are not supplied. IBM Tivoli Monitoring Version 5.1 provides some new commands to be used. Table 9-1 shows some commands, comparing them with old versions and the classic edition. Table 9-1 IBM Tivoli Monitoring Version 5.1 CLI IBM Tivoli Monitoring Version 5.1 CLI

Tivoli Distributed Monitoring for Windows 3.7 CLI

wdmcmd

wtmntcmd

Distributed Monitoring (Classic Edition) CLI

wdmconfig wdmdiscovery wdmdistrib wdmeng

wdistrib wtmnteng

wdmdumpprf

wdumpsnt

wdmloadprf

wloadsnt

wdmeditprf

wsetmon

wdmheartbeat wdmlseng

wtmntlseng

wdmmn

wtmntmn

wdmmngcache

230

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

wlseng

IBM Tivoli Monitoring Version 5.1 CLI

Tivoli Distributed Monitoring for Windows 3.7 CLI

wdmrm

wtmntaddrm,wtmntdefrm, wtmntrmrm

wdmtrceng

wtmnttrceng

Distributed Monitoring (Classic Edition) CLI

For a complete guide about commands, refer to IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569.

Maintenance commands In this section, we explain, with examples, some of the maintenance commands provided with IBM Tivoli Monitoring Version 5.1.

wdmconfig This creates and updates the configuration file ($DBDIR/dmml/.config) on a ManagedNode, it accepts the following parameters: 򐂰 transport.server.ip.addressIP IP address or host name of the CommonListener component of Tivoli Business Systems Manager, which listens for messages from the systems being managed. 򐂰 transport.server.mqe.port Port number of the CommonListener component. 򐂰 adapter.working.dir Working directory that will be used by the adapter. The default, which you are recommended to use, is the IBM Tivoli Monitoring Version 5.1 middle layer directory ($DBDIR/dmml). 򐂰 trace.filename File name to which the trace messages from the adapter will be written. The default is dm.trc. 򐂰 adapter.trace.enable Set this to true if you want to store all trace messages regarding the operations of the adapter. The messages are stored in the file identified in trace.filename. The default is false. 򐂰 transport.trace.enable Set this to true if you want to store all messages regarding the transport of adapter-acquired data to the CommonListener. The messages are stored in the file identified in trace.filename. The default is false.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

231

򐂰 adapter.trace.level If you have enabled adapter trace messages, set this to low, medium, or high, according to the level of detail you require. The default is low. 򐂰 transport.trace.level If you have enabled adapter trace messages, set this to low, medium, or high, according to the level of detail you require. The default is low. 򐂰 transport.mqe.usefiller Set this to true if the ManagedNode/gateway on which the adapter is installed is running Windows NT, 4.0, or Service Pack 5; otherwise leave as the default value of false. 򐂰 dmml.trace_size Specifies the size of components (heartbeat, task engine, request-manager, endpoint-upcall) trace in bytes; default is 500000 bytes. 򐂰 dmml.trace_level Specifies the level of the components (heartbeat, task engine, request-manager, endpoint-upcall) trace from 0 (minimal) to 4 (verbose); the default is 1. 򐂰 heartbeat.send_events_to_tbsm Set this to true if you want to send heartbeat events to the Tivoli Business Systems Manager; otherwise leave as the default value of false. 򐂰 heartbeat.send_events_to_tec Set this to true if you want to send heartbeat events to the Tivoli Enterprise Console server; otherwise leave as the default value of false. 򐂰 heartbeat.send_events_to_notice Set this to true if you want to send heartbeat events to the IBM Tivoli Monitoring Version 5.1 notice group; otherwise leave as the default value of false. 򐂰 heartbeat.reboot_engine_if_down Set this to true if you want heartbeat to restart the engine that was stopped abnormally. A heartbeat event is sent if you configured sending events.

232

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 heartbeat.tec_server If you have set heartbeat.send_events_to_tec to true, enter here the name of the Tivoli Enterprise Console server. Specifies the number of threads that the Request Manager uses to handle internal requests. It is approximately the number of endpoints that can be managed simultaneously. The value can be tuned depending on the workload of the ManagedNode on which the Request Manager runs. The default value is 10. 򐂰 request-manager.automatic_cancel_frequency Specifies the time interval (in seconds) after which the Request Manager checks if the applications are using the requests that they have submitted. When an application is not using the requests, the Request Manager cancels the requests submitted by the application. The default value is 600 seconds. 򐂰 request-manager.request_expiration_period Specifies the number (....x....) of periods allowed for an application to retrieve data. If an application does not receive data during.....x....periods, then the Request Manager cancels the request for that application. The default value is 3 periods. This means that, for an application that submitted a request with a refresh time of 10 minutes, if the application is not getting any data for 10*3=30 minutes, then the request gets canceled. 򐂰 tbsma.jre_root This parameter is set during the installation of the Tivoli Business Systems Manager Adapter (see 12.3, “Install/configure IBM Tivoli Monitoring Version 5.1 TBSM Adapter” on page 409) and you do not normally need to change it manually. However, if, for example, you want to install the adapter on a group of gateways using one instance of the install action/command, you will then need to change this parameter on any gateways in the group that have JRE installed at a location different than that supplied in the Install Options dialog box. Set this parameter to the complete path of the root directory of Java Runtime Environment 1.3.0 (excluding the directory/bin). Note: The actual wdmconfig command does not have a syntax check, so be careful when setting the parameters. In Example 9-4 on page 234, using the wdmconfig command, we have configured the trace of components size to 800000 bytes and the trace details to verbose. Then we checked those values in file $DBDIR/dmml/.config.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

233

Example 9-4 Using wdmconfig tividc11:/>wdmconfig -m all -D dmml.trace_size=8000000 -D dmml.trace_level=4 tividc11:/tivoli/tividc11.db/dmml>pg .config | grep dmml.trace dmml.trace_level = 4 dmml.trace_size = 8000000

wdmmn This stops or starts selected IBM Tivoli Monitoring Version 5.1 processes on one or all ManagedNodes/gateways, as shown in Example 9-5. The processes that can be started and stopped are: 򐂰 򐂰 򐂰 򐂰

Task engine (tmnt_task_eng), flag -t Tivoli Business Systems Manager Adapter, flag -b Heartbeat engine (tmnt_gtw), flag -h Resource model engine (tmnt_rm_eng), flag -r

Example 9-5 Stopping and starting process tividc11:/>wdmmn -stop -m all -b -h -t -r Stopping task_engines on managed nodes..... Stopped task_engine on Managed Node:1931340152.2.7#TMF_ManagedNode::Managed_Node# Stopped task_engine on Managed Node:1931340152.1.348#TMF_ManagedNode::Managed_Node# Stopped task_engine on Managed Node:1931340152.20.7#TMF_ManagedNode::Managed_Node# Stopping tbsm-engine on managed node 1931340152.20.7#TMF_ManagedNode::Managed_Node#... Stopping tbsm-engine on managed node 1931340152.1.348#TMF_ManagedNode::Managed_Node#... Stopping tbsm-engine on managed node 1931340152.2.7#TMF_ManagedNode::Managed_Node#... Stopping hb-engine on managed node 1931340152.20.7#TMF_ManagedNode::Managed_Node#... hb-engine stopped Stopping hb-engine on managed node 1931340152.1.348#TMF_ManagedNode::Managed_Node#... hb-engine stopped Stopping hb-engine on managed node 1931340152.2.7#TMF_ManagedNode::Managed_Node#... hb-engine stopped Stopping rm-engine on managed node 1931340152.20.7#TMF_ManagedNode::Managed_Node#... rm-engine stopped Stopping rm-engine on managed node 1931340152.1.348#TMF_ManagedNode::Managed_Node#... rm-engine stopped

234

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Stopping rm-engine on managed node 1931340152.2.7#TMF_ManagedNode::Managed_Node#... rm-engine stopped tividc11:/>wdmmn -start -m all -b -h -t -r Starting task_engines on managed nodes..... Started task_engine on Managed Node:1931340152.2.7#TMF_ManagedNode::Managed_Node# Started task_engine on Managed Node:1931340152.1.348#TMF_ManagedNode::Managed_Node# Started task_engine on Managed Node:1931340152.20.7#TMF_ManagedNode::Managed_Node# Starting rm-engine on managed node 1931340152.20.7#TMF_ManagedNode::Managed_Node#... rm-engine started Starting rm-engine on managed node 1931340152.1.348#TMF_ManagedNode::Managed_Node#... rm-engine started Starting rm-engine on managed node 1931340152.2.7#TMF_ManagedNode::Managed_Node#... rm-engine started

wdmcmd Stops or restarts IBM Tivoli Monitoring Version 5.1 on one or more endpoints from a managed node/gateway or server. You can specify endpoints or profile manager. If you specify a profile manager you should specifies the profile manager whose subscribers are the target of the command, as shown in Example 9-6. Example 9-6 Stopping the engine of profile manager subscribers tividc11:/>wgetsub @ProfileManager:endpoints endpoints_linux endpoints_windows tividc11:/>wgetsub @ProfileManager:endpoints_linux vmlinux5 rhlinux71 tividc11:/>wgetsub @ProfileManager:endpoints_windows itso26 itso27 tividc11:/>wdmcmd -stop -p endpoints#tividc11-region Stopping engine on endpoint 1931340152.16.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1931340152.16.517+#TMF_Endpoint::Endpoint# Stopping engine on endpoint 1931340152.26.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1931340152.26.517+#TMF_Endpoint::Endpoint# Stopping engine on endpoint 1931340152.25.517+#TMF_Endpoint::Endpoint#

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

235

Stopped engine on endpoint 1931340152.25.517+#TMF_Endpoint::Endpoint# Stopping engine on endpoint 1931340152.24.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 1931340152.24.517+#TMF_Endpoint::Endpoint#

wdmeng This stops or starts profiles or resource models at endpoints; also deletes profiles at endpoints, as shown in Example 9-7. Example 9-7 Stopping a specific resource model # Viewing active resource models tividc11:/>wdmlseng -e vmlinux5 -verbose Forwarding the request to the engine... The following profiles are running: linux#tividc11-region DMXFileSystem: Running LowPercInodesAvail 100 % FragmentedFileSystem 100 % LowKAvail 100 % pf.unix.itm#tividc11-region DMXMemory: Running LowStorage 100 % Thrashing 100 % LowSwap 100 % # Stopping the DMXFileSystem resource model tividc11:/>wdmeng -e vmlinux5 -p linux#tividc11-region DMXFileSystem -stop Forwarding the request to the engine... The operation has been successfully completed. # Querying the engine, note that DMXFileSystem resource model is stopped tividc11:/>wdmlseng -e vmlinux5 -verbose Forwarding the request to the engine... The following profiles are running: linux#tividc11-region DMXFileSystem: Stopped LowPercInodesAvail 100 %

236

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

FragmentedFileSystem 100 % LowKAvail 100 % pf.unix.itm#tividc11-region DMXMemory: Running LowStorage 100 % Thrashing 100 % LowSwap 100 %

wdmrm This adds, lists, or removes a specified default resource model at the Tivoli management region server or ManagedNode/gateway from where it is issued, as shown in Example 9-8. It also adds the NLS catalog to an already installed default resource model. Example 9-8 Viewing installed resource models tividc11:/>wdmrm -list Tivoli Distributed Monitoring - Listing resource models Tivoli Distributed Monitoring - Resource model utility Resource -> DMXCpu NLS name product_id major_version minor_version platform message catalog zip file

: : : : : : :

CPU none 1 1 aix4-r1\hpux10\linux-ix86\linux-s390\solaris2 DMXCpu DMXCpu.zip

9.2.2 Heartbeat maintenance If you are using the heartbeat functionality, be aware that when you delete an endpoint from the Tivoli environment, and this endpoint is being monitored by the heartbeat, it will not be removed automatically from the heartbeat database. To remove an endpoint from the heartbeat database, run the command: wdmmngcache -m gateway -d endpoint_label

For further information about heartbeat, refer to Chapter 7, “Heartbeat” on page 167.

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

237

9.2.3 Directories In this section we explain the directory structure created on ManagedNode after the installation of IBM Tivoli Monitoring Version 5.1 and the directory structure created on a endpoint after a successful profile distribution.

TMR Server/ManagedNode The directory structure is: 򐂰 $DBDIR/AMW IBM Tivoli Monitoring Version 5.1 logs directory 򐂰 $DBDIR/dmml Directory containing.config file, epcache.db directory, which is used by the heartbeat, and rqcache.db, which is used by the request-manager 򐂰 $BINDIR/TME/Tmw2k IBM Tivoli Monitoring Version 5.1 binaries 򐂰 $BINDIR/TMNT_TEC IBM Tivoli Monitoring Version 5.1 BAROC files and rule base 򐂰 $BINDIR/../lcf_bundle/Tmw2k IBM Tivoli Monitoring Version 5.1 binaries for endpoints 򐂰 $BINDIR/../lcf_bundle.40/Tmw2k IBM Tivoli Monitoring Version 5.1 binaries for endpoints 򐂰 /Tivoli/msg_cat/C/tmw2k* IBM Tivoli Monitoring Version 5.1 message catalogs

UNIX/Linux endpoint The directory structure is: 򐂰 $LCF_TOOLSDIR/../JRE/DMAE Directory containing the link for the JRE binary or the JRE installation 򐂰 $LCF_DATDIR/LCFNEW 򐂰 $LCF_DATDIR/LCFNEW/AMW 򐂰 $LCF_DATDIR/LCFNEW/AMW/logs Message and trace logs 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Rm Resource models

238

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix, 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Classes Untars the ILT; Needs to be in CLASSPATH 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Config Saves the Conf files 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Dec Saves the JavaScripts 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Mof Moves the Mof files; These are deleted as soon as they are parsed 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/bin Contains the dm_engine binaries file (JAR files) and shell scripts to set up the environment and start the dm_engine 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/customscripts 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/data Working directory CIM Classes, Profiles, and logged data persist here; Other files are: engine.pid, engsync, lock files, log_level, and log_size 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/lib Where the native libraries of RM are saved; This directory needs to be in LIBPATH 򐂰 $LCF_DATDIR/LCFNEW/Tmw2k/Unix/probes 򐂰 $LCF_DATDIR/cache/bin/interp/TME/Tmw2k Binary tmw2k_ep

Windows endpoint The structure is: 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\Config Saves the Conf files 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\Dec Saves the Visual Basic code 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\Mof Mof files are moved here; They are deleted as soon as they are parsed

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

239

򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k/bin Binaries for the engine, logger, analyser, actionmanager, event aggregator, and event correlator 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\bin\tm2kconf Tmw2kProfiles distributed 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\db Contains the data (mdb format) recorded by the logger 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\Rm Resource models 򐂰 %LCF_DATDIR%\LCFNEW\Tmw2k\Unix Profile cores 򐂰 %LCF_DATDIR%\cache\bin\w32-ix86\TME\Tmw2k Binary tmw2k_ep.exe

9.2.4 Logs and files In this section we describe some files used by IBM Tivoli Monitoring Version 5.1 that should be periodically monitored, otherwise their size could use considerable disk space.

TMR Server The files to be used on the TMR Server are: 򐂰 $DBDIR/distmgr.log, used by MDist2 򐂰 $DBDIR/rpt2log, used by MDist2 򐂰 /tmp/states, used by MDist2 in UNIX TMR 򐂰 %DBDIR%/tmp/states, used by MDist2 in Windows TMR 򐂰 Messages and traces under $DBDIR/AMW/logs, since each Tmw2kProfile distribution process creates a log message under this directory 򐂰 /tmp/AMW

240

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Gateway/ManagedNode The files to be used on the gateway/ManagedNode are: 򐂰 /tmp/states, used by MDist2 in UNIX TMR 򐂰 %DBDIR%/tmp/states, used by MDist2 in Windows TMR 򐂰 Messages and traces under $DBDIR/AMW/logs 򐂰 $DBDIR/tmp/AMW

Endpoint If you set the trace_level on the endpoint to verbose, be sure to set the parameter trace_size to a reasonable value.

9.3 New classes in the Tivoli database This section explores the new classes and objects created in the Tivoli database. To better understand this section you must have a good knowledge of the Tivoli database design and commands to extract data from the database. For more information about the Tivoli database refer to Section 3.1, “Tivoli object database concepts and tools” in Maintaining Your Tivoli Environment, SG24-5013. After the IBM Tivoli Monitoring Version 5.1 installation, the classes and objects discussed in the following sections are created in the Tivoli database.

9.3.1 Classes The following are the classes: 򐂰 Tmw2kProfile – The new profile for the product. – The executable for this is $BINDIR/TME/Tmw2k/tmw2k_profile_core. – Program runs as deamon with timeout 1 sec. 򐂰 TMWin2kEndPoint – Endpoint class containing the downcall methods. – The ep method executable is /TME/Tmw2k/tmw2k_ep. – The ep method is executed “per method.”

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

241

򐂰 DMMiddleLayerProcessor ManagedNode class containing the methods of task engine, TBSM adapter, and heartbeat engine. Implemented in different executables: – $BINDIR/TME/Tmw2k/tmnt_task_eng – $BINDIR/TME/Tmw2k/tmnt_tbsm_eng – $BINDIR/TME/Tmw2k/tmnt_hb_eng All programs are executed as threaded daemons. 򐂰 DMMiddleLayerCollector Gateway class containing the upcall methods for task engine, TBSM adapter, and heartbeat engine (whose behavior is inherited by gateway). Implemented in $BINDIR/TME/Tmw2k/tmnt_gtw. The methods are all executed “per method.”

9.3.2 Distinguished tmw2kController 򐂰 Object that uses Mdist to distribute the RM zipped files. The related binary is $BINDIR/TME/Tmw2k/tmw2k_send. 򐂰 It is a shared daemon with time out to 60 secs. 򐂰 Only runs on the server.

9.3.3 Name registry resource types ResModelInfo This resource type is added at installation time. For each RM a ResModelInfo resource is created: 򐂰 Contains the default values of each RM. 򐂰 Contains the name of the gzipped files. To view the classes distinguished and name registry, you can use the following commands: wlookup -ar Classes wlookup -ar distinguished

To see all resource models installed, use: wdmrm -list

242

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

To see all ResModelInfo entries: wlookup –ar ResModelInfo

To look at the content of each ResModelInfo: TNR=`wlookup NameRegistry` RESINFO=`idlcall $TNR lookup '"ResModelInfo" "TMW_MemoryModel"'`

Chapter 9. Configure and maintain IBM Tivoli Monitoring Version 5.1

243

244

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

10

Chapter 10.

Workbench This chapter gives an overview of the IBM Tivoli Monitoring product, its underlying architecture, and its components. A main component of this new architecture is the resource model, which defines what to monitor, what thresholds to react on, and how to react. We will go into particular detail regarding the Workbench, which is used to build and maintain resource models. The Workbench is an Integrated Development Environment (IDE) for developing and modifying resource models for Tivoli Monitoring. The Workbench produces resource models in the output format of a tar file. The tar file is copied over to the TMR Server where IBM Tivoli Monitoring is installed, and can then be added to the environment by using the wrmadd command. The following sections of this chapter will detail the steps, using real-world examples, used to develop and modify resource modes. A thorough understanding of the IBM Tivoli Monitoring architecture is recommended before reading the following sections. The IBM Tivoli Monitoring Workbench User’s Guide Version 5.1.1, SH19-4571, provides a complete overview of the IBM Tivoli Monitoring architecture, as well as a detailed overview of the mechanics of the tool. Note: The code samples developed in this chapter were provided as tools to learn how to use the IBM Tivoli Monitoring 5.1 Workbench and how to develop custom resource models. Although all of the custom resource models developed in this chapter were tested, they were not designed nor developed for production use.

© Copyright IBM Corp. 2002

245

This chapter will outline four examples used to illustrate the use of the Workbench and how to develop and modify resource models. The following examples will be used in this chapter: 1. In the first example we will make modifications to TMW_PhysicalDiskModel. In this example we will demonstrate how to add additional WMI/CIM attributes to an existing resource model and change some of the Visual Basic script code logic. Along with modifying TMW_PhysicalDiskModel, we will demonstrate how to use the Visual Basic script debugger, and the Sax Basic debugger by Saxsoft, which comes with the Workbench. 2. In the second example we will demonstrate how to create a new resource model. In this example we will create a a new resource class using CIM/WMI properties not supplied in any of the existing IBM Tivoli Monitoring resource model classes. In this example we will use TMW_LogicalDiskModel as a template for our new resource model. This example will also highlight the uses of WQL filtering and the use of the MOF compiler. 3. The third example will illustrate how to create a new UNIX-based JavaScript resource model. In this example we will use the existing DMX_Cpu resource class and collect additional properties not currently used by the DMX_Cpu model. We will define new threshold elements and implement our own decision tree code. In the example we will illustrate debugging techniques used in the Windows environment, as well as trace and log on the UNIX platforms. 4. The last example in this chapter will demonstrate the use of parameters. We will take the existing DMX_FileSystem resource model and turn it into a parametric-based model (that is, a model where a user can enter parameters in the Profile to control the behavior of the model). In this example we will create a resource model that allows the user to enter the file system and threshold to be monitored. In addition, this chapter gives a brief outline of new features of IBM Tivoli Monitoring 5.1 to provide the background knowledge required to get the most out of this book. For more information about the new functions of IBM Tivoli Monitoring 5.1, please refer to IBM Tivoli Monitoring Workbench User’s Guide Version 5.1.1, SH19-4571. The following topics are discussed in this chapter: 򐂰 򐂰 򐂰 򐂰

246

10.1, “New features” on page 247 10.2, “Using the Workbench” on page 248 10.3, “Resource model examples” on page 281 10.4, “Tools and extra information” on page 376

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Note: Information on how to retrieve the source for all of the examples used in this chapter can be found in Appendix A, “Additional material” on page 529.

10.1 New features This section describes the new features added to the IBM Tivoli Monitoring product.

10.1.1 Compatibility mode The new Workbench provides an easy way to import from Tivoli Distributed Monitoring (Classic Edition). A new migration helper script has been added to analyze and migrate Tivoli Distributed Monitoring (Classic Edition) monitors to IBM Tivoli Monitoring resource models. A new wizard is available in the Workbench to import Tivoli Distributed Monitoring (Classic Edition) 3.7 monitor collections and custom scripts into resource models. See Chapter 5, “Migrating Classic Edition monitors” on page 111, for more details.

10.1.2 Custom scripting support The new Workbench now allows the use of shell commands and custom scripts (bash, Perl) inside resource model as additional data sources. A new method, shell, has been added so that scripts can be invoked in the resource models’ (Java and Visual Basic) script decision tree code. See 10.3.4, “ITSO_FileSystem Example” on page 353, for an example.

10.1.3 New response actions The Tivoli Distributed Monitoring Workbench Version 4.1 engine allowed only CIM method invocation as response actions for critical events. Now with IBM Tivoli Monitoring it is possible to run shell programs or custom scripts along with CIM methods as responses and recover from abnormal conditions reported by resource models. See 10.3.4, “ITSO_FileSystem Example” on page 353, for an example.

10.1.4 Customizable messages for TEC and TBSM events It is possible now to send events with messages that use event variables from the Java and Visual Basic script code. If we are sending an event saying that a service is stopped, before this enhancement the message was A key service is stopped. Now we can have a parametrized message like The service

Chapter 10. Workbench

247

@ServiceName@ is stopped. Where the token @ServiceName@ is replaced at runtime by the value of the event attribute ServiceName. We can also use more than one event attribute in the message. See 10.3.4, “ITSO_FileSystem Example” on page 353, for an example.

10.1.5 Wizard for UNIX resource models Now, in IBM Tivoli Monitoring, the wizards have been improved to generate code in Java and Visual Basic Script code that meets the standards used by both JavaScripting engines: Microsoft Script Host (on Windows) and Rhino (on UNIX). The IBM Tivoli Monitoring 5.1 Workbench also provides a wizard to generate JavaScript resource models that run on Windows. See 10.3.4, “ITSO_FileSystem Example” on page 353 for an example.

10.1.6 New service object APIs Several new methods have been added in IBM Tivoli Monitoring. Here is a short summary of the new methods. Please refer to the IBM Tivoli Monitoring Workbench User’s Guide Version 5.1.1, SH19-4571, for more details. 򐂰 Mapping table APIs (CreateMap, SetMapNumElement, SetMapStrElement, etc.) 򐂰 SendEventEx and LogInstEx 򐂰 CollectClassData Probe Triggering (DMCallNumProbe, DMCallStrProbe) 򐂰 Custom Program triggering (Shell)

10.1.7 New CLI support The new Workbench provides a simple CLI to use some of its functionality from command prompt. This basically helps in automating resource model package builds from the source workspaces, and modifying some of its components without running the GUI-based tool. See 10.4.1, “WorkBench Command Line Interface” on page 376, for more details.

10.2 Using the Workbench Before we begin creating the examples, a quick review of the basics of the Workbench might be helpful. We will begin by exploring one of the simpler resource models shipped with IBM Tivoli Monitoring, the PhysicalDiskModel. We will use this model later in our examples to illustrate some of the advanced features of the Workbench.

248

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Let us begin by opening the IBM Tivoli Monitoring icon, as depicted in Figure 10-1.

Figure 10-1 Opening the IBM Tivoli Monitoring Workbench

Once we have opened IBM Tivoli Monitoring, we will need to open the sample project file for the PhysicalDiskModel (shipped with the product). We can select File -> Open, as depicted in Figure 10-2.

Figure 10-2 Opening the browser in IBM Tivoli Monitoring

In the browser we can navigate to the samples directory in the install location of IBM Tivoli Monitoring, as depicted in Figure 10-3 on page 250.

Chapter 10. Workbench

249

Figure 10-3 Navigating to the samples directory

In the samples directory there are four subdirectories. Each of these subdirectories contains a sample source for resource models. 򐂰 Exchange This directory contains four sample resource models for Microsoft Exchange. See the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569, for a discussion on these models. 򐂰 TWCM This directory contains six sample resource models (three for Windows and three for UNIX) for assisting in the Tivoli Web Component Manager migration. For more details on this subject see Chapter 6, “Migrating from Tivoli Web Component Manager” on page 161. 򐂰 UNIX This directory contains the sample source for the eight resource models shipped with IBM Tivoli Monitoring for the UNIX/Linux platforms. We will be using a couple of these files in our examples later on.

250

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Windows This directory contains the source for the fourteen resource models shipped with IBM Tivoli Monitoring. Note: When opening the browser in IBM Tivoli Monitoring, the default file type is *.dmws. The dmws files are the resource model samples developed in Visual Basic Script. The resource models developed in JavaScript have the file extension of dmjsws. If we do not change the default file type we might not see all the files in the current directory. For example, in the TWCM directory there are six files (three dmws and three dmjsws). In order to see all files in all directories from the browser, it is best to change the file type to All Files (*.*). As we stated earlier we are going to open up the PhsicalDiskModel to demonstrate the basics of using the Workbench. We will need to select the sample source provided with IBM Tivoli Monitoring for this model. The source for this model can be found in the TMW_PhysicalDiskModel.dmws project file, as shown in the browser list in Figure 10-4.

Figure 10-4 Browser listing of the Windows sample (dmws) files

Chapter 10. Workbench

251

TMW_PhysicalDiskModel.dmws is a non-editable file that contains all of the element definitions for the PhysicalDisk resource model. This is sometimes referred to as a project file. We can open this file and explore each of its elements. If we double-click the left mouse button we will see a full display of the TMW_PhysicalDiskModel resource model in the Workbench, as depicted in Figure 10-5.

10.2.1 The Workbench panes The Workbench is made of three panes, as shown in Figure 10-5.

Figure 10-5 PhysicalDiskModel opened in Workbench

252

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The panes are: 򐂰 Pane 1 The first pane contains most of the resource model elements that make up a particular model. These elements are ordered in a tree-like hierarchy. The root node of the tree is the name of the resource model and describes the model’s general settings. In this case, it is the TMW_PhysicalDiskModel. In most cases any element in the tree can be opened by a double left-click the mouse button. Some elements in the tree have multiple options associated with it. A right-click the mouse button on any element in the tree will list the associated options for that element. For example, a right-click the mouse button on the TMW_PhysicalDisk element under Dynamic Model/CIM Classes will display three options (Modify, Remove, and Collection Test). We will be illustrating the use of many of these options later in the examples. 򐂰 Pane 2 This pane contains the Java or Visual Basic script code used in the resource model. The functions in this code are used to initialize, set configuration options, and run the decision tree logic. The code in this pane will be discussed in detail in all of the examples, as well as in a further discussion in 10.2.3, “Looking at the PhysicalDiskModel” on page 257. 򐂰 Pane 3 This pane is only used for models that interact with the Windows platforms (that is, CIM/WMI-based). This pane is used when debugging a resource model and can be used to display the aggregate status of event indications. An example of this is illustrated in 10.3.1, “ITSO_PhysicalDiskModel example” on page 282.

10.2.2 Elements of a resource model All resource models are made up of seven basic elements, as described below: 򐂰 The dynamic model The dynamic model element is where all of the CIM classes and properties are defined for the resource model. Under the dynamic model element there are two sub-elements: – CIM classes: This is where all of the resource model class definitions are defined. CIM classes are defined and modified by invoking the Workbench CIM browser. An example of this is detailed in Figure 10-11 on page 261.

Chapter 10. Workbench

253

– DM classic probes: This is where Tivoli Distributed Monitoring (Classic Edition) 3.7 compatibility mode monitors are defined. These monitor definitions are imported from dumped MCSL files or from source CSL files. This topic will be discussed in detail in Chapter 5, “Migrating Classic Edition monitors” on page 111. 򐂰 Events The event element is where all of the resource model indications are defined. This is where the default attributes that make up an event along with the aggregation defaults (holes and occurrences) are defined. This topic is discussed in “The event element of the PhysicalDiskModel” on page 265. Note: An event element in a resource model is really just an indication definition (as seen in an IBM Tivoli Monitoring 5.1 Profile). The event element is not necessarily the same thing as a TEC or TBSM event. The IBM Tivoli Monitoring event aggregator processes all indications using event element (holes and occurrences) definitions. Once it passes through the event aggregation, based on the number of holes and occurrences and the profile definitions, it can then become a TEC or TBSM event. 򐂰 Thresholds The threshold element is where the different resource model class property thresholds are defined. For example, in TMW_PhysicalDiskModel there are three threshold properties (HighBytesSec,HighQLength, and HighPercentUsage). These thresholds are set when the resource model is initialized on the endpoint and retrieved on each cycle time by the decision tree logic. The decision tree logic uses the threshold elements to determine if an indication should be sent. 򐂰 Parameters The parameters element is used for parametric resource models. Parametric resource models are designed to allow the user to customize the resource model definitions from the IBM Tivoli Monitoring 5.1 profile. Some examples of parametric-based resource models are TMW_ParamPort, TMW_ParamEventLog, and the TMW_ParamServices resource models. For example, the TMW_ParamServices resource model allows a user to customize the NT or W2K monitored services defined in the profile from the Tivoli Desktop. We will be creating a new resource model later in this chapter that will illustrate an example of using the parameter element. See 10.3.4, “ITSO_FileSystem Example” on page 353.

254

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Logging The logging element is used to define the properties of the resource model that are to be logged. All of the examples discussed in this chapter will include logging elements. 򐂰 Dependencies The Dependencies element is used to define file dependencies that are to be pushed with the resource model when the profile is distributed to the endpoint. Examples of dependency files are DLLs, MOF files, and executable script files. We will also be demonstrating the use of dependencies in some of the examples later on. 򐂰 Decision tree script The decision tree script is made up of a set of function routines that provide the heart of the logic of the resource model. These functions set default configuration definitions and thresholds, and process them by generating indications and logging events. There are three primary functions used by the IBM Tivoli Monitoring 5.1 engine at runtime and one for debugging. – SetDefaultConfiguration: This function sets runtime definitions based on resource model element settings. This routine is run once when the IBM Tivoli Monitoring 5.1 profile is pushed to the endpoint. The code in the function should not be modified because it is automatically generated by the Workbench. For example, if we use the Workbench to change the event element definitions, it will automatically change the corresponding definition in this function. – Init: This function is called after the SetDefaultConfiguration function and can be used to create user settings or change default settings. Remember that some of the values defined in the resource model elements can be overridden in the IBM Tivoli Monitoring profile. This routine runs after the profile settings have been initialized. – VisitTree: This routine runs every cycle time after the IBM Tivoli Monitoring profile has been pushed. This function contains all of the decision tree logic. The VisitTree function retrieves threshold values defined in the SetDefaultConfiguration function and processes the decision tree logic through a series of “if” statements. When a threshold exceeded is detected, the VisitTree generates indications and/or logging events. All of our examples will demonstrate customizing VisitTree code.

Chapter 10. Workbench

255

– Main: This routine is only used for debugging. The Windows only based (that is, Visual Basic script) models will display this routine in the Workbench as the first function. The JavaScript-based resource models will not display this function in the Workbench. The Workbench will generate an output file that contains all four routines, including Main, when debugging JavaScript-based resource models. In this case the output JavaScript source file will be created in the IBM Tivoli Monitoring install directory under the subdirectory named scripts. This file is passed to the MicroSoft Script Debugger. See notes later for more information on using the Microsoft Script Debugger at the current release. Figure 10-6 shows the flow between the IBM Tivoli Monitoring 5.1 engine and the decision tree functions.

Figure 10-6 Decision tree script logic

256

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

10.2.3 Looking at the PhysicalDiskModel The following sections discuss the PhysicalDiskModel.

Displaying the properties of the PhysicalDiskModel To open up the properties of a resource model, we can double left-click the mouse button or right-click the mouse button on the root node object. In this case it is the TMW_PhysicalDisk object. We see the display depicted in Figure 10-7.

Figure 10-7 TMW_PhsyicalDiskModel general settings

The following are descriptions of the properties: Internal name

This is the internal name of the resource model. Changes to this setting will automatically update the SetDefaultConfiguration function Svc.SetModelName method call. See Figure 10-10 on page 260 for an example of these settings in the SetDefaultConfiguration function. The internal name can be retrieved in the VisitTree function by calling the Svc.GetModelName method. This name must be alphanumeric, starting with an alphabetic letter, and containing no blanks.

Descriptive name

This is the name that appears in the IBM Tivoli Monitoring 5.1 profile list of resource models. See Figure 10-8 on page 258 for an example of the Descriptive Name in the profile.

Chapter 10. Workbench

257

Figure 10-8 Example of descriptive name in a profile

258

Description

This is the text that describes the resource model. This is used for documenting the resource model. These settings can be used to document the decision tree logic, as well as explain the threshold settings.

Category internal name

This name is the internal name for grouping resource models on the profile category display. All resource models that have the same category internal name will be displayed under the same category option, as seen in Figure 10-9 on page 259.

Category descriptive name

This is the name that is displayed in the profile category display. An example of this can be seen in the Category pull-down options in Figure 10-9 on page 259.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-9 Category profile pull-down option

Cycle time

This is the default cycle time for the resource model. This value can be changed in the profile before profile distribution. This setting will change the Svc.SetCycleTime in the SetDefaultConfiguration function. See Figure 10-10 on page 260 for the SetDefaultConfiguration function for the PhysicalDiskModel. This setting can also be retrieved in the VisitTree function using the Svc.GetCycleTime method.

Major and minor version

This is the internal version of the resource model.

Supported platforms

This is a list of the platforms that the resource model supports. The supported platforms are defined when the resource model is created.

The properties created or modified from the general settings dialog will automatically update the code in the SetDefaultConfiguration function, as seen in Figure 10-10 on page 260.

Chapter 10. Workbench

259

' General info section ' Svc.SetModelName “TMW_PhysicalDiskModel” Svc.SetProfileName “157999030” Svc.SetCycleTime 120 '

Figure 10-10 PhysicalDiskModel General Settings

The dynamic model element of the PhysicalDiskModel The PhysicalDiskModel resource model uses only one resource class, the TMW_PhysicalDisk class. We can look further at this class by opening the CIM browser supplied with the Workbench. See Figure 10-11 on page 261 for an example of opening the CIM browser. To open up the CIM browser we will select the TMW_PhysicalDisk resource class under the dynamic model/CIM classes hierarchy. When opening up the CIM browser from the Workbench, the first pop-up window is used to designate the CIM name space we want to open. For Windows CIM/WMI-based models always use root/CIMV2. However, later when we start working with the UNIX/Linux-based models we will have to open the root/default name space. The next window displayed is a login window. This window always defaults to the signed-on user (just click OK).

260

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-11 Opening the CIM browser

The CIM browser now displays all of the properties of the PhysicalDisk resource class defined in the PhysicalDiskModel. See Figure 10-12 on page 262. Note: The Workbench CIM browser does not automatically position the browser to the correct CIM class in the Classes window (the left pane), even though it displays all of the correct properties for the selected class. If another class is selected, all of the properties are erased from the selected properties window. It is important to cancel out of the browser in this scenario. However, if we are creating a new resource class, as will be demonstrated later, a new class will have to be selected.

Chapter 10. Workbench

261

Figure 10-12 PhysicalDisk resource class definitions

262

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The following is an overview of property fields that make up the PhysicalDisk resource class settings. Class properties

These are the attribute values (in CIM terminology they are called properties) that are selected for processing. The SetDefaultConfiguration will have a Svc.DefineClass method call with all of the properties selected from this class. See Figure 10-1 on page 264 for the SetDefaultConfiguration function for the PhysicalDiskModel. The IBM Tivoli Monitoring analyzer will collect all of the properties that are in the selected list each cycle time.

Collection info

This is used to optionally set sort options for data that may need to be in sorted order.

Filtering

This is where optional filters can be placed on the collection process before the VisitTree function is called. For example, if we did not want to process any physical disks, where the PercentDiskTime property was less than 25 percent we could add a WQL filter, as seen in Figure 10-13 on page 264. The WHERE Clause field is passed to WMI as WQL code by the engine to filter data that is to be returned. For more information on WQL see 10.4.3, “Microsoft tools” on page 379.

Chapter 10. Workbench

263

Figure 10-13 Using WMI WQL

The properties created or modified from the dynamic model element will automatically update the code in the SetDefaultConfiguration function, as seen in Example 10-1. Example 10-1 PhysicalDiskModel dynamic model properties ' Dynamic model section ' Svc.DefineClass "CIM", "TMW_PhysicalDisk", "ROOT\CIMV2:TMW_PhysicalDisk", "","AvgDiskSecXfer,PercentDiskWriteTime,AvgDiskBytesXfer,DiskXfersSec,PercentDi skReadTime,AvgDiskReadQLength,DiskReadBytesSec,DiskReadsSec,DiskBytesSec,Curren tDiskQLength,AvgDiskQLength,PercentDiskTime,AvgDiskWriteQLength,DiskWritesSec,D iskWriteBytesSec", "PhysicalDisk", "None", "", 0, 1 '

264

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The event element of the PhysicalDiskModel The PhysicalDiskModel has six event elements that are used by the decision tree, VisitTree, to send indications. These elements are defined in the resource model and define the event name along with all of the attributes of the event and the aggregation settings. Once the resource model is added to the Tivoli environment, the aggregation settings can be updated from a IBM Tivoli Monitoring profile, as seen in Figure 10-14.

Figure 10-14 Profile indications

We will take a look at one of the event elements in this example. In the PhysicalDiskModel, the TMW_SlowPhysicalDrive indication is generated by the VisitTree function when the following property conditions are true: 򐂰 The CurrentDiskQLength property is greater than the threshold setting. 򐂰 The DiskBytesSec property is greater than the threshold setting.

Chapter 10. Workbench

265

None of this logic can be found in the event element settings. This can only be derived from analyzing the VisitTree code, which we will be doing later in this chapter in Example 10-5 on page 278. The threshold values can be seen in the threshold elements discussed in the next section. The event element defines the properties that make up the event as well as the aggregation settings (holes and occurrences) that are used by the event aggregator. Figure 10-15 on page 267 is an example of the TMW_SlowPhysicalDrive event element in the Workbench.

266

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-15 TMW_SlowPhysicalDrive event element settings

Chapter 10. Workbench

267

The following is an overview of the TMW_SlowPhysicalDrive properties:

268

Internal name

This is the internal name used for this indication. This name will be defined in the SetDefaultConfiguration function with the Svc.DefineEvent method. See Figure 10-2 on page 269 for the SetDefaultConfiguration function for the PhysicalDiskModel. The internal name is referenced in the VisitTree function by the SvcSendEvent and Svc.SendEventEx methods. The SendEvent and SendEventEx methods pass indications to the event aggregator to determine what action to take. The internal name is also used as a prefix to variables passed to shell scripts. For example, if a Perl script is invoked as a response from the TMW_SlowPhysicalDrive, one of the variables that would be set in the Perl script at runtime would be TMW_SlowPhysicalDrive_PercentDiskTime.

Attributes

These are the attributes of the event. If the event aggregator generates a TEC or TBSM event (based on holes and occurrences), these are the properties that will be passed into the slots of the TEC event and attributes of the TBSM event. These attributes are set in the VisitTree code before the SendEvent or SendEventEx methods are invoked.

Aggregation settings

This is where the aggregation keys are defined, along with the number of occurrences and holes for this indication.

Notification

These are the definitions for event notification. If the indication becomes an event based on the output of the event aggregator, these settings will be used to set the severity.

String resources

These settings are used for the attributes of the event to be sent. The descriptive name is the display name. The message is used to set the msg slot for TEC events and attributes for TBSM events. In this example the message text is as follows: The physical drive @PhysicalDisk@ is too slow. @PhysicalDisk@ will be replaced with the PhysicalDisk variable set in the VisitTree routine.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Actions

The Actions button is used to define response actions for the event. CIM Methods and/or shell programs can be defined as response actions to the event element. If a response action is defined, it will run after event aggregation has determined that an event should be generated. For example, if TMW_SlowPhysicalDrive is set to 3 occurrences and 0 holes and has a send_email.pl script defined as an action, then send_email.pl will run locally on the monitored machine after the third consecutive indication has occurred. Unlike compatibility mode scripts and scripts invoked by the Svc.Shell method, response actions do not have a time-out value. We will be using an example of the response action in 10.3.4, “ITSO_FileSystem Example” on page 353.

The properties created or modified from the event element will automatically update the SetDefaultConfiguration function, as seen in Example 10-2. Example 10-2 PhysicalDiskModel event element properties ' Event definition section ' Svc.DefineEvent “TMW_HighPhysicalDiskReadBytesSec”, "DiskReadBytesSec,DiskReadSec,PercentDiskRead", “PhysicalDisk” Svc.DefineEvent “TMW_PhysicalPossibleFrag”, "PercentDiskTime,DiskBytesSec", "PhysicalDisk" Svc.DefineEvent “TMW_HighPhysicalDiskXferRate”, "DiskXfersSec,DiskReadsSec,DiskWritesSec,PercentDiskReadTime,PercentDiskWriteTi me", “PhysicalDisk” Svc.DefineEvent “TMW_HighPhysicalDiskWriteBytesSec”, "DiskWriteBytesSec,DiskWriteSec,PercentDiskWrite", “PhysicalDisk” Svc.DefineEvent “TMW_SlowPhysicalDrive”, "CurrentDiskQLength,PercentDiskTime,AvgQLength,AvgReadQLength,AvgWriteQLength,D iskReadBytesSec,DiskWriteBytesSec", “PhysicalDisk” Svc.DefineEvent “TMW_HighPhysicalPercentDiskTime”, "PercentDiskTime,PercentReadTime,PercentWriteTime", “PhysicalDisk” '

The threshold element of the PhysicalDiskModel The PhysicalDiskModel has three threshold elements that are used by the decision tree logic (that is, VisitTree) to determine if an indication should be sent. These elements are defined in the resource model and become part of the resource model tar file. Once they are added to the Tivoli environment they can be updated from an IBM Tivoli Monitoring profile, as seen in Figure 10-16 on page 270.

Chapter 10. Workbench

269

Figure 10-16 PhysicalDiskModel thresholds in a profile

Figure 10-17 on page 271 is an example of the TMW_HighPercentUsage threshold element in the Workbench.

270

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-17 PhysicalDiskModel HighPercetnUsage threshold profile display

Chapter 10. Workbench

271

The following is an overview of the HighPercentUsage properties: Internal name

This is the internal name that is used by the VisitTree function to determine if a property has exceeded a threshold value. In the VisitTree function the Svc.GetThreshold method is invoked to retrieve and compare this value. The threshold is defined in the SetDefaultConfiguration function by the Svc.DefineThreshold method. Remember the SetDefaultConfiguration definitions are updated automatically by the Workbench when we update the element in the elements pane (that is, pane 1).

Default value

This is the default value associated with the internal name. The HighQLength is 3 by default.

Descriptive name

This is the name that is used for the IBM Tivoli Monitoring profile display, as seen in Figure 10-16 on page 270.

Description

This is used for documentation purposes.

The properties created or modified from the threshold element will automatically update the SetDefaultConfiguration function, as seen in Example 10-3. Example 10-3 PhysicalDiskModel threshold element properties ' Thresholds section ' Svc.DefineThreshold “HighBytesSec”, 1572864 Svc.DefineThreshold “HighQLength”, 3 Svc.DefineThreshold “HighPercentUsage”, 90 '

Note: There are no parameter elements in the PhysicalDiskModel. Parameter elements will be discussed later in example 4 in 10.3.4, “ITSO_FileSystem Example” on page 353.

The logging element of the PhysicalDiskModel The PhysicalDiskModel has three logging elements that are used by the decision tree logic (that is, VisitTree) to determine if an indication should be logged. These elements are defined in the resource model and become part of the resource model tar file. Once they are added to the Tivoli environment they can be updated from an IBM Tivoli Monitoring 5.1 profile, as seen in Figure 10-18 on page 273.

272

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-18 PhysicalDiskModel logging element profile display

Figure 10-19 on page 274 is an example of the percent disk usage logging element in the Workbench.

Chapter 10. Workbench

273

Figure 10-19 PhysicalDiskModel percent disk usage element profile display

The following is an overview of the HighPercentUsage properties:

274

Context

This is the name that is passed as the first parameter of the Svc.LogInst or Svc.LogInstEx methods. This name is also the display name in the IBM Tivoli Monitoring Web Health Console. The Historical Data display can be seen in Figure 10-20 on page 275, as the Contexts pull-down.

Resource

This is the name that is used as the second argument of the Svc.LogInst and Svc.LogInstEx methods. This name is also what is displayed in the IBM Tivoli Monitoring Web Health Console historical data, as seen in Figure 10-20 on page 275, as the Resources pull-down.

Properties

These are the properties that are to be logged under this this context/resource. The PhysicalDiskModel HighPercentUsage logging element has a key of PhysicalDisk. This can bee seen in the graph in Figure 10-20 on page 275.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-20 is a display of the IBM Tivoli Monitoring Web Health Console using the PhysicalDiskModel for historical data (that is, logging).

Figure 10-20 PhysicalDiskModel Web Health Console display

Figure 10-21 on page 276 is a graph of the PhysicalDiskModel historical data using the IBM Tivoli Monitoring Web Health Console. This graph is looking at physical disk1. The PhysicalDisk property is the key for the high percent usage logging element.

Chapter 10. Workbench

275

Note: Notice in Figure 10-21 that the bar chart is maxed out at 100 percent on some of the bars. This is because the PercentDiskTime CIM property generated by PerfMon maxes out at 100 percent. In our first example we will customize the PhysicalDiskModel to address this.

Figure 10-21 PhysicalDiskModel Web Health Console graphic display

The properties created or modified from the logging element will automatically update the SetDefaultConfiguration function, as seen in Example 10-4 on page 277.

276

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Example 10-4 Physicaldiskmodel logging element properties ' Logging definition section ' Svc.DefineLogInst “Bytes Transferred”, “PhysicalDisk”, “PhysicalDisk”, “DiskBytesSec”, “PhysicalDisk” Svc.DefineLogInst “Queue Length”, “PhysicalDisk”, “PhysicalDisk”, “AvgQLength”, “PhysicalDisk” Svc.DefineLogInst “Percent Disk Usage”, “PhysicalDisk”, “PhysicalDisk”, “PercentDiskTime”, “PhysicalDisk” '

Note: There are no dependencies elements in the PhysicalDiskModel. We will be demonstrating this in some of the examples later in this chapter.

The Decision Tree Script of the PhysicalDiskModel As we have seen thus far, the PhysicalDiskModel is made up of four key components: Dynamic model elements, event elements, threshold elements, and logging elements. We have been focusing on the three specific elements: 򐂰 The TMW_SlowPhysicalDrive event element 򐂰 The HighQueueLength threshold element 򐂰 The Percent Disk Usage_PhysicalDisk logging element It is the PhysicalDiskModel decision tree script code that ties these three elements together. The PhysicalDiskModel Decision Tree script is a Visual Basic script based model that uses Microsoft’s WMI to retrieve CIM data. WMI is Microsoft’s implementation of the CIM architecture on Windows machines. The VisitTree in the PhysicalDiskModel really only does two things: 1. It generates indications. 2. It logs data. So, first let us look at how the TMW_SlowPhysicalDrive indication gets sent to the event aggregator. Note: Remember the event aggregator is not in the decision tree code. It is part of the IBM Tivoli Monitoring engine. The PhysicalDiskModel collects data from the WMI Performance Monitoring Provider (PerfProv). Most of the Windows-based resource models collect data from PerfProv as well. In the dynamic model element of the PhysicalDiskModel, all of the properties that are defined will be collected each cycle time before the VisitTree function is called.

Chapter 10. Workbench

277

The TMW_SlowPhysicalDrive indication gets generated from the VisitTree function when the CurrentDiskQLength and the DiskBytesSec exceed their threshold values. The Percent Disk Usage_PhysicalDisk logging element gets processed on every cycle regardless of whether the threshold is met or not. We can follow the code for TMW_SlowPysicalDrive and the logging elements in Example 10-5. This code is an example of the PhysicalDiskModel code and has been modified for the purposes of this discussion. Example 10-5 PhysicalDiskModel example VisitTree 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 39 30 31 32 33 34 35 36 37 38 39

Public Function VisitTree(Svc As Object) As Long Dim numAvgQLength As Long, numCurrentDiskQLength As Long, numAvgReadQLength As Long Dim numAvgWriteQLength As Long, bytDiskReadBytesSec As Long, bytDiskWriteBytesSec As Long Dim numDiskReadsSec As Long, numDiskWritesSec As Long, numDiskXfersSec As Long Dim numDiskBytesSec As Long, numPercentDiskTime As Long, numPercentDiskReadTime As Long Dim numPercentDiskWriteTime As Long, numTotalDisks As Long, flagHighPhysicalQLength As Long Dim flagHighPhysicalXferRate As Long, flagHighPhysicalReadRate As Long, flagHighPhysicalWriteRate As Long Dim flagHighPhysicalPercentTime As Long, i As Long Dim strPhysicalDisk As String

278

i = 0 numTotalDisks = Svc.GETNUMOFINST(“TMW_PhysicalDisk”) Do While i < numTotalDisks flagHighPhysicalQLength = 0 flagHighPhysicalXferRate = 0 flagHighPhysicalReadRate = 0 flagHighPhysicalWriteRate = 0 flagHighPhysicalPercentTime = 0 strPhysicalDisk = Svc.GetStrProperty(“TMW_PhysicalDisk”, i, “PhysicalDisk”) If strPhysicalDisk “_Total” Then strPhysicalDisk = Mid$(strPhysicalDisk, 1, 1) bytDiskReadBytesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskReadBytesSec”) bytDiskWriteBytesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskWriteBytesSec”) numDiskReadsSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskReadsSec”) numDiskWritesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskWritesSec”) numCurrentDiskQLength = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “CurrentDiskQLength”) numAvgQLength = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “AvgDiskQLength”) numAvgReadQLength = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “AvgDiskReadQLength”) numAvgWriteQLength = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “AvgDiskWriteQLength”) numDiskBytesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskBytesSec”) bytDiskReadBytesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskReadBytesSec”)

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88

bytDiskWriteBytesSec = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “DiskWriteBytesSec”) numPercentDiskTime = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “PercentDiskTime”) numPercentDiskReadTime = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “PercentDiskReadTime”) numPercentDiskWriteTime = Svc.GetNumProperty(“TMW_PhysicalDisk”, i, “PercentDiskWriteTime”) 'Log instance data stuff Svc.LogInst “Percent Disk Usage”, “PhysicalDisk”, numPercentDiskReadTime, strPhysicalDisk Svc.LogInst “Bytes Transferred”, “PhysicalDisk”, numDiskBytesSec, strPhysicalDisk Svc.LogInst “Queue Length”, “PhysicalDisk”, numAvgQLength, strPhysicalDisk If numCurrentDiskQLength > Svc.GetThreshold(“HighQLength”) Then flagHighPhysicalQLength = 1 End If If numDiskBytesSec > Svc.GetThreshold(“HighBytesSec”) Then flagHighPhysicalXferRate = 1 End If If bytDiskReadBytesSec > Svc.GetThreshold(“HighBytesSec”) Then flagHighPhysicalReadRate = 1 End If If bytDiskWriteBytesSec > Svc.GetThreshold(“HighBytesSec”) Then flagHighPhysicalWriteRate = 1 End If If numPercentDiskTime > Svc.GetThreshold(“HighPercentUsage”) Then flagHighPhysicalPercentTime = 1 End If If flagHighPhysicalQLength = 1 Then If flagHighPhysicalXferRate = 1 Then Svc.SENDEVENT “TMW_SlowPhysicalDrive”, _ numCurrentDiskQLength, _ numPercentDiskTime, _ numAvgQLength, _ numAvgReadQLength, _ numAvgWriteQLength, _ bytDiskReadBytesSec, _ bytDiskWriteBytesSec, _ strPhysicalDisk Else Some code deleted

Chapter 10. Workbench

279

89 90 91 92

i = i + 1 Loop VisitTree = 0 End Function

Tip: A thread exists for each resource model running on the engine. Each thread runs a simplified pseudo code loop: While (Engine Running) { CollectCIMData(); Call VisitTree(); Sleep(cycle time); }

Let us examine the code in Example 10-5 on page 278. 򐂰 Line 1 This is the VisitTree function definition. 򐂰 Line 2–10 These lines are the variable definitions. The variables that we are concerned with for TMW_SlowPyhsicalDrive are numCurrentQLength, numDiskBytesSec, and numPercentDiskTime. 򐂰 Line 14 This is the method call that returns the number of instances that have been collected by the IBM Tivoli Monitoring 5.1 engine. In our environment there were two physical disks on our W2K machine. However, the Svc.GETNUMOFINST returned three instances (disk0, disk1, and _Total). Some of the Perfmon-provided properties will return a _Total instance where it is appropriate. In line 25 we will skip this instance. 򐂰 LIne 23 When the VisitTree function is called, all of the data that was collected by the IBM Tivoli Monitoring 5.1 analyzer is available to the VisitTree function via method calls. Individual properties can be retrieved using the Svc.GetStrProperty or the Svc.GetNumPropery methods as seen in lines 26–45. In this example we are retrieving the physical disk name on each instance (that is, disk0, disk1, and _Total). 򐂰 Line 25 In the shipped PhysicalDiskModel code _Total instances is discarded. A customized model could make use of this instance. _Total is the total of all instances.

280

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Line 27 This is a call to the Svc.GetNumProperty method to retrieve the DiskReadBytesSec property. 򐂰 Line 32 This is a call to the Svc.GetNumProperty method to retrieve the CurrentDiskQLength. 򐂰 Line 42 This is a call to the Svc.GetNumProperty method to retrieve the PercentDiskTime. 򐂰 Line 48–52 These lines are unconditional calls to the Svc.LogInst methods for the three logging elements defined in Figure 10-18 on page 273. 򐂰 Line 54–56 This code checks the CurrentQLength value retrieved from CIM/WMI against the static threshold definition defined in the SetDefaulConfiguration function and sets a flag. The Svc.GetThreshold method returns values stored by the Svc.DefineThreshold in the SetDefaultConfiguration function. See Figure 10-3 on page 272. 򐂰 Line 58–60 This code checks the DiskBytesSec value retrieved from CIM/WMI against the static threshold definition defined in the SetDefaulConfiguration function and sets a flag. The Svc.GetThreshold method returns values stored by the Svc.DefineThreshold in the SetDefaultConfiguration function. See Figure 10-3 on page 272. 򐂰 Line 74–84 If the HighPhysicalQLength and HighPhysicalXfreRate flags are set, then the Svc.SENDEVENT method is invoked, passing all the TMW_PhysicalDisk class properties retrieved in lines 27–46. 򐂰 Line 89–90 This line increments and loops back to get the next instance. 򐂰 LIne 91–92 Return back to the IBM Tivoli Monitoring 5.1 engine.

10.3 Resource model examples This section will explore the techniques of customizing existing resource models.

Chapter 10. Workbench

281

10.3.1 ITSO_PhysicalDiskModel example If we were to look at the raw data that Perfmon provides for the physical and logical disk performance objects, we might see that the total of PercentDiskRead and PercentDiskWrite time do not always total PercentDiskTime. This is because PercentDiskTime caps at 100 percent and on very busy drives (for example, SCSI and RAID devices) the calculation for disk utilization can exceed 100 percent. Microsoft’s fix for this in NT 4.0 was to add a new counter that is not capped called AvgDiskQLength. This counter is in decimal format so if the PercentDiskTime is 83 percent then the AvgQLength will be 0.83. The Perfmon GUI actually converts the AvgDiskQLength value to percentage and displays it as PercentDiskTime, as can be seen in Figure 10-22. The IBM Tivoli Monitoring 5.1 shipped PhysicalDisk and LogicalDisk models both use the capped PercentDiskTime value in the decision tree logic. In this example we are going to emulate what the Perfmon GUI does in our new resource model by recalculating the PercentDiskTime based on the AvgDiskQLength counter. The algorithm we are going to use in the Visual Basic Script code is as follows: PercentDiskTime = AvgQLength * 100

Tip: The first example will demonstrate how to customize an existing resource model. The source code for this example can be found in Appendix A, “Additional material” on page 529. All of the source is contained in ITSO_PyhsicalDiskModel.dmws.

Figure 10-22 Perfmon GUI display of PercentDiskTime (not capped)

282

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Cloning a resource model To create our new resource model in this example we are going to clone TMW_PhysicalDiskModel. Our new resource model will be named ITSO_PhysicalDiskModel. There really is not a cloning feature in the Workbench, so we are going to copy an existing model and save it as a new one. In this example we are going to open the TMW_PhysicalDiskModel.dmws project file and immediately save it as ITSO_PhysicalDiskModel.dmws, as seen in Figure 10-23.

Figure 10-23 Cloning TMW_PhysicalDiskModel

Changing the elements in our new resource model Since the goal of this model is to calculate the PercentDiskTime the same way the Perfmon GUI does, we are going to have to change the Visual Basic script decision tree code as described above. However, before we make that change we will have to modify some of the elements defined for our new model so that it will be unique. In this example we will walk through each of the elements that will need to be changed for this new ITSO_PyhsicalDiskModel resource model.

Changing the properties of our new resource model First we must change the general settings (properties) of our new resource model. To open up the properties of a resource model, we can double left-click the mouse or right-click the root node object. In this case it is the TMW_PhysicalDisk object. We see the display depicted in Figure 10-24 on page 284.

Chapter 10. Workbench

283

Figure 10-24 ITSO_PhysicalDiskModel general settings

The following are descriptions of the properties:

284

Internal name

This is the new name of our model ITSO_PyhsicalDiskModel.

Category internal name

ITSO will be the name we will use to group all of the example resource models created in this chapter.

Category descriptive name

This is the name that is displayed in the profile category display. An example of this can be seen in the category pull-down options in Figure 10-25 on page 285.

Descriptive name

This will be used as the display name for the resource model.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-25 ITSO resource models

Renaming the event elements When we are cloning an existing resource model, we need to make sure the event elements are unique. In this example we are going to rename all of the event elements to ITSO_, as seen in Figure 10-26 on page 286.

Chapter 10. Workbench

285

Figure 10-26 Renaming the event elements

Changing the ITSO_PhysicalDiskModel decision tree For this example the code change was a very simple one. All we had to do was add one statement in the Visual Basic script code to recalculate the PercentDiskTime based on theAvgQLength value, as seen in Figure 10-27 on page 287.

.

286

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-27 Recalculating PercentDiskTime

Testing ITSO_PhysicalDiskModel using the Workbench For Visual Basic script resource models, the Workbench links to the Sax Basic Editor and Debugger by SaxSoft.com. For more details on the SaxSoft product see 10.4.4, “Saxsoft” on page 380. The help for debugging resource models in the Workbench was not that robust, so the first thing we did was to download the Sax Basic Editor. There we found the online help dialog that comes with the Sax Basic Editor to be very helpful in explaining the debugging pane seen in Figure 10-30 on page 290. Before we begin testing, we need to change the threshold element related to PercentDiskTime to a number higher than 100 to ensure that our new model really works the way we want it to (that is, PhysicalDiskTime > 100). We are going to change the HighPercentUsage threshold element to 101 for our testing,

Chapter 10. Workbench

287

as seen in Figure 10-28. Remember, these changes will automatically update the SetDefaultConfiguration function in our resource model. For testing purposes we also changed the default cycle time to 10 seconds in the general settings for the resource model. Note: We cannot change any of the resource model elements while in debug mode. When we are about to debug, it is best to make any threshold element changes we are going to need for debugging purposes before invoking the debugger. Then after we are finished debugging, set them back to normal best practice numbers.

Figure 10-28 Changing the HighPercentUsage threshold element

In order to test this resource model, we had to simulate a busy disk drive that would cause the AvgDiskQLength value to exceed 1.0 (that is, our new PercentDiskTime > 100 percent). For this exercise we used the a utility provided by the Microsoft NT Resource Kit called probe. We have set up a bat file called diskmax.bat that will emulate maximum throughput by reading sequential 64 K records from a 20 Mb file without using the file system cache. The samples of these files are in 10.4.3, “Microsoft tools” on page 379. This utility is also used in the Tivoli Distributed Monitoring public education class during resource model labs. See http://www.tivoli.com/services/education/ for more details on IBM/Tivoli education.

288

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

To start debugging our ITSO_PhysicalDiskModel, we need to select DecisionTree -> Run or press function key 5 on the keyboard. See Figure 10-29 for an example. This will start running the main function in the decision tree code. This is a good way to syntax check the decision tree code. If there is a syntax error in the code, the Workbench will post an error pop-up window and highlight the failing instruction in red.

Figure 10-29 Starting the debugger for Visual Basic based resource models

After the debugger is running for a few seconds and we have not seen any syntax errors, we can then pause the debugger to add some of the variables we want to follow to the Watch window. To select the pause, use DecisionTree -> Pause. Since most of the time the decision tree code is waiting (that is, the cycle time), it should pause on Wait TMWService.GetCycleTime() in the main function. This can be seen in Figure 10-30 on page 290.

Chapter 10. Workbench

289

Figure 10-30 Running the Debugger in the Workbench

The next step is to add some variables to the Watch window so we can see their values change while we are stepping through the code. An easy way to add variables to the Watch window is by positioning the cursor over the variable name in the VisitTree function and using the Debug -> Add Watch menu option or Ctrl+F9, as seen in Figure 10-31 on page 291. In this example we are going to watch the following variables:

290

numAvgQLength

This is the value pulled from PerfProv that has the AvgDiskQLength we want to use to recalculate PercentDiskTime.

numPercentDiskTime

This is the value we are going to change based on the new calculation (numAvgQLength * 100).

numTotalDisks

This is the total number of instances that will be returned. PervProv will return a single instance for each physical disk plus an instance called _Total for the total of all disks.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

strPhysicalDisk

This is the name of the physical disk for the current instance.

In our debugging example we are going to run the diskmax.bat script on the D: physical disk. When we are debugging, we want to skip past the C: physical disk. As we will see in the code, the _Total is already skipped in the code, as seen in Figure 10-34 on page 295.

Figure 10-31 Adding variables to the Watch window

Since we are still paused at this time, in the Main function we will get some error messages for the four variables we have added to the Watch window, as seen in Figure 10-32 on page 292. This is okay because the variables have not been defined yet. These variables will be initialized when the VisitTree function is called. Just ignore the error messages at this point.

Chapter 10. Workbench

291

Figure 10-32 Watch window error messages

Now we can begin debugging our code using the Workbench. In the Workbench there are a number of debugging functions that can be used. Below is a list of some of the debugging tools provided by the Sax Basic Editor that can be used in the Workbench:

292

Step into

Execute the current line. If the current line is a subroutine or function call, stop on the first line of that subroutine or function. Use Debug -> Step Into or F8.

Step over

Execute to the next line. If the current line is a subroutine or function call, execute that subroutine or function completely. Use Debug -> Step Over or Shift+F8.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Step out

Step out of the current subroutine or function call. Use Debug -> Step Out or Ctrl+F8.

Step to cursor

Execute until the line the cursor is on is the current line. Use Debug -> Step to Cursor or F7.

Toggle break

Toggle the break point on the current line. Use Debug -> Toggle Break or F9.

Clear all breaks

Clear all break points. Use Shift+Ctrl+F9. Note that this option is not listed on the Debug menu but can still be used.

Quick watch

Show the value of the expression under the cursor in the immediate window. Use Debug -> Quick Watch or Shift+F9.

Add watch

Add the expression under the cursor in the watch window. Use Debug -> Add Watch or Ctrl+F9.

Browse

Show the methods of the expression under the cursor. Use Debug -> Browse. To see language extensions, built-in instructions/functions, and user-defined procedures/variables, press Ctrl+Space on a blank line anywhere in the VisitTree function.

Set next

Set the next statement to be executed. Only statements in the current subroutine/function can be selected. Use Debug -> Set Next.

Show next

Show the next statement to be executed. Use Debug -> Show Next.

At this point we are still paused at the wait statement in the Main function. We can use the Step to Cursor option to stop the debugger at VisitTree. If we place our cursor on the VisitTree function in the Visual Basic script code, we can then select the F7 key to have the debugger stop at the beginning of VisitTree, as seen in Figure 10-33 on page 294.

Chapter 10. Workbench

293

Figure 10-33 Using the Step to Cursor option

At this point we can either set breakpoints or just use the Step Into option. In this example we used the Step Into option to walk through the code. The first thing we want to do for this example is to make sure we are in physical drive 1 (that is, the D: drive). This is the drive that we are going to run the diskmax.bat scripts from. This can be seen in Figure 10-34 on page 295. We can see in the Watch window all of the variables and their values.

294

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-34 Watch window example

Next we need to step past the new line of code we added in this example. This is the line where we are recalculating the PercentDiskTime value. This can be seen in Figure 10-35 on page 296.

Chapter 10. Workbench

295

Figure 10-35 Watch window example

Notice that in the Watch window all the variables are set to the values we expected (that is, numPercentDiskTime = 200 and numAvgQLength is 2). That is a very busy drive. Also notice that the Svc.LogInst method will now log the new PercentDiskTime that will be more in line with what Perfmon GUI displays. After we are sure our calculation is working correctly, we can just let the resource model run and watch the event aggregator Status pane at the bottom of the screen. We wait for an indication to be generated. In this case ITSO_HighPhysicalDiskReadBytesSec was set to 10 occurrences and 1 hole. Figure 10-36 on page 297 shows that an indication would have been sent if this was running from the IBM Tivoli Monitoring 5.1 analyzer and not from the Workbench.

296

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-36 Indication Sent from the Debugger

Building the ITSO_PhysicalDiskModel resource model After we have successfully tested our new resource model in the Workbench, we need to start preparing it for usage with the Tivoli environment. The Build menu option allows us to build some of the necessary files required to use the resource model. This can be seen in Figure 10-37 on page 298.

Chapter 10. Workbench

297

Figure 10-37 Build menu items

The following is a description of some of the build options we will use for the ITSO_PyhsicalDiskModel resource model: Build package

298

We use this option to create the tar files that we will use to add to the IBM Tivoli Monitoring 5.1 environment. We will create an ITSO_PhysicalDiskModel.tar file. Then we will add this file to IBM Tivoli Monitoring 5.1 by issuing the wdmrm command from the TMR Server. See 10.4.2, “Resource model internal format” on page 377, for details on the format of the tar file.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Build TEC BAROC

We will use this option to generate output TEC BAROC files that we will have to import, compile, and load into our TEC Server in order to receive TEC events.

Build .Html Documentation

We will use this option to create HTML documentation for our new resource model. An example of the ITSO_PhysicalDiskModel HTML documentation generated in this example can be seen in Figure 10-38.

Figure 10-38 ITSO_PhysicalDiskModel HTML documentation

Chapter 10. Workbench

299

We are now ready to add the ITSO_PhysicalDiskModel.tar file to IBM Tivoli Monitoring 5.1 by using the wdmrm command. First we need to FTP the built tar file from our development system over to our TMR Server. Then we need to enter the following command on the TMR Server: wdmrm -add ITSO_PhysicalDiskModel.tar

For more information on the wdmrm command please see the IBM Tivoli Monitoring 5.1 User’s Guide, SH19-4569. Next we create a Tmw2kProfile with the new ITSO_PhysicalDiskModel and distribute it to a Windows 2000 endpoint. We then start our diskmax.bat scripts on the endpoint and wait for the indication. After running diskmax.bat for a couple of minutes on the endpoint, we issued a wdmlseng -e tividc15 -verbose command. The output of this command can be seen in Figure 10-39.

Figure 10-39 wdmlseng command for the ITSO_PhysicalDiskModel

Finally, an event is not an event unless we can see it in TEC. Figure 10-40 on page 301 is an example of our generated event from this exercise.

300

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-40 ITSO_PhysicalDiskModel TEC event

10.3.2 ITSO_LogicalDiskModel example Tip: In this example we will demonstrate how to add a new resource model with a new class definition. The source code for this example can be found in Appendix A, “Additional material” on page 529. All of the source is contained in ITSO_LogicalDisk.dmws. In the previous example we resolved some of the issues using the Perfmon provided PercentDiskTime disk counter. In Windows 2000, Microsoft added a new counter that is a better metric for calculating disk time. A new counter called PercentIdleTime was added to both the logical and physical disk objects. This counter accumulates when there are no outstanding requests for a disk. Using this value we can now calculate the Percent Disk Busy Time by using the following algorithm: PercentDiskBusy = 100 - PercentIdleTime

The current shipped versions of the IBM Tivoli Monitoring 5.1 PhysicalDiskModel and LogicalDiskModel resource models do not include the new PercentIdleTime property. In this example we will demonstrate how to create a new resource model and define a new resource class.

Chapter 10. Workbench

301

Note: On Windows 2000 and NT systems the PhysicalDisk counters are only turned on by default. In order to use properties in the TMW_LogicalDisk resource class in IBM Tivoli Monitoring 5.1, the Perfmon LogicalDisk counters must be turned on and the endpoint has to be rebooted. To turn on the LogicalDisk counters, enter the following command from a cmd.exe window: diskperf -yv

The -yv enables the LogicalDisk performance counters when the system is restarted. Just a -y enables both PhysicalDisk and LogicalDisk.

Creating a new Resource Class In CIM/WMI there are two ways to create classes. The first way is through the WMI programming interface, either through the scripting API or the C++ COM API. See the following URL for more information: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmisdk/wmi/scr ipting_api_for_wmi.asp

Even though the APIs provide a rich set of methods to update the CIM/WMI repository, most modifications are done by using the MOF Compiler command. A Managed Object Format (MOF) file is a language created by the DMTF as part of Web-based Enterprise Management (WBEM). MOF is based on the Interface Definition Language defined by DCE, the Open Group IDL. Note: The DCE IDL is not the same as other IDLs. For example, OMG’s IDL for CORBA is used conceptually for the same purpose, but the two languages are different. Since even a short overview of MOF would take a whole chapter, we are going to just point out the options we used in our MOF file. The dmtf.org publishes the CIM V2 specification. The CIM V2 specification can be downloaded and is a great resource for learning about the MOF language and concepts. The CIM V2 specification can be found at: http://www.dmtf.org/standards/cim_spec_v22/index.php

In this example, in order to add our new resource class, we will have to create a MOF file. For this example we are going to create a new resource class based on the TMW_Resource class. Since we know that we are going to be using the LogicalDisk Perfmon object, we can use the TMW_LogicalDisk resource class as an example for our MOF. There are two ways we can create our MOF from the existing TMW_LogicalDisk class. The first way is to find the shipped IBM Tivoli Monitoring 5.1 MOF files, and cut and paste the TMW_LogicalDisk MOF statements into a new file. The second way is to use the WMI SDK CIM Studio

302

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

tool provided by Microsoft. CIM Studio is an excellent tool for working with CIM/WMI objects. The CIM Studio is an advanced CIM browser and a MOF Generator. See Figure 10-41 for an example of using the CIM Studio MOF Generator. For more information on CIM Studio, see 10.4.3, “Microsoft tools” on page 379.

Figure 10-41 CIM Studio MOF Generator

We used the CIM Studio MOF Generator to create a new MOF file based on the TMW_LogicalDisk class, as seen in Figure 10-41 on page 303. We saved the file as ITSO_LogicalDisk and removed most of the properties out of the file. Then we added the new PercentIdleTime property, as can be seen in Example 10-6.

Chapter 10. Workbench

303

Example 10-6 ITSO_LogicalDisk.mof //************************************************************************** //* File: ITSO_LogicalDisk.mof //************************************************************************** #PRAGMA NAMESPACE("\\\\.\\ROOT\\CIMV2") [DYNAMIC: ToInstance, PROVIDER(“PerfProv”), CLASSCONTEXT(“local|LogicalDisk”)] CLASS ITSO_LogicalDisk : TMW_Resource { [KEY, PROPERTYCONTEXT(“Logical Disk”)] STRING LogicalDisk; [PROPERTYCONTEXT(“Avg. Disk Queue Length”)] UINT32 AvgDiskQLength; [PROPERTYCONTEXT(“% Idle Time”)] UINT32 PercentIdleTime; };

The structure of the MOF is fairly obvious. However, the following is a detailed explanation of the ITSO_LogicalDisk.mof:

304

PRAGMA

The PRAGMA is a directive that specifies which name space our new class is to be added to. All of the IBM Tivoli Monitoring 5.1 Windows TMW based resource classes are defined in the root\CIMV2 name space and the UNIX DMX based resource classes are defined in root/DEFAULT.

PROVIDER

The text inside the square brackets is called qualifiers. The first qualifier in this MOF is a PROVIDER qualifier. The PROVIDER qualifier specifies the name of the provider that is responsible for implementing class methods. In this case we are using Microsoft provider PerfProv.

Class

The class is the name of our new resource class. In this case it is ITSO_LogicalDisk.

KEY

The KEY is also a qualifier. In this example the KEY qualifier is the logical disk property. The LogicalDisk will be the key for our new ITSO_LogicalDisk class. The STRING LogicalDisk are the data type and property name that will be used for the perfmon logical disk counters.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

PROPERTYCONTEXT

The last two qualifiers define two more properties that will be referenced in our new resource class. The PercentIdleTime is the new property that we want to define.

Tip: If we are going to add new Perfmon counters to a resource class, we will need to get the actual name defined by the Perfmon Counter Name. We can get this name by opening the perfmon GUI or by using a utility called the Performance Data Block Dump Utility on the Windows 2000 Resource Kit. See 10.4.3, “Microsoft tools” on page 379, for more details on the Windows 2000 Resource Kit. After we have defined our resource class as a MOF file, we will need to compile it. We can compile it using the mofcomp.exe command or by using the Workbench. The easiest way to compile a MOF file is to invoke the mofcomp.exe command as follows: mofcomp ITSO_LogicalDisk.mof

An example of this can be seen in Figure 10-42.

Figure 10-42 Compiling a MOF from the command line

We could have also compiled our MOF from the Workbench. To compile a MOF from the Workbench, we need to open up the CIM browser from the Workbench and select the MOF Compiler icon, as seen in Figure 10-43 on page 306. Note: The placement of the MOF Compiler icon is different depending on how we enter the CIM browser. If we enter the CIM browser from a wizard, the icon is in the middle lower part of the window. If we enter the CIM browser by selecting a dynamic model element, the icon will be in the upper right.

Chapter 10. Workbench

305

Figure 10-43 Using the MOF Compiler from the Workbench

Creating a resource model using the wizard Now that we have compiled our new ITSO_LogicalDisk.mof file and a new ITSO_LogicalDisk resource class exists in the root/CIMV2 name space, we can begin to build our resource model. There are three different ways to create a resource model from the Workbench. The three methods are as follows: Resource model wizard

306

The resource model wizard provides a series of dialog boxes that guide us through the whole process of creating a simplified resource model without writing scripts or code.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Step-by-step resource model

This approach provides dialog boxes that guide us through the procedure in the order defined in the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569. However, when we have completed this procedure, we must write the script that governs the resource model.

Empty resource model

Using this approach, we can define the resource model elements without following a specific order.

Note: The redbook Introducing Tivoli Distributed Monitoring Workbench Version 4.1, SG24-6534, listed some limitations of using the wizard. In our experience these are only limitations if we do not fully understand the elements of a resource model. 򐂰 When we use the resource model wizard, it prompts us to select a resource class. While we are in the panel, we can only select one class. However, after the wizard is complete we can always add more class definitions. 򐂰 The resource model wizard builds a limited-functionality VisitTree routine without a lot of complex correlation (that is, if-then-else logic). Again, if we understand how the Decision Tree element works and we know how to code, this is not a limitation. In most cases we have to create or modify the VisitTree code regardless of how it was created. In our experience it is always better to start out using the resource model wizard to generate the template for the resource model, unless we are cloning a resource model (that is, the first example). The resource model wizard generates a good structure for the VisitTree function. If resource models are developed using the resource model wizard, this will force a level of conformity. In this example we choose to use the resource model wizard. First we need to select File -> New to get a list of the languages available for our new resource model. In this case, because it is a Windows/WMI-based model, we choose VBA Resource Model, as can be seen in Figure 10-44 on page 308. VBA is the Visual Basic Script option.

Chapter 10. Workbench

307

Figure 10-44 Resource model wizard languages

Next the wizard will prompt for the options available to create the resource model. As we stated earlier, we almost always select the resource model wizard option. Figure 10-45 is an example of the window listing the wizard options.

Figure 10-45 Resource model wizard methods

Restrictions: There are some limitations when using JavaScript with the IBM Tivoli Monitoring 5.1 Workbench. The Workbench does not support JavaScript for Windows at this release. It also does not support debugging of UNIX resource models at the current release. Full support in the Workbench for JavaScript on Windows and UNIX platforms will be available in a future release. After we select the resource model wizard, the wizard process will prompt us through a set of windows asking for different information pertaining to the elements of our new resource model.

308

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The following is a walkthrough and explanation of the resource model wizard process that was used for the ITSO_LogicalDisk example. The first window asks for the type of resource model that will be created, as seen in Figure 10-46.

Figure 10-46 Resource model wizard data source type

Chapter 10. Workbench

309

The data source types are as follows: CIM/WMI

This tells the wizard whether this model will be a CIM-based resource model or a compatibility mode model.

DM Classic Edition Collection

This tells the wizard that we are creating a compatibility mode resource model and that we will be migrating a MCSL or CSL based collection into this resource model. For more information on this, see Chapter 5, “Migrating Classic Edition monitors” on page 111.

Custom script

This tells the wizard that we are creating a resource model based on a custom numeric or string monitor. This is also used for compatibility mode monitors. For more information on this, see Chapter 5, “Migrating Classic Edition monitors” on page 111.

The next few screen shots will be examples of how we created the ITSO_LogicalDisk resource model.

Dynamic model element for ITSO_LogicalDisk The first screen after we select the resource model wizard selection will ask us to sign on to the CIM browser. This will start the process of defining the ITSO_LogicalDisk resource elements. An example of signing on to the CIM browser can be seen in Figure 10-11 on page 261. The first screen after we sign on to the CIM browser will display all of the compiled resource classes in the root/CIMV2 name space. This is where we need to define the dynamic model element of our new resource model. The dynamic model element in this case will be the new resource class we defined in Example 10-6 on page 304. We can do a find for the ITSO_LogicalDisk resource class that we compiled in Figure 10-42 on page 305. Figure 10-47 on page 311 is an example of using the find to locate our new resource class.

310

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-47 Resource model wizard - Finding ITSO_LogicalDisk resource class

After we find the ITSO_LogicalDisk resource class, we need to make sure it has the correct properties we are planning on using (specifically, PercentIdleTime). An example of the selected ITSO_LogicalDisk resource class can be seen in Figure 10-48 on page 312.

Chapter 10. Workbench

311

Figure 10-48 Resource Model Wizard: Select a Class window

We then select the Next button to define the properties in the resource class. In this example we are going to use the following properties: 򐂰 AvgDiskQLength 򐂰 LogicalDisk 򐂰 PercentIdleTime These can be seen in the Figure 10-49 on page 313.

312

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-49 Resource model wizard - Defining properties

In the ITSO_LogicalDisk example we also added a WQL filter for PercentIdleTime, as seen in Figure 10-50 on page 314.

Chapter 10. Workbench

313

Figure 10-50 Resource Model Wizard: Filtering window

In this example we are going to add a filter so that we will never retrieve data for LogicalDisk less than 70 percent idle (that is, more than 30 percent busy). It would have been nice to use a PercentBusy time here as a filter. The problem with that is that PerfProv does not supply a PercentBusyTime property. This is a variable we are going to create in our VisitTree function. We can only use WQL filters on WMI properties. Note: Filtering is not supported for UNIX resource models in the current IBM Tivoli Monitoring 5.1 release.

314

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Event and threshold elements for the ITSO_LogicalDisk The next window in the wizard is the Event Triggering Conditions screen. This screen will prompt us to define event and threshold elements for the new resource model. In the ITSO_LogicalDisk resource model example, we are going to select the PercentIdleTime and the AvgQLength properties as the event elements, as can be seen in Figure 10-51.

Figure 10-51 Resource model wizard event elements

Note: Notice that in this example we have defined PercentIdleTime as greater than 75 percent. This is not a mistake. We are actually going to create a new variable called PercentBusyTime in the VisitTree code and rename the event element and property as PercentBusyTime after the wizard has completed.

Chapter 10. Workbench

315

By selecting both PercentIdleTime and AvgQLength properties, the wizard will generate two event elements. Therefore, ITSO_LogicalDisk should have two indications. We took the defaults on all of the other event element settings. By selecting triggering conditions on each of the properties, the wizard will also create two threshold elements—one for PercentIdleTime and one for AvgQLength.

Logging elements for ITSO_LogicalDisk The resource model only generates one call to Svc.LogInstEx for all of the selected properties. Again, we can always go into the VisitTree code and change this after the wizard completes. In this example we are going to select all three properties to log every time the Decision Tree code gets invoked. An example of the wizard screen for logging elements can be seen in Figure 10-52.

Figure 10-52 Resource model wizard logging elements

Before the resource model wizard completes, it will display one last screen prompting for the default cycle time. This can be seen in Figure 10-53 on page 317.

316

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-53 Resource model wizard default cycle time

This will complete the resource model wizard.

Modifications after the resource model wizard completes After the resource model wizard completes, it generates all of the default elements based on the selections we made during the wizard. An example of a resource model wizard generated model can be seen in Figure 10-54 on page 318.

Chapter 10. Workbench

317

Figure 10-54 Resource model wizard generated model

Before we build and start using this resource model, we are going to make a few modifications. The first modification we will do is rename the resource model from the generated name to ITSO_LogicalDisk. We will also modify some other general settings. We will change the category and category descriptive name so that ITSO_LogicalDisk displays in the IBM Tivoli Monitoring 5.1 profile under the Add Resource Model pull-down window. All of the ITSO resource models will be defined to display as ITSO Resource Models. Figure 10-55 on page 319 shows how we modified these general settings in the ITSO_LogicalDisk resource model.

318

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-55 ITSO_LogicalDisk General Settings window

Chapter 10. Workbench

319

The next modification we need to make is to rename the event elements to ITSO_. We also want to use a new variable, PercentBusyTime, instead of the PercentIdleTime. To do this, we are going to rename the PercentIdleTime event element that the wizard generates to PercentBusyTime. We will also have to remove the PercentIdleTime property from the event definitions and add the new PercentBusy Variable. An example of these changes can be seen in Figure 10-56 on page 321, pointed out by the arrows. Note: If we change an event element in the Workbench IDE, the Workbench will automatically update all references to the event in the SetDefaultConfiguration function, as well as references in the Svc.method calls. However, it will not modify any logic changes in the VisitTree code.

320

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-56 Changing the event elements

Chapter 10. Workbench

321

Next we need to modify the wizard-generated threshold elements. This is done mainly just for consistency purposes. The wizard-generated threshold element was based on the PercentIdleTime property. Now that we are going to use PercentBusyTime, we will change the name of the threshold element, as can be seen in Figure 10-57.

Figure 10-57 Changing the threshold elements in ITSO_LogicalDisk

In the next step we need to modify the LogicalDisk logging element. Here again, we want to replace the PercentIdleTime with the new PercentBusyTime that we are going to create in the VisitTree code. An example of this can be seen in Figure 10-58 on page 323.

322

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-58 Changing the logging element in ITSO_LogicalDisk

Next we need to add our new resource class (that is, ITSO_LogicalDisk.mof) as a dependency element. The MOF file for the new resource class needs to be added to the resource model so that when the resource model is distributed, the IBM Tivoli Monitoring 5.1 engine can compile the MOF in the local CIM database. To add a dependency element we need to right-click the mouse button on the platform-specific dependency we want to add. In this case we want to add the ITSO_LogicalDisk.mof file to the w32-ix86 platform. An example of adding a dependency element can be seen in Figure 10-59 on page 324.

Chapter 10. Workbench

323

Figure 10-59 Adding a dependency element

Finally we need to make some modification to our VisitTree code. We are going to make the following modifications: 򐂰 򐂰 򐂰 򐂰 򐂰

324

Add a new PercentBusyTime variable. Calculate the new PercentBusyTime based on (100 - PercentIdelTime). Add logic to skip the _Total instance. Change the PercentIdleTime comparison logic to PercentBusyTime. Move some of the Svc.method calls to support the logic changes.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Example 10-7 is an example of the VisitTree code that was modified after the wizard generation, followed by an overview of some of the code changes. Example 10-7 ITSO_LogicalDisk VisitTree function 1 Public Function VisitTree(Svc As Object) As Long 2 3 Dim curPercentIdleTime As Double 4 Dim curPercentBusyTime As Double 5 Dim curAvgDiskQLength As Double 6 Dim curLogicalDisk As String 7 8 Dim hPropTable As Integer 9 Dim numOfInstances As Long 10 Dim idx As Long 11 Dim ParamCount As Long 12 Dim ParamIdx As Long 13 Dim Different As Boolean 14 15 hPropTable = Svc.CreateMap() 16 17 numOfInstances = Svc.GETNUMOFINST(“ROOT\CIMV2:ITSO_LogicalDisk”) 18 For idx = 0 To numOfInstances - 1 19 20 curLogicalDisk = Svc.GetStrProperty(“ROOT\CIMV2:ITSO_LogicalDisk”, idx, “LogicalDisk”) 21 22 'if we are looking at _Total, do not run through the threshold checks 23 If curLogicalDisk “_Total” Then 24 25 Svc.RemoveMapAll(hPropTable) 26 Svc.SetMapStrElement(hPropTable,”LogicalDisk”,curLogicalDisk) 27 28 curPercentIdleTime = Svc.GetNumProperty("ROOT\CIMV2:ITSO_LogicalDisk", idx, “PercentIdleTime”) 29 curPercentBusyTime = 100 - curPercentIdleTime 30 Svc.SetMapNumElement(hPropTable,"PercentBusyTime",curPercentBusyTime) 31 32 curAvgDiskQLength = Svc.GetNumProperty("ROOT\CIMV2:ITSO_LogicalDisk", idx, “AvgDiskQLength”) 33 Svc.SetMapNumElement(hPropTable,"AvgDiskQLength",curAvgDiskQLength) 34 35 If (curPercentBusyTime > Svc.GetThreshold("Thr_PercentBusyTime_gt") ) Then 36 37 Svc.SetMapNumElement(hPropTable,"UpperBound",Svc.GetThreshold("Thr_PercentBusyTime_gt")) 38 Svc.SendEventEx "ITSO_LogicalDisk_PercentBusyTime_too_high", hPropTable 39 End If 40 41 If (curAvgDiskQLength > Svc.GetThreshold(“Thr_AvgDiskQLength_gt”) ) Then 42

Chapter 10. Workbench

325

43 Svc.SetMapNumElement(hPropTable,"UpperBound",Svc.GetThreshold(“Thr_AvgDiskQLength_gt”)) 44 Svc.SendEventEx “ITSO_LogicalDisk_AvgDiskQLength_too_high”, hPropTable 45 End If 46 47 Svc.LogInstEx (“ITSO_LogicalDisk_Availability”,"ITSO_LogicalDisk", hPropTable) 48 49 End If 50 Next 51 52 Svc.DestroyMap(hPropTable) 53 54 VisitTree = 0 55 56 End Function

Let us examine the code in Example 10-7 on page 325. 򐂰 Line 4 Add the new PercentBusyTime variable. 򐂰 Line 15 Create a new map. The resource model wizard generates code using the new IBM Tivoli Monitoring 5.1 mapping table APIs. The Svc.CreateMap creates a new map. The new SendEventEx and LogInstEx methods require mapping tables as input. All new resource models should use the SendEventEx and the LogInstEx instead of the deprecated SendEvent and LogInst methods. 򐂰 Line 22–23 The resource model wizard generated code built a loop to process all instances returned by CIM/WMI. The PerfProv provider will return an instance for each logical disk and also a _Total instance. In this line we added code to ignore the _Total instance. 򐂰 Line 25 On each iteration we clear our defined mapping table (that is, hPropTable). 򐂰 Line 26 The Svc.SetMapStrElement will add a string element to the hPropTable, in this case the name of the logical disk (that is, D:). 򐂰 Line 29–30 This is the new calculation of PercentBusyTime. We are also adding this numeric element to our hPropTable map. 򐂰 Line 35 This the new comparison against the PercentBusyTime threshold and the call to generate the indication if the threshold is exceeded.

326

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Line 47 This an unconditional call to log PercentBusyTime and the LogicalDisk. Note: The SendEvent and LogInst methods are documented as deprecated methods (that is, they are not recommended). All new resource models should use the SendEventEx and the LogInstEx methods instead. Next we tested our new ITSO_LogicalDisk resource model using the Workbench. We started the resource model in debug mode and ran the diskmax.bat scripts until an event was generated. The event can be seen in the red line in Figure 10-60 on page 328.

Chapter 10. Workbench

327

Figure 10-60 Testing the ITSO_LogicalDisk resource model

10.3.3 ITSO_LoadAvg example This example will demonstrate how to use the step-by-step wizard to create a UNIX-based resource model using JavaScript. It will also demonstrate how to use trace logs and the Rhino tool for debugging. The source code for this example can be found in Appendix A, “Additional material” on page 529. All of the source is contained in ITSO_LoadAvg.dmjsws.

328

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

In this example we are going to create a new resource model that uses the LoadAvg property as a threshold. We are going to use the DMXCpu resource model as a template for our new ITSO_LoadAvg. The shipped DMXCpu resource model decision tree logic uses the PercentCpuIdle time and the PercentCpuSystem time for its threshold logic. However, in the DMXCpu resource class, LoadAvg1, LoadAvg5, and LoadAvg15 are properties that are available. These properties are not used in the DMXCpu decision tree logic (that is, for threshold checking); they are only logged. In this example we are going to create a new resource model that uses LoadAvg1 in the decision tree logic. LoadAvg1 is the CPU-run queue length average over a one minute period. The LoadAvg can be another indicator of a busy CPU. Tip: How to find the MOF for the UNIX/Linux-based resource models: When an IBM Tivoli Monitoring 5.1 profile is distributed to a Windows machine, the engine loads and compiles all of the TMW_ MOF files on the first push. The TMW_Resource10.mof file contains all of the Windows resource class definitions in one file. On UNIX/Linux-based resource models, the MOF files are included as part of the resource model tar file. When a UNIX/Linux resource model is pushed to an endpoint, the engine compiles only the MOFs associated with that resource model for the selected resource model. To view a UNIX/Linux based resource class MOF file, the actual MOF file needs to be extracted. From the Workbench, open the selected (*.dmjsws) project file and right-click on the ALL dependency element. In this example we need to extract the DMXCpu.mof file from the DMXCpu.dmjsws project file. An example of this can be seen in Figure 10-61 on page 330. We will save a copy of the DMXCpu.mof file in a local directory so that we can later compile it on a Windows machine using the CIM browser MOF compiler.

Chapter 10. Workbench

329

Figure 10-61 Extracting Resource Class MOF from UNIX/Linux resource model

Note: It is important to note there are at least two MOF files associated with each resource model. One MOF file is the resource class definitions. The other MOF file is the one that defines the model-to-resource relationship. In the UNIX/Linux resource models, the MOF that we add or extract from the dependency element is the resource class definition. The MOF file that gets created from the Workbench menu Build -> Export Mof is the model-to-resource MOF. This file contains mostly event/indication information. By default, both MOF files will have the same name.

330

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Example 10-8 is an example of the DMXCpu MOF file. Example 10-8 DMXCpu resource class //************************************************************************** //Distributed Monitoring (Advanced Edition) V4R1 (tm) //Product ID=5698-EMN //(C)Copyright IBM Corp. 1999, 2001 All Rights Reserved. //US Government Users Restricted Rights - Use, duplication or //disclosure restricted by GSA ADP Schedule Contract with IBM Corp. //************************************************************************** [ Description (“Unix CPUs info”), provider(“com.tivoli.dmunix.ep.touchpoint.cimom.ifc.M12JavaProvider”), M12_Instrumentation {"Java.com.tivoli.dmunix.ep.ilts.DMXCpuIlt | | ENUM", "Java.com.tivoli.dmunix.ep.ilts.DMXCpuIlt | | GET"} ] class DMXCpu { [key]string spare; real32 idleTime; real32 userTime; real32 sysTime; real32 loadAvg1; real32 loadAvg5; real32 loadAvg15; };

On UNIX/Linux platforms the provider environment is Java. The provider agent is incorporated in the IBM Tivoli Monitoring 5.1 product based on CIM specifications. This instrumentation is called Touchpoint. At a macro level Touchpoint can be viewed as a single component that provides a standard interface to managed resources for use by management tools. Touchpoint is tightly associated with the resource, and provides very little (if any) functionality unless controlled by a management tool. There are two surfaces to the Touchpoint—the Touchpoint Interface (TSL = Touchpoint Service Layer) and the Resource Interface (ILT = Instrumentation Library Type). Looking at DMXCpu.mof we see that the LoadAvg1, LoadAvg5, and LoadAvg15 are properties that are available in the DMXCpu resource class. All of the properties in this class are provided by the DMXCpu ILT provider.

Chapter 10. Workbench

331

Creating the resource model using the step-by-step wizard To create our new resource model, in this example we are going to use the stepby-step resource model wizard. The step-by-step wizard will walk through each of the elements that make up a resource model, but it will not create a VisitTree function. We choose the step-by-step wizard in this example merely to show alternatives for developing resource models. Tip: In our development efforts while working on this redbook, we found that using the resource model wizard was a far more effective option for creating resource models than the step-by-step or the empty wizard. The resource model wizard generates a good template for all of the elements of a resource model (including a VisitTree function). First we select File -> New from the Workbench menu bar to start creating our new resource model. All UNIX/Linux resource models need to be JavaScript based. After selecting the JavaScript as the language, the first window displayed is the Wizard type selection window, which can be seen in Figure 10-62.

Figure 10-62 Resource model creation wizards

After we select the step-by-step resource model, the General Settings window will be displayed. This is where we will define the general properties of our new resource model, as can be seen in Figure 10-63 on page 333. A detailed description of all the fields in the General Settings window can be seen in Figure 10-7 on page 257. In this example we are only going to support the Solaris and AIX platforms, as noted by the check boxes in Figure 10-63 on page 333.

332

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-63 ITSO_LoadAvg General Settings window

Chapter 10. Workbench

333

Dynamic model element for ITSO_LoadAvg resource model The next window displayed by the step-by-step wizard is the Dynamic Model window. This window is where we select one or more resource classes that will define the dynamic model elements. The step-by-step wizard will allow multiple classes to be selected, unlike the resource model wizard. However, when using the resource model wizard, it is very easy to add more classes after the wizard runs. Figure 10-64 is an example of the Dynamic Model window. In this window we need to select the Add button to launch the CIM browser. Attention: Make sure the name space is root/DEFAULT instead of root/CIMV2 for UNIX/Linux resource models.

Figure 10-64 Adding dynamic model elements for ITSO_LoadAvg

Once we are in the CIM browser, we will need to compile the DMXCpu.mof into the root/DEFAULT name space before we can select it for our resource model. In this example we are not going to create a new MOF file. We are just going to use the shipped DMXCpu resource class. This class will not be loaded by default in the CIM database of the machine that we are using the Workbench from (that is, our Windows 2000 machine). Figure 10-65 on page 335 and Figure 10-66 on page 336 are examples of how to compile a MOF from the CIM browser. For an example of how to select the MOF compiler from the CIM browser, see Figure 10-43 on page 306.

334

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-65 Compiling a MOF from the Workbench CIM browser

Warning: When a MOF file is extracted from a UNIX/Linux resource model from the Workbench, it will contain UNIX CR characters, not DOS CR/LFs. In order to compile this MOF into the Windows WMI, the UNIX CRs need to be converted to DOS CR/LFs. The WMI-based MOF compiler will not understand the UNIX CR and will complain about the MOF file syntax. In Wordpad, if we save it as a text file (that is, DMXCpu.mof) without the .txt extension, it will convert the CRs to CR/LFs. After selecting Next, you will see Figure 10-66 on page 336.

Chapter 10. Workbench

335

Figure 10-66 Compiling a MOF from the Workbench

After the MOF file is compiled into the root/DEFAULT name space, it can be selected as a dynamic model element, as seen in Figure 10-67 on page 337. In this example we selected the LoadAvg1, LoadAvg5, LoadAvg15, and spare properties. Spare is the instance of the CPU on a multi-processor box (for example, 0–3 on a four way).

336

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-67 Selecting the DMXCpu Resource Class

Chapter 10. Workbench

337

Event element for ITSO_LoadAvg resource model The next window in the step-by-step wizard will prompt for event element definitions. In this example we are only going to create one event element (that is, indication). This can be seen in Figure 10-68 on page 339. We selected LoadAvg5 and LoadAvg15 as properties to be included in the indication as well.

338

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-68 Creating event elements for ITSO_LoadAvg

Chapter 10. Workbench

339

Threshold element for ITSO_LoadAvg1 The next window displayed by the step-by-step wizard is the threshold element window. The property that is going to trigger the ITSO_LoadAvg1_too_high indication is LoadAvg1. In this window we need to add a threshold element for the LoadAvg1 property. An example of this can be seen in Figure 10-69.

Figure 10-69 Creating threshold elements for ITSO_LoadAvg

The next window in the step-by-step wizard will prompt for parameter elements. In this example we are not going to use parameter elements.

Creating logging elements for ITSO_LoadAvg The next window displayed by the step-by-step wizard is the logging element window. In this window we will define all of the elements that we would like to log and possibly see in the Health Console. Figure 10-70 on page 341 is an example of adding logging elements in the Workbench.

340

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-70 Creating logging elements for ITSO_LoadAvg

Creating dependencies for ITSO_LoadAvg The next window displayed by the step-by-step wizard is the dependency element window. In this window we will define all of the generic dependencies for the ITSO_LoadAvg resource model. Since this model is based on the DMXCpu resource model, we will want to extract all of the dependent files from the DMXCpu model. Figure 10-71 on page 342 shows a list of dependency elements in the DMXCpu resource model that we will need to extract.

Chapter 10. Workbench

341

Figure 10-71 Extracting dependency elements

The following is a description of the dependency elements required for the ITSO_LoadAvg resource model: 򐂰 DMXCpu.tar This tar file contains the Java Instrumentation Library Type (ILT) code common for all of the UNIX platforms. It usually consists of the ILT class and the JNI class. 򐂰 DMXCpu.mof This is the resource class definition that will be compiled into the UNIX CIM database.

342

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 libDMXCpu.a This is the AIX shared library used by the provider to gather the CPU information. 򐂰 libDMXCpu.so This is the Solaris shared object used by the provider to gather the CPU information. Tip: When extracting MOF, tar, and shared library files, we will need to be careful with the names of the files. For example, if we extract the Java ILT tar file into the same directory where we build the resource model tar file, they will both by default have the same name. This is also true for the two MOF files described earlier, as well as some of the shared objects files. In the DMXCpu resource model, Solaris and Linux have the same shared object file names. We recommend creating a directory structure under our resource model development directory to avoid this problem. An example can be seen in Figure 10-72.

Figure 10-72 LoadAvg development directory structure

Chapter 10. Workbench

343

The step-by-step wizard dependency element window will only allow the addition of generic dependencies (that is, ALL for all platforms). After the wizard is complete, we will have to add the platform-specific dependencies. In this example we will add the AIX and Solaris shared libraries. An example of adding generic dependencies in the step-by-step wizard can be seen in Figure 10-73 and Figure 10-74 on page 345.

Figure 10-73 Adding the DMXCpu.mof file as a dependency

344

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-74 Adding the DMXCpu.tar file as a dependency

After the step-by-step wizard completes, we need to add the AIX-shared library (libDMXCpu.a) to the aix4-r1 dependency element and the Solaris (libDMXCpu.so) to the solaris2 dependency element. Figure 10-75 on page 346 is an example of our step-by-step generated resource model after we have added the additional dependency elements.

Chapter 10. Workbench

345

Figure 10-75 ITSO_LoadAvg step-by-step wizard-generated resource model

Notice in Figure 10-75 the VisitTree function is empty. The step-by-step wizard does not generate working VisitTree code.

346

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Creating the Decision Tree for ITSO_LoadAvg Finally let us look at the decision tree code that was developed for the ITSO_LoadAvg resource model. The LoadAvg1 property in the DMXCpu resource class is what we want to key off of. If the LoadAvg1 exceeds the threshold value that are set as a default in the resource model or overridden in the IBM Tivoli Monitoring 5.1 profile, we want to generate an indication. We will also log the event as well. Example 10-9 is an example of the VisitTree code that was modified after the wizard generation, followed by an overview of some of the code. Example 10-9 ITSO_LoadAvg VisitTree code 1 function VisitTree(Svc) 2 { 3 Svc.Trace(2,"Start VisitTree10"); 4 5 var curloadAvg1; 6 var curloadAvg5; 7 var curloadAvg15; 8 var curthrValue; 9 10 var curspare; 11 12 var hPropTable; 13 var numOfInstances; 14 var idx; 15 var ParamCount; 16 var ParamIdx; 17 var Different; 18 19 hPropTable = Svc.CreateMap(); 20 21 numOfInstances = Svc.GetNumOfInst("ROOT\\DEFAULT:DMXCpu"); 22 for ( idx = 0; idx < numOfInstances; idx++) { 23 24 Svc.RemoveMapAll(hPropTable); 25 26 curloadAvg1 = Svc.GetNumProperty("ROOT\\DEFAULT:DMXCpu", idx, “loadAvg1”); 27 curloadAvg5 = Svc.GetNumProperty(“ROOT\\DEFAULT:DMXCpu”, idx, “loadAvg5”); 28 curloadAvg15 = Svc.GetNumProperty(“ROOT\\DEFAULT:DMXCpu”, idx, “loadAvg15”); 29 curspare = Svc.GetStrProperty(“ROOT\\DEFAULT:DMXCpu”, idx, “spare”); 30 31 Svc.SetMapNumElement(hPropTable,"LoadAvg1",curloadAvg1); 32 Svc.SetMapNumElement(hPropTable,"LoadAvg5",curloadAvg5); 33 Svc.SetMapNumElement(hPropTable,"LoadAvg15",curloadAvg15); 34 Svc.SetMapStrElement(hPropTable,"name",curspare); 35

Chapter 10. Workbench

347

36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 }

//debugging curthrValue = Svc.GetThreshold("thrHigh_LoadAvg1") Svc.Trace(3, "VisitTree10 curthrValue="+curthrValue); Svc.Trace(3, "VisitTree10 curLoadAvg1="+curloadAvg1); Svc.Trace(3, "VisitTree10 curLoadAvg5="+curloadAvg5); Svc.Trace(3, "VisitTree10 curLoadAvg15="+curloadAvg15); //debugging if (curloadAvg1 > Svc.GetThreshold(“thrHigh_LoadAvg1”) ) { //debugging Svc.Trace(2,"VisitTree10 thrHigh_LoadAvg1"); //debugging Svc.SendEventEx ("ITSO_LoadAvg1_too_high",hPropTable); Svc.LogInstEx("Load Average", "LoadAvg1",hPropTable); } } Svc.DestroyMap(hPropTable); Svc.Trace(2,"End VisitTree10"); return (0);

Let us examine the code in Example 10-9 on page 347. 򐂰 Line 3 This is a call to the SvcTrace method. This can be used for debugging. The first parameter is the debug level. The IBM Tivoli Monitoring 5.1 supports four levels (0–3). We chose debug level 2 for entry and exit logging in this example. 򐂰 Line 19 This line creates a new map. The resource model wizard generates code using the new IBM Tivoli Monitoring 5.1 mapping table APIs. The Svc.CreateMap generates a new map. The new SendEventEx and LogInstEx methods require a mapping table as input. All new resource models should use the SendEventEx and the LogInstEx instead of the deprecated SendEvent and LogInst methods.

348

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Line 21 This is a call to get the number of instances provided by DMXCpuIlt. DMXCpuIlt is the TouchPoint provider used by the DMXCpu resource class. The DMXCpuIlt provider only returns one instance for CPU resources. It returns the average of all processors. In this example we could have just picked up instance 0 and not created a loop in the code. We felt it was a better practice to get in the habit of always checking the number of instances returned by the provider and not to hardwire instance numbers in the code. Also note that the name space is ROOT\DEFAULT, not ROOT\CIMV2. 򐂰 Line 24 On each iteration we are going to clear the defined mapping table (that is, hPropTable). 򐂰 Line 26–29 These lines retrieve the loadAvg1, loadAvg5, loadAvg15, and spare properties. 򐂰 Line 31–34 These lines add the properties as map elements to be used later by SendEventEx and LogInstEx. 򐂰 Line 36–42 This is an example of using the trace method for debugging. Notice we set the debug level in the Svc.Trace method call to level 3. 򐂰 Line 44 This is our check for the loadAvg1 property against the predefined threshold. Remember that the threshold is defined in the SetDefaultConfiguration function. 򐂰 Line 46–48 We put some debug lines in the code to see when we actually exceed the threshold. 򐂰 LIne 50–51 These lines are the calls to the SendEventEx and LogInstEx methods.

Debugging a UNIX-based resource model The IBM Tivoli Monitoring Workbench User’s Guide Version 5.1.1, SH19-4571, in the Installation chapter states that we can use JavaScript debugging if we install the Microsoft Debugger. We could not get the IBM Tivoli Monitoring 5.1 Workbench to work properly with the Microsoft Debugger in this release. This will be addressed in a future release. We encountered two issues when trying to debug our JavaScript code. One issue was that without a debugger we could not

Chapter 10. Workbench

349

syntax check the code. The only way to check for syntax errors in the JavaScript code was to distribute the IBM Tivoli Monitoring 5.1 profile to the endpoint and check the trace_dmxengine.log log. The log message would only state that there was a syntax error and did not tell what line in the code was the offender. The second issue is that without a debugger it is hard to test our code logic. After a few times trying to debug the UNIX resource models without a debugger, it became clear that we needed a debugger. We found the Rhino debugger.

Using the Mozilla Rhino JavaScript source-level debugger The Mozilla Rhino JavaScript shell has a debugger for debugging JavaScript scripts. The debugger is a Java program. We were able to use the Rhino debugger for syntax checking our code. After we installed the Rhino shell and updated our classpath variable, we were able to invoke the debugger from a cmd.exe window, as can be seen in Figure 10-76. For more information on how to download and install the Mozilla Rhino shell, see 10.4.5, “Rhino” on page 380.

Figure 10-76 Invoking the Mozilla Rhino JavaScript Debugger

Before we can syntax check our VisitTree code, we have to export the JavaScript code from the Workbench. To export the JavaScript code we selected Build -> Export Java Script from the Workbench menu. Then from the Rhino JavaScript Debugger Java Console we select File -> Run and open up the exported JavaScript code. If there are any syntax errors the debugger will display the line of the offending statement. An example of this can be seen in Figure 10-77 on page 351.

350

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-77 Syntax checking a JavaScript with the Rhino script debugger

Note: We were able to do some limited debugging of our UNIX-based resource models using the Rhino Debugger. However, none of the IBM Tivoli Monitoring 5.1 method calls will work from the debugger. To debug the JavaScript code we had to comment out the method calls and supply dummy numbers in place of the method calls. We also had to add some stub code in the beginning of the JavaScript, as follows: var Svc; RetVal = VisitTree();

This will works fine only for limited logic testing of specific code. Unfortunately, for now most of the debugging will have to be done using the logs on the endpoint.

Chapter 10. Workbench

351

Using the logs on an endpoint to debug a resource model Most of the debugging will have to be done on a test endpoint after the IBM Tivoli Monitoring 5.1 profile has been distributed. An effective way to debug a UNIX resource model is to add Svc.Trace method calls in the code like the ones in Example 10-9 on page 347. The first step in debugging using the logs is to turn the trace level up to the highest value, debug level 3. On our test UNIX endpoint, the following is the command we used to set the trace level for debugging: wdmtrceng -e tividc12 ““ 3 1000000

On UNIX/Linux the wdmtrceng command does not use the file name. The values supplied are used for all trace logs on UNIX/Linux. See the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569, for more details on the wdmtrceng command. In Example 10-9 on page 347 we prefaced all of our messages in the Svc.Trace method calls with the literal VisitTree10. We did this so we could use the grep command to see our trace calls from the VisitTree function. We could have supplied any unique name in the trace method. The following is an example of the command we used to see all of our trace messages from the IBM Tivoli Monitoring 5.1 logs directory: grep VisitTree10 trace_dmxengine.log

Figure 10-78 on page 353 is an example of some of the log messages that were found for this resource model.

352

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-78 trace_dmxengine.log debugging example

10.3.4 ITSO_FileSystem Example In the final example in this chapter we are going to build a parametric-based resource model. We are going to use the DMXFileSystem resource class provided by IBM Tivoli Monitoring 5.1. The current version of the DMXFileSystem resource model does not generate an indication for the percentage of used space for file systems. It only includes a LowKAvail (threshold by kbytes). If we want to use the current model to monitor the percentage of used space we have to do it by percent inodes. The DMXFileSystem resource model also does not accept parameters. It only allows the setting of one threshold value for all file systems. For example, if we need to monitor a file system to see if it is using more than 10 percent space, and another file system using more than 15 percent, we would have to create two different IBM Tivoli Monitoring 5.1 profiles. Even then we would wind up getting duplicate events for all the >15 percent file systems. In this example we are going to build a resource model that accepts parameters as input and electively monitors specific file systems by percent used. It will also allow input of percent used by file system.

Chapter 10. Workbench

353

Tip: This example will demonstrate how to use the resource model wizard to create UNIX/Linux-based resource models that will accept parameters as input. This is also referred to as a parametric resource model. This example will also demonstrate how to run a Perl script as a response action to an event generated by IBM Tivoli Monitoring 5.1. The source code for this example can be found in Appendix A, “Additional material” on page 529. All of the source is contained in ITSO_FileSystem.dmws. In order to use some of the properties in the DMXFileSystem resource class we need to look at its MOF definition. An example of the DMXFileSystem MOF can be seen in Example 10-10. Example 10-10 DMXFileSystem resource class //************************************************************************** //Distributed Monitoring (Advanced Edition) V4R1 (tm) //Product ID=5698-EMN //(C)Copyright IBM Corp. 1999, 2001 All Rights Reserved. //US Government Users Restricted Rights - Use, duplication or //disclosure restricted by GSA ADP Schedule Contract with IBM Corp. //************************************************************************** [ Description (“Unix File Systems info”), provider(“com.tivoli.dmunix.ep.touchpoint.cimom.ifc.M12JavaProvider”), M12_Instrumentation {"Java.com.tivoli.dmunix.ep.ilts.DMXFileSystemIlt | | ENUM", "Java.com.tivoli.dmunix.ep.ilts.DMXFileSystemIlt | | GET"} ] class DMXFileSystem { [key]string mountPoint; sint32 totalKBytes; sint32 usedKBytes; sint32 availKBytes; sint32 totalInodeNumber; sint32 usedInodeNumber; sint32 availInodeNumber; };

354

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

For our example we want to calculate the percentage used for each file system. The DMXFileSystemIlt provider returns an instance of totalKbytes, usedKbytes, and availKbytes for each mounted file system. The calculation in our VisitTree should be very simple and look in pseudo code as follows: percentUsed = (usedkBytes/totalKBytes) * 100; if (percentUsed > ThresholdValue) { send indication }

Before we get started using the wizard we need to get all of our dependency files that we are going to use for this model. Since we are using the same resource class as the DMXFileSystem resource model we can extract all of its dependency files. An example of extracting the dependent files can be seen in Figure 10-61 on page 330. The following is a list of the dependency files that need to be extracted: 򐂰 DMXFileSystem.tar This tar file contains the Java Instrumentation Library Type code common for all of the UNIX platforms. It usually consists of the ILT class and the JNI class. 򐂰 DMXFileSystem.mof This is the resource class definition that will be compiled into the UNIX CIM database. 򐂰 libDMXFileSystem.a This is the AIX shared library used by the provider to gather the FileSystem information. 򐂰 libDMXFileSystem.so This is the Solaris shared object used by the provider to gather the FileSystem information. 򐂰 libDMXFileSystem.so This is the Linux-s390 shared object used by the provider to gather the FileSystem information. Notice this is the same file name as the Solaris-shared object library. We need to create a directory structure for storing the different objects by platform. Before we start generating the resource model we need to compile the DMXFileSystem.mof file. We need to compile the MOF file on the development system of where we will be using the Workbench. We need to do this so that when the wizard prompts us to create a dynamic model element we will be able

Chapter 10. Workbench

355

to select the DMXFileSystem resource class. In the previous examples we have shown two ways to compile a MOF file. Our personal favorite is mofcomp.exe. Figure 10-79 is an example of how we compiled the DMXFileSystem.mof in this example.

Figure 10-79 Compiling the DMXFileSystem.mof

Note: In a previous example we compiled a MOF that used a WMI-based provider. In that example we didn’t have to specify a name space because it defaulted to root/CIMV2. For this example we have to supply the “-N:root/default” parameter to correctly compile the MOF into the default name space and not the CIMV2 name space.

Creating the ITSO_FileSystem resource model As we stated earlier, we are going to generate this resource model by using the resource model wizard. By now we have provided numerous examples of how to invoke the wizards so we are just going to focus on the windows that are important to this specific resource model. We begin by selecting File -> New -> Java Script Resource Model from the Workbench menu. Then we select Resource model wizard from the wizard types window. In the ITSO_FileSystem example we are going to support the AIX, Solaris, and Linux-s390 platforms. This resource model will also be a CIM-based resource model, as can be seen in Figure 10-80 on page 357.

356

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-80 ITSO_FileSystem selecting data source types

The next window presented by the wizard is a list of resource classes. We should see the DMXFileSystem class in the list if our MOF compile was successful. We can select the DMXFileSystem resource class and select the Next button, as seen in Figure 10-81 on page 358.

Chapter 10. Workbench

357

Figure 10-81 ITSO_FileSystem selecting the resource class

In the ITSO_FileSystem example we are concerned about the percentUsed of each file system. We need to select the usedKBytes and the totalKBytes properties to be able to calculate the percentUsed value. We are also going to select the mountPoint property to be able to tell what file system we are looking at. Finally we added the availKBytes property for an additional field for historical logging. Figure 10-82 on page 359 is an example of the selected properties in the wizard.

358

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-82 ITSO_FileSystem Selected Properties window

Note: We are not going to use any WQL filters in this example, so we skipped the Filter window in the wizard. The next screen in the resource model wizard is where we will create the event and threshold elements. In this example the event element that we want to create is based on a derived value that does not exist yet (that is, percentUsed). At this point in the wizard we do not have an option to add a new field to set a trigger condition (that is, the threshold). What we are going to do is trick the wizard. We are going to select the usedKBytes property and set the threshold to 35 percent, as if that was really the percentUsed. This way the wizard will generate all the appropriate elements and the VisitTree code. It is easier to have the wizard generate all the relationships between the event and threshold

Chapter 10. Workbench

359

elements than it is to add them manually after the wizard completes. After the wizard completes we can then come back and change the generated event and threshold elements. Figure 10-83 is an example of the selected usedKBytes property in the Event Triggering Conditions window presented by the wizard.

Figure 10-83 ITSO_FileSystem specifying event triggering conditions

The wizard will next prompt for logging elements. In this example we will select all of the elements in the list. Here again we will have to go back and add our new percentUsed variable after the wizard completes. The final window in the resource model wizard will prompt for the cycle time. We entered the default 120 for 120 seconds. Figure 10-84 on page 361 is an example of the generated resource model after the wizard completes.

360

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-84 ITSO_FileSystem wizard-generated resource model

Modifications after the wizard completes After the resource model wizard completes we still have to modify some of the wizard generated elements. The first modification is to rename the generated resource model name and change some of the default general settings, as seen in Figure 10-85 on page 362.

Chapter 10. Workbench

361

Figure 10-85 ITSO_FileSystem modifying the general settings

Modifying the ITSO_FileSystem event element Next we need to change some of the properties in the event element generated by the wizard. An example of the changes can be seen in Figure 10-86 on page 363. The arrows show the properties that were changed. The usedKBytes property was changed to percentUsed. This is the new variable we are going to define in the VisitTree code.

362

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-86 ITSO_FileSystem modifying the event element

Chapter 10. Workbench

363

In this example we also want to run a Perl script as a response action to the percentUsed indication. What we would like to do is run our Perl script every time the IBM Tivoli Monitoring 5.1 analyzer generates an event (that is, it meets the holes and occurrences of indications). The Perl script in Example 10-11 is just a simple program that demonstrates how to invoke a Perl program from a response action and lists all the environment variables available from a response action script. Example 10-11 Example Perl script #!/usr/opt/perl5/bin/perl $date = `date`; $log = "/tmp/dumpvars.out"; open(LOG,">$log"); $msg = "Variable Dump on $date "; print LOG "$msg\n"; $env = `env`; print LOG "$env\n"; close(LOG); exit 0;

In the event element window we select the Actions button and then select Add Program. Figure 10-87 on page 365 is an example of our added Perl script to the ITSO_FileSystem_pctusedKBytes_High indication. See Figure 10-88 on page 366 for more details on the variable output.

364

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-87 ITSO_FileSystem response actions

Variables from a response action script When a response action script runs, all of the event element properties are created as variables in the script. The names of the variables are generated as compound structures using the name of the event concatenated with the name of the property. Figure 10-88 on page 366 is an example of some of the variables created by the ITSO_FileSystem resource model.

Chapter 10. Workbench

365

Figure 10-88 ITSO_FileSystem variables

Modifying the ITSO_FileSystem threshold element Next we need to change the threshold element generated by the wizard. An example of the changes can be seen in Figure 10-89 on page 367. The arrows show the properties that were changed. The usedKBytes property was changed to percentUsed. This is the new variable we are going to define in the VisitTree code.

366

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-89 ITSO_FileSystem modifying the threshold element

Modifying the ITSO_FileSystem logging element Next we need to change the logging element generated by the wizard. An example of the changes can be seen in Figure 10-90 on page 368. In the logging element we are going to log all of the available properties that we have selected in the dynamic model element, (that is, mountPoint, userKBytes, availKBytes, totalKBytes), along with the new percentUsed variable we are going to create in the VisitTree function.

Chapter 10. Workbench

367

Figure 10-90 ITSO_FileSystem modifying the logging element

Creating an ITSO_FileSystem parameter element In the ITSO_FileSystem example we are going to allow the users to enter specific file systems with their associated threshold values. To create a resource model that accepts parameters we need to add a parameter element. We kept this example simple by only adding one parameter element called file system. We will make this a string list parameter type and add each of our file systems and the associated threshold values in the list. Figure 10-91 on page 369 is an example of the parameter element that was used in this example. For more information on different types of parameters that can be added see the IBM Tivoli Monitoring Workbench User’s Guide Version 5.1.1, SH19-4571.

368

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-91 ITSO_FileSystem adding parameter elements

Later, in the VisitTree code description for this example, we will see that there are methods defined to retrieve the count of values, Svc.GetStrParameterCount, and retrieve each value, Svc.GetStrParameters, in the parameter list. The parameter list is defined in the Workbench as a parameter element and can be overridden in the IBM Tivoli Monitoring 5.1 profile, as seen in Figure 10-92 on page 370.

Chapter 10. Workbench

369

Figure 10-92 IBM Tivoli Monitoring 5.1 Profile parameters

Adding ITSO_FileSystem dependency elements The files that were extracted earlier in the list on page 355 need to be added as dependency elements to the ITSO_FileSystem resource model. Figure 10-93 on page 371 is an example of all the completed elements for the ITSO_FileSystem resource model.

370

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 10-93 ITSO_FileSystem resource model elements

Creating the Decision Tree for ITSO_FileSystem At a high level the ITSO_FileSystem VisitTree code will retrieve a list of all the values from the parameter element and compare them against each file system instance returned from the DMXFileSystem resource class. For each matched file system the VisitTree code will compare the associated threshold with the file system to the new percentUsed value. If a value is not supplied with a file system

Chapter 10. Workbench

371

in the parameter list it will default to the threshold element value. When a matched file system exceeds the percentUsed threshold value an indication will be generated and all the associated properties will be logged. Let us take a look at the VisitTree code for the ITSO_FileSystem resource model in Example 10-12. Example 10-12 ITSO_FileSystem VisitTree code 1function VisitTree(Svc) 2{ 3 Svc.Trace(2,"Start VisitTree20"); 4 5 var percused = 0; 6 var bInvalidSpace = false; 7 var curusedKBytes = 0; 8 var curtotalKBytes = 0; 9 var curavailKBytes = 0; 10 var curmountPoint="" ; 11 12 var SendEventMap; 13 var LogInstMap; 14 var numOfInstances; 15 var numOfParms; 16 var idx; 17 var ParmIdx; 18 var parmtmp 19 var strmountPoint;; 20 21 var parmtmpa = new Array(); 22 var ParmMount = new Array(); 23 var ParmSize = new Array(); 24 25 SendEventMap = Svc.CreateMap();//Create event table map 26 LogInstMap = Svc.CreateMap();//Create log table map 27 28 numOfParms = Svc.GetStrParameterCount("filesystem");//Get the number of parms 29 30 for ( ParmIdx = 0; ParmIdx < numOfParms; ParmIdx++) { 31 32 parmtmp = Svc.GetStrParameter("filesystem", ParmIdx);//Get each parm bot (mnt and thr) 33 parmtmpa = parmtmp.split(/\s+/);//Regex put values in array mntp in [0] and size in [1] 34 ParmMount[ParmIdx] = parmtmpa[0] + ",";//Add "," for uniqueness 35 36 //if size is not null use default threshold parm 37 if (parmtmpa[1] == "" || parmtmpa[1] == null) { 38 ParmSize[ParmIdx] = Svc.GetThreshold(“Thr_percentUsed_gt”); 39 } else { 40 ParmSize[ParmIdx] = parmtmpa[1];

372

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

41 } 42 } 43 //Get count of mountpoints 44 numOfInstances = Svc.GetNumOfInst("ROOT\\DEFAULT:DMXFileSystem"); 45 46 //Process each mountpoint 47 for ( idx = 0; idx < numOfInstances; idx++) { 48 49 strmountPoint = Svc.GetStrProperty("ROOT\\DEFAULT:DMXFileSystem", idx, "mountPoint"); 50 //Match the input mountpoint for uniqueuness 51 curmountPoint = strmountPoint + ","; 52 //Loop through each parm 53 for ( ParmIdx = 0; ParmIdx < numOfParms; ParmIdx++) { 54 //Only process parms specified in profile 55 if (ParmMount[ParmIdx].indexOf(curmountPoint) >= 0 ) { 56 57 Svc.RemoveMapAll(SendEventMap);//Clear both maps 58 Svc.RemoveMapAll(LogInstMap); 59 //Get properties 60 curusedKBytes = Svc.GetNumProperty("ROOT\\DEFAULT:DMXFileSystem", idx, "usedKBytes"); 61 curavailKBytes = Svc.GetNumProperty("ROOT\\DEFAULT:DMXFileSystem", idx, "availKBytes"); 62 curtotalKBytes = Svc.GetNumProperty("ROOT\\DEFAULT:DMXFileSystem", idx, "totalKBytes"); 63 bInvalidSpace = (curusedKBytes < 0) || (curtotalKBytes ParmSize[ParmIdx] ) { 83 84 //Add to event map 85 Svc.SetMapNumElement(SendEventMap,"percentUsed",percused); 86 Svc.SetMapNumElement(SendEventMap,"UpperBound",ParmSize[ParmIdx]); 87 Svc.SetMapStrElement(SendEventMap,"mountPoint",strmountPoint);

Chapter 10. Workbench

373

88 89 //Send indication 90 Svc.SendEventEx ("ITSO_FileSystem_pctusedKBytes_High",SendEventMap); 91 92 //Add to log map 93 Svc.SetMapNumElement(LogInstMap,"percentUsed",percused); 94 Svc.SetMapNumElement(LogInstMap,"UpperBound",ParmSize[ParmIdx]); 95 Svc.SetMapNumElement(LogInstMap,"usedKBytes",curusedKBytes); 96 Svc.SetMapNumElement(LogInstMap,"availKBytes",curavailKBytes); 97 Svc.SetMapNumElement(LogInstMap,"totalKBytes",curtotalKBytes); 98 Svc.SetMapStrElement(LogInstMap,"mountPoint",strmountPoint); 99 //Log event 100 Svc.LogInstEx ("ITSO_FileSystem_Availability","ITSO_FileSystem", LogInstMap); 101 102 } //End Threshold 103 } //End Mountpoint match 104 } //End ParmIdx loop 105 } //End Mountpoint Instances 106 107 Svc.DestroyMap(SendEventMap);//Clean up 108 Svc.DestroyMap(LogInstMap); 109 110 Svc.Trace(2,"End VisitTree20");//Bye Bye 111 112 return (0); 113 114} //End VisitTree

Let us examine the code in Example 10-12 on page 372. 򐂰 Line 28 This is a call to the SvcGetStrParameterCount method. This method will return the count of values in the array that will be used in a loop to interpret them. 򐂰 Line 30–42 This is a loop that will load in all of the parameter values. This is done for performance reasons. The output of this loop will create two arrays, ParmMount and ParmSize. Line 32 is a call to the Svc.GetStrParamter method. This method passes the instance number as an argument and returns the value in the parameter list. Line 33 is an example of using REGEX in JavaScript. On line 38 if the size is not specified in the parm list for a specific file system we will use the one defined on the threshold element. 򐂰 Line 44 This line will get the number of mounted file systems. This is the data that will be gathered by the DMXFileSystemIlt provider.

374

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 Line 47 This is the start of the loop to process all mounted file systems passed by the DMXFileSystemIlt provider. 򐂰 Line 49 This a call to retrieve the name of the mountpoint for each file system instance. 򐂰 Line 51 We are padding the comma as a delimiter for the file system. This will make each file system name unique (for example, /usr will not be found for both /usr and /usr/local). 򐂰 Line 53–44 We will loop through all of the values specified in the parameter list and only process the file systems that match. The indexOf method is a JavaScript method and can be very useful for doing string comparisons. 򐂰 Line 60–62 These are the calls to get the usedKBykes, availKBytes, and totalKBytes properties. 򐂰 Line 67 This is our primary calculation. This is the new data value we are creating in this routine that will be used for threshold comparisons (percentUsed). 򐂰 Line 74–79 These lines are used for debugging only. In the log we can watch each file system and see all the values before the comparison. If we do not have a debugger we have to create one using the log. 򐂰 Line 82 This is our threshold comparison. Remember, we may be using the value passed in the parameter list and not the one specified as the threshold element for this comparison. 򐂰 Line 84–90 If the file system exceeds the threshold then we will generate an indication (line 90). 򐂰 Line 92–100 We will also log all of the properties used in this function.

Chapter 10. Workbench

375

10.4 Tools and extra information This section will provide useful information about various tools and techniques that will help you be successful with resource models.

10.4.1 WorkBench Command Line Interface The IBM Tivoli Monitoring 5.1 Workbench provides a very simple CLI to use some of its functions from command prompts. This basically helps in automating resource model package builds from the source workspaces, and to modify some of its components without running the GUI-based tool. Before we start using the Workbench CLI we might want to add the Workbench bin directory into the path. The command usage is: wdmwbcli wdmwbcli wdmwbcli wdmwbcli wdmwbcli wdmwbcli wdmwbcli

resmodworkspace resmodworkspace resmodworkspace resmodworkspace resmodworkspace resmodworkspace resmodworkspace

-bldpkg outputPackage -expmsgcat messagecatalog -expjmsgcat javamessagecatalog -bldtecbaroc tecbaroc -setdep interp depfile [outputworkspace] -setdectree dectreefile [outputworkspace] -setver major minor [outputworkspace]

Where: resmodworkspace

Is the *.dmws of dmjsws project file

outputPackage

Is the output tar file

messagecatalog

Is the output message catalog

javamessagecatalog Is the output Java message catalog tecbacor

Is the output TEC BAROC file

interp

Is the dependency interpreter type

depfile

Is the dependency file name associated with the interpreter type

outworkspace

Is the name of the output *.dmws or *.dmjsws project file

dectreefile

Is the decision tree file

major

Is the resource model major version number

minor

Is the resource model minor version number

The command usage is: wdmwbcli resmodworkspace -bldpkg outputPackage

376

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Builds the resource model tar from the source workspace. wdmwbcli resmodworkspace -expmsgcat messagecatalog

Exports the resource model message catalog. wdmwbcli resmodworkspace -expjmsgcat javamessagecatalog

Exports the resource model Java message catalog to use with Web Health Console. wdmwbcli resmodworkspace -bldtecbaroc tecbaroc

Exports the TEC baroc file with the definitions of all the events generated by the source resource model. wdmwbcli resmodworkspace -setdep interp depfile [outputworkspace]

Adds the specified depfile as a resource model dynamic dependency for the given interp (use all to add the file for all the available interps). wdmwbcli resmodworkspace -setdectree dectreefile [outputworkspace]

Sets the JavaScript or Visual Basic Code to the specified resource model. JavaScript files can be set only to .dmwjsws workspaces, while Visual Basic files can be set only to .dmws. wdmwbcli resmodworkspace -setver major minor [outputworkspace]

Changes the version of the specified resource model. Examples: wdmwbcli ITSO_FileSystem.dmjsws -bldpkg ITSO_FileSystem.tar wdmwbcli ITSO_FileSystem.dmjsws -expmsgcat ITSO_FileSystem.msg

10.4.2 Resource model internal format After a resource model is built from the Workbench it gets created in an output tar file. Each tar file consists of three types of files: conf

The conf file is the default configuration for the resource model. It is used by IBM Tivoli Monitoring 5.1 to determine how to install the resource model.

cat

The cat file is the message catalog for building TEC and TBSM events.

zip

The one or more *.zip file(s) are the platform dependent files needed by the resource model. On a Windows machine if we extract the file and rename the extension from .zip to .tar.gz we can use WinZip to browse the file.

Chapter 10. Workbench

377

Figure 10-94 is an example of using WinZip to see what is inside the ITSO_FileSystem.tar file.

Figure 10-94 Browsing the ITSO_FileSystem.tar file

If we want to see what is inside one of the .zip files for a specific platform we can extract the file and rename it. For example, if we extract __aix4-r1__ITSO_FileSystem.zip to __aix4-r1__ITSO_FileSystem.tar.gz we can open up the file using WinZip, as seen in Figure 10-95.

Figure 10-95 Browsing the ITSO_FileSystem zip file

378

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Note: On UNIX systems the .zip extension can be unzipped by using the following command: gzip -d -S .zip __aix4-r1__ITSO_FileSystem.zip

10.4.3 Microsoft tools The Microsoft tools are discussed in the following sections.

WMI SDK See: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmisdk/wmi/wmi _start_page.asp

WMI providers See: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmisdk/wmi/man aged_objects_and_providers.asp

Windows NT and 2000 Resource Kit Two tools that are very effective for creating artificial load balances on a Windows 2000 endpoint are cpustres.exe and probe.exe. In the PhysicalDisk and LogicalDisk examples we used the diskmax.bat command. This is a command that has been customized to use the probe.exe tool. Appendix A, “Additional material” on page 529, has instructions on how to download files from the Web. There is a diskmax.zip file at that site and it has all of the diskmax files that were used in the examples.

Microsoft script debugger See: http://msdn.microsoft.com/downloads/default.asp?url=/downloads/topic.asp?URL=/M SDN-FILES/028/001/175/topic.xml

How to use the Microsoft script debugger: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdbug/Html/sdb ug_17.asp

Chapter 10. Workbench

379

10.4.4 Saxsoft The following is the Saxsoft Web site: http://www.saxsoft.com

10.4.5 Rhino The following are Rhino Web sites: http://www.mozilla.org/rhino/ http://www.mozilla.org/rhino/download.html

The Rhino shell allows us to run scripts from files or interactively at a command line.

Rhino shell Download the zip file for rhino. The zip file will contain a single JAR file, js.jar. We then need to add the JAR file to the class path. We can start the Rhino shell using the command We can execute a JavaScript by passing the file as an argument: java org.mozilla.javascript.tools.shell.Main test.js

There are many options for running scripts. See the associated documentation for more details.

Rhino Debugger The Mozilla Rhino JavaScript shell includes a debugger. The debugger runs as a Java program: java org.mozilla.javascript.tools.debugger.Main

The options are the same as the shell. For more information see: http://www.mozilla.org/rhino/debugger.html

380

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Part 4

Part

4

Integration In this part we describe how to integrate IBM Tivoli Monitoring Version 5.1 with: 򐂰 Tivoli Enterprise Console 򐂰 Tivoli Business Systems Manager 򐂰 Data Warehouse

© Copyright IBM Corp. 2002

381

382

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

11

Chapter 11.

TEC integration This chapter describes how to integrate IBM Tivoli Monitoring Version 5.1 with Tivoli Enterprise Console. The integration allows for events generated by IBM Tivoli Monitoring Version 5.1 to be forwarded to Tivoli Enterprise Console. The basic integration involves the following main tasks: 򐂰 Enabling IBM Tivoli Monitoring Version 5.1 to forward events to Tivoli Enterprise Console – Identify the Tivoli Enterprise Console server in IBM Tivoli Monitoring Version 5.1 profiles and enable the forwarding of events to Tivoli Enterprise Console. – Enable the heartbeat function to send events to Tivoli Enterprise Console. 򐂰 Configuring Tivoli Enterprise Console to receive and display events – Import the IBM Tivoli Monitoring Version 5.1 event classes into the Tivoli Enterprise Console server. – Import the IBM Tivoli Monitoring Version 5.1 rule sets.

© Copyright IBM Corp. 2002

383

11.1 Enabling IBM Tivoli Monitoring Version 5.1 to forward events to TEC To add the capability of forwarding events to Tivoli Enterprise Console when creating a profile, you must perform the procedure in the next section.

11.1.1 Configuring a profile to send events to TEC Following is the procedure for configuring an IBM Tivoli Monitoring Version 5.1 profile to send events to TEC. 1. Open an existing Tmw2kProfile profile, or create and configure a new one. An example can be seen in Figure 11-1 on page 385.

384

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 11-1 Profile with a resource model added

2. Select Edit -> Properties from the menu to display the Properties dialog box, as shown in Figure 11-2 on page 386.

Chapter 11. TEC integration

385

Figure 11-2 Profile Properties window

3. Check the Send TEC Events box to enable sending events to the Tivoli Enterprise Console server. The Using section of the dialog is enabled. You must choose the delivery mode for sending the events. The choices are: – Non-TME (Unsecure) Delivery You must specify the event server port and server location where the event will be sent. Specify the event server port if the TEC server runs on Windows (default is 5529); otherwise, you must set this value to 0. Enter the server location as a fully qualified Tivoli Enterprise Console server host name (DNS name) or IP address. – TME (Secure) Delivery Select the Tivoli Enterprise Console server from the Choose TEC Server list. The Tivoli Enterprise Console Adapter Configuration Facility (ACF) must be installed on the gateway if you choose the TME (secure) delivery mode. 4. Select OK to save and exit the profile properties dialog. You may now continue with adding the desired resource models. IBM Tivoli Monitoring Version 5.1 also allows you to specify whether an event is sent to Tivoli Enterprise Console for each specific indication.

386

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

5. Referring back to Figure 11-1 on page 385, select the resource model you would like to modify and click Edit. The Edit Resource Model panel will appear. 6. Select the Indications... button to edit the properties for TEC event enabling. As an example, Figure 11-3 shows the Indications and Actions dialog for the Processor Busy indication of the processor resource model. To enable sending this event to Tivoli Enterprise Console, check Send TEC Events and click the Apply button. Note that the Send TEC field changes to YES. You can also change the severity value of the event, which is similarly reflected in the Severity field. For an explanation of the number of holes and occurrences, see 8.3, “Defining health” on page 182.

Figure 11-3 Indications and Actions panel

7. Close this panel by clicking Apply Changes and Close. Once this profile is distributed to an endpoint, IBM Tivoli Monitoring Version 5.1 will forward the events to Tivoli Enterprise Console.

Chapter 11. TEC integration

387

Also as a result of each profile distribution, a file with a .conf extension is created on the endpoint: 򐂰 For UNIX: $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Tec/Conf/EventServer.conf 򐂰 For Windows: %LCF_DATDIR%/LCFNEW/Tmw2k/tec/Conf/EventServer.conf Depending on the options you choose when setting the profile properties, the file may contain the text shown in Example 11-1. Example 11-1 TEC adapter configuration file # # # # # # #

Distributed Monitoring for Unix Tec Adapter Configuration File This file contains the settings for the Tec Adapter itso_demo#geesun-region

ServerLocation=@EventServer ServerPort=0 BufferEvents=YES BufEvtPath=/opt/Tivoli/lcf/dat/1/LCFNEW/Tmw2k/Unix/./Tec/cache/EventServer_0.da t NO_UTF8_CONVERSION=YES

As an example, the profile we have configured has been distributed, and is now sending events and clearing events to our TEC server. The TEC Console shown in Figure 11-4 on page 389 shows events received from the out process resource model. The event received was of type ProcessKilledOrNotExisintg, and was closed by the clearing event TMWClearing, by the supplied rules we installed in our rulebase. The output of the reception log is shown in Example 11-2. Example 11-2 Reception log contents 1~106~65537~1019824741(Apr 26 13:39:01 2002) ### EVENT ### ProcessKilledOrNotExisting;modelname="DMXProcess";profilename="unix#geesun-regi on";name="tivoli";eventid="1019824736350";severity="CRITICAL";event_key="Proces sKilledOrNotExisting|name=tivoli;";hostname="geesun";origin="192.168.1.40";adap ter_host="geesun";date="4/26/2002 1:38:56 PM";msg="The process tivoli has been killed or does not exist.";END ### END EVENT ### PROCESSED 1~107~65537~1019824791(Apr 26 13:39:51 2002) ### EVENT ###

388

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

TMW_Clearing;modelname="DMXProcess";profilename="itso_demo#geesun-region";event name="ProcessKilledOrNotExisting";eventid="1019824725028";event_key="ProcessKil ledOrNotExisting|name=tivoli;";slotkey="ProcessKilledOrNotExisting|name=tivoli; ";severity="HARMLESS";hostname="geesun";origin="192.168.1.40";adapter_host="gee sun";date="4/26/2002 1:39:46 PM";msg="The process tivoli has been killed or does not exist.";END

Figure 11-4 TEC Console with events received from a resource model

11.1.2 Configuring the heartbeat function to send events to TEC The heartbeat function is an integral part of IBM Tivoli Monitoring Version 5.1. It regularly monitors the endpoints, checking whether they are running correctly. For more information on installing and configuring the heartbeat, refer to Chapter 7, “Heartbeat” on page 167. To enable forwarding of heartbeat events to Tivoli Enterprise Console, run the following command on the TMR server: wdmconfig -m managed_node -D heartbeat.send_events_to_tec=true -D heartbeat.tec_server=EventServer

Where -m managed_node specifies the ManagedNode/gateways on which the product configuration will be updated.

Chapter 11. TEC integration

389

The above parameters are updated in the $DBDIR/dmml/adapter.config file (%DBDIR%\dmml for Windows systems). We configured the heartbeat using the commands shown in Example 11-3. Example 11-3 Configuring the heartbeat root:/usr/local/Tivoli/bin/solaris2/TMNT_TEC ==>wdmconfig -m geesun -D heartbeat.send_events_to_tec=TRUE -D eartbeat.tec_server=EventServer root:/usr/local/Tivoli/bin/solaris2/TMNT_TEC ==>wdmheartbeat -m geesun -s 20 Processing ManagedNode geesun..

The endpoint was stopped and restarted, and the IBM Tivoli Monitoring Version 5.1 engine was also restarted, as shown in Example 11-4. Example 11-4 Restarting the endpoint and engine ==>wdmcmd -stop -e geesun Stopping engine on endpoint 2000580590.5.517+#TMF_Endpoint::Endpoint# Stopped engine on endpoint 2000580590.5.517+#TMF_Endpoint::Endpoint# ==>wdmcmd -restart -e geesun Starting engine on endpoint 200058090.5.517+#TMF_Endpoint::Endpoint# Started engine on endpoint 2000580590.5.517+#TMF_Endpoint::Endpoint#

When the engine is stopped a HeartBeat_DMAgentDown event is sent. When the engine is started a HeartBeat_DMAgentAlive event is sent, which will close the HeartBeat_DMAgentDown event. This is shown in Figure 11-5 on page 391.

390

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 11-5 Heartbeat events for the engine

When the endpoint is stopped a HeartBeat_EndpointUnreachable event is sent. This event is closed when the HeartBeat_DMAgentAlive is received. This is shown in Figure 11-6 on page 392.

Chapter 11. TEC integration

391

Figure 11-6 Heartbeat events for the endpoint

11.2 Configure TEC to receive events To forward IBM Tivoli Monitoring Version 5.1 events to a Tivoli Enterprise Console server, first verify that Tivoli Enterprise Console Version 3.7 plus Patch 004 or later have been installed. In order to use the TME Secure Delivery, you must also install the Adapter Configuration Facility (ACF) on both the Tivoli server and Tivoli management gateways used to distribute profiles to the endpoints. You then need to import the IBM Tivoli Monitoring Version 5.1 BAROC and rule files into the rule base you are planning to use. To accomplish this, you can either run the script provided by Tivoli or import the classes manually. The provided script will import all classes and rule sets that are provided by IBM Tivoli Monitoring Version 5.1, whereas with the manual procedure, you can to only choose import the classes you plan to use.

392

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

11.2.1 Using the Tivoli-provided script The IBM Tivoli Monitoring Version 5.1 script to update the rule base is called dmae_tec_inst.sh, and can be found in the $BINDIR/TMNT_TEC directory on the node where you installed IBM Tivoli Monitoring Version 5.1. As an argument, it takes the name of the rule base you wish to update. You can also specify the -restartsvr option to restart the Tivoli Enterprise Console server at the end of the script: dmae_tec_inst.sh rbname [-restartsvr]

Where rbname is the name of the rule base you want to use. Note: If IBM Tivoli Monitoring Version 5.1 is not installed on your TEC server, then you would have to copy the entire $BINDIR/TMNT_TEC directory to your TEC server, where you will run the script. The script automatically loads the specified rule base once it has finished running.

11.2.2 Manual update of TEC rule base If you used the Tivoli-provided script to update the rule base, then you can skip this section; otherwise, you need to perform the procedures manually. In most cases, you can use the command line or the graphical interface to perform the specific tasks. For more information, refer to the Tivoli Enterprise Console Reference Manual, GC32-0666, and the Tivoli Enterprise Console Rule Builder’s Guide, GC32-0669 (both of these come with the products). Perform the following steps: 1. Select an existing rule base or create a new rule base. You may use the command wlsrb to view all available rule bases. 2. Import the required IBM Tivoli Monitoring Version 5.1 BAROC files located in the $BINDIR /TMNT_TEC directory. To ensure that the classes are loaded correctly, you must import the files in the order below. The order in which the imported BAROC files are loaded is determined by the .load_classes file located in the $BINDIR/TME/TEC directory. a. Import the Tmw2k.baroc file, which contains the base IBM Tivoli Monitoring Version 5.1 event class and therefore must be imported before any other IBM Tivoli Monitoring Version 5.1 classes. b. Import the BAROC files for all the resource models whose events will be forwarded to Tivoli Enterprise Console server. A complete list is provided in Table 11-1 on page 395.

Chapter 11. TEC integration

393

c. Import the hb_events.baroc file to forward heartbeat events. For more information on heartbeat configuration, see 11.1.2, “Configuring the heartbeat function to send events to TEC” on page 389. 3. Import the required rules provided by IBM Tivoli Monitoring Version 5.1. The rule files can be found in the $BINDIR/TMNT_TEC directory. These rules are required for processing of clearing events. See 11.2.7, “Clearing events” on page 399, for details. Import the following rule files into the rule base: a. The IBM Tivoli Monitoring Version 5.1 clearing event rules file is dmae_events.rls. b. The heartbeat clearing event rules file is hb_events.rls. 4. Compile and load the rule base. 5. Stop and restart the Tivoli Enterprise Console server.

11.2.3 Viewing IBM Tivoli Monitoring Version 5.1 events in TEC The Tivoli Enterprise Console server is now ready to receive events from the IBM Tivoli Monitoring Version 5.1 resource models whose BAROC files have been imported into the active rule base. To see the events sent by IBM Tivoli Monitoring Version 5.1, open the Tivoli Enterprise Console main panel and click the All icon. For more details, see the Tivoli Framework 3.7.1 User's Guide, GC31-8433.

11.2.4 IBM Tivoli Monitoring Version 5.1 event aggregation and TEC When a resource model is added to a profile, a cycle time can be specified, which is the interval when resource monitoring data is gathered. For example, a resource model with a cycle time of 10 would take snap shots of system information every 10 seconds. If the profile is configured to forward events to the Tivoli Enterprise Console, then potentially, every 10 seconds, a Tivoli Enterprise Console event would be sent. However, to avoid event flooding, IBM Tivoli Monitoring Version 5.1 provides the ability to restrict the number of events. To better understand this, we need to review how events are generated. Events are used to validate the persistence of a certain indication. This persistence is defined in terms of occurrences and holes within a specific cycle time. Occurrences are how many times the problem needs to occur in order to send the indication. Holes are the maximum number of consecutive gaps between occurrences in consecutive cycles. For a detailed explanation of holes and occurrences, see 8.3, “Defining health” on page 182.

394

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

In the Indications and Actions dialog box in an IBM Tivoli Monitoring Version 5.1 profile you can use the value for the number of holes in conjunction with the number of occurrences and the cycle time to define a time window for the generation of an event. By this, you can specify a kind of filtering that would take place at the endpoint, thus limiting the number of events that are sent to Tivoli Enterprise Console. The default values provided with the IBM Tivoli Monitoring Version 5.1 resource models are determined to provide the optimum monitoring capabilities. As you gain more experience with the product, and to better serve your needs, you may decide to modify these values.

11.2.5 IBM Tivoli Monitoring Version 5.1 event classes The IBM Tivoli Monitoring Version 5.1 TEC classes, along with their associated BAROC files, are listed in Table 11-1. BAROC files correspond to their respective resource models. IBM Tivoli Monitoring Version 5.1 classes for Windows begin with the prefix TMW_. IBM Tivoli Monitoring Version 5.1 BAROC files can be found in the $BINDIR/TMNT_TEC directory. Table 11-1 IBM Tivoli Monitoring Version 5.1 event classes Base Tivoli Distributed Monitoring 4.1 event classes

Tmw2k.baroc TMW_Event TMW_ActionResult

TMW_TastResult TMW_Clearing Windows NT/2000 event classes

TMW_LogicalDisk.baroc

TMW_HighLogicalDiskXferRate TMW_HighLogicalDiskReadBytesSec TMW_HighLogicalDiskWriteBytesSec TMW_HighLogicalPercentDiskTime

TMW_LowLogicalDiskSpace TMW_LogicalPossibleFrag TMW_SlowLogicalDrive

TMW_PhysicalDiskModel.baroc

TMW_HighPhysicalDiskXferRate TMW_HighPhysicalDiskReadBytesSec TMW_HighPhysicalDiskWriteBytesSec

TMW_HighPhysicalPercentDiskTime TMW_PhysicalPossibleFrag TMW_SlowPhysicalDrive

TMW_MemoryModel.baroc

Chapter 11. TEC integration

395

TMW_MemoryLeakInPB TMW_MemoryLeakInSC TMW_MemoryLeakInSD TMW_LowAvailCausingManyProblems TMW_LowAvailCausingHardPaging TMW_LowAvailCausingSoftPagePagefileResize TMW_LowAvailCausingSoftPaging TMW_LowAvailHighWS

TMW_LowAvail TMW_LowAvailHighCache TMW_LowCopyReadHits TMW_LowDataMapHits TMW_LowMDLReadHits TMW_LowPinReadHits TMW_HighPaging TMW_PageFileResizing TMW_LowAvailWithSmallPageFile

TMW_Process.baroc

TMW_ProcessHighCPU

TMW_ProcessHandleLeak

TMW_Processor.baroc

TMW_BusyHardware TMW_CPUCantKeepUpWithHW TMW_HighProcesses

TMW_ProcessorBusy TMW_HighPercentUsageDelta TMW_HWKeepingCPUBusy

TMW_EventLog.baroc

TMW_EventID9 TMW_EventID11 TMW_EventID15 TMW_EventID2011

TMW_EventID2511 TMW_EventID3013 TMW_EventID7023

TMW_NetworkIntCard.baroc

TMW_HighBroadcastFrames TMW_RedirectorOverloadedAffectingSegment TMW_HighCurrentCommands TMW_ServerAffectingRedirector TMW_AdjustWorkItems TMW_SegmentAffectingRedirector TMW_RedirectorOverloaded

TMW_ServerOverloaded TMW_ServerOverloadedAffectingSegment TMW_RedirectorAffectingServer TMW_SegmentAffectingServer TMW_HighErroredRatio TMW_NICOverworked TMW_NICTooSlow

TMW_Services.baroc

TMW_ServicesFailingService

TMW_ServicesStoppedService

TMW_TCPIP.baroc

TMW_HighPing TMW_HighFragRatio

TMW_SegmentsReXmit

TMW_PrintModel.baroc

TMW_HighCurrentPercentTime TMW_HighOutOfPaperErrors TMW_HighJobErrors

396

TMW_HighJobErrorsPerDay TMW_HighNotReadyErrors TMW_HighNotReadyErrorsPerDay

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

TMW_CorrelatedEvents.baroc

TMW_SlowHardDrive TMW_HighDriveXferRate TMW_HighDiskReadBytesSec TMW_HighDiskWriteBytesSec TMW_HighPercentDiskTime TMW_PossibleFrag TMW_BusyDriveFromPaging

TMW_BusyDriveFromLowAvail TMW_FaultyDiskSubsystem TMW_CriticalMemoryLeaksInWS TMW_ProcessHoggingCPU TMW_CongestedTCPNetwork TMW_CriticallyLowDiskSpace UNIX/LINUX event classes

DMXCPU.baroc

Low_IdleCPUUsage

High_SysCPUUsage

DMXFile.baroc

FileChanged FilesAttributeChange

FileNotPresent

DMXFileSystem.baroc

LowKAvail FragmentedFileSystem

LowPercInodesAvail

DMXMemory.baroc

LowStorage Thrashing

LowSwap

DMXNetworkInterface.baroc

HighOutErrorPacks HighInputErrPacks InterfaceNotEnabled IntNotSupported

HighPacktsCollision InterfaceNotOperat IntStatUnknown

DMXProcess.baroc

ProcessKilledOrNotExisting ProcessHighCPU

ProcessStopped HighZombieProcesses

DMXSecurity.baroc

IllegalOwner DuplicatedAccount WrongMode HighLoggingNumber SuspectSuperGroup

NotRegularRootAccount FileNotExisting PasswdNull SuspectSuperUser IllegalGroup Solaris event classes

Chapter 11. TEC integration

397

DMXNetworkRPCNFS.baroc

HighNFSSrvRead HighNFSBufferSize HighNFSSrvGetattr NetworkSlow HighNFSSrvWrites HighPercRPCBadCalls

HighPercRetrans HighTimeoutsAnd_Badxids HighPercDupReqs NetworkBusy HighNFSSrvReadLink Heartbeat event classes

hb_event.baroc

HeartBeat_Event HeartBeat_Off HeartBeat_EndpointUnreachable HeartBeat_EndpointMigrated

HeartBeat_DMAgentDown HeartBeat_DMAgentAlive HeartBeat_ResourceModelsInError

11.2.6 TMW_Event All events sent by resource models will inherit from the event TMW_Event; an example is shown in Example 11-5. Example 11-5 Definition of TMW_Event in Tmw2k.baroc TEC_CLASS : TMW_Event ISA EVENT DEFINES { source: default= "TMNT"; sub_source: default= "N/A"; sub_origin: default= "N/A"; adapter_host: default= "N/A"; msg_catalog: default= "none"; msg_index: default= 0; repeat_count: default= 0; severity: default = WARNING; modelname : STRING; profilename : STRING; eventid : STRING; event_key : STRING; }; END

398

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

11.2.7 Clearing events The clearing event, TMW_Clearing, provides the ability to close a IBM Tivoli Monitoring Version 5.1 event that has previously been generated. Clearing events are generated by IBM Tivoli Monitoring Version 5.1 when a fault condition has been corrected and has returned to normal. When IBM Tivoli Monitoring Version 5.1 sends a clearing event to Tivoli Enterprise Console, the event is processed by rules defined in tmw_events.rls. As an example, Figure 11-7 on page 400 illustrates the event attributes of ProcessKilledOrNotExisting of the resource model TMW_Processes. The corresponding clearing event TMW_Clearing is shown in Figure 11-8 on page 401. Note that the clearing event is shown for demonstration purposes only. Normally you would not see the clearing event, since it is dropped by the rule set as soon as it has been processed.

Chapter 11. TEC integration

399

Figure 11-7 ProcessKilledOrNotExisting event

400

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 11-8 Clearing event

11.2.8 The supplied rule sets Two rule set files are provided in the $BINDIR/TMNT_TEC directory. dmae_events.rls is shown in Example 11-6. Example 11-6 dmae_events.rls rule: tmw_clearing: ( /* * When a clearing event is sent, the matching events are closed */ description: 'Drop all the old events', event: _event of_class 'TMW_Clearing' where [ origin: _origin, hostname: _hostname, eventid: _eventid ], reception_action: ( all_instances(event: _old_ev of_class within ['TMW_Event']

Chapter 11. TEC integration

401

where [ origin: equals _origin, hostname: equals _hostname, eventid: equals _eventid, status:outside ['CLOSED'] ] ), change_event_status(_old_ev, 'CLOSED'), change_event_administrator(_old_ev, 'DM(AE) Clearing Event Rule'), drop_received_event ) ).

hb_events.rls is shown in Example 11-7. Example 11-7 hb_events.rls rule: all_events_clearing: ( /* * When there is a change in the status of the heartbeat, the old status event * is closed */ description: 'Drop all the old events', event: _event of_class within ['HeartBeat_Off','HeartBeat_EndpointUnreachable', 'HeartBeat_EndpointMigrated', 'HeartBeat_DMAgentDown', 'HeartBeat_DMAgentAlive', 'HeartBeat_ResourceModelsInError' ] where [ origin: _origin, hostname:_hostname ], reception_action: ( all_instances(event: _old_ev of_class within ['HeartBeat_Off', 'HeartBeat_EndpointUnreachable', 'HeartBeat_EndpointMigrated', 'HeartBeat_DMAgentDown', 'HeartBeat_DMAgentAlive', 'HeartBeat_ResourceModelsInError' ] where [ origin: equals _origin,

402

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

hostname: equals _hostname, status:outside ['CLOSED'] ] ), change_event_status(_old_ev, 'CLOSED'), change_event_administrator(_old_ev, 'DM(AE) Heartbeat Clearing Event Rule') )

11.2.9 Event definitions - Baroc files The event definitions have changed slightly in IBM Tivoli Monitoring Version 5.1 when compared to Tivoli Distributed Monitoring Advanced Edition 4.1. The two files that have changed are hb_events.baroc and Tmw2k.baroc, both of which can be found in the directory $BINDIR/TMNT_TEC. The modified event definitions are now shown.

hb_events.baroc There is a new event called HeartBeat_DMAgentRestarted, shown in Example 11-8. Example 11-8 Definition of HeartBeat_DMAgentRestared TEC_CLASS : HeartBeat_DMAgentRestarted ISA HeartBeat_Event DEFINES { severity: default = WARNING; }; END

Tmw2k.baroc There are changes to two event definitions: TMW_ActionResult and TMW_TaskResult. We will look at this in turn. 򐂰 TMW_ActionResult There are two new slots called program and return code defined in this event, and the slot description has been changed to descriptive_name, as shown in Example 11-9. Example 11-9 Definition of TMW_ActionResult: Definition changes shown in bold TEC_CLASS : TMW_ActionResult ISA TMW_Event DEFINES { method : STRING ; program : STRING ;

Chapter 11. TEC integration

403

descriptive_name : STRING ; return_code : INT32 ; severity: default = WARNING; }; END

򐂰 TMW_TaskResult There is one new slot called return code defined in this event, and the slot tasklib has changed to task_libs, and the slot taskname has changed to tasks, as shown in Example 11-10. Example 11-10 Definition of TMW_TaskResult: Definition changes shown in bold TEC_CLASS : TMW_TaskResult ISA TMW_Event DEFINES { task_libs : LIST_OF STRING ; tasks : LIST_OF STRING ; return_codes : LIST_OF INT32 ; eventtriggername : STRING ; severity: default = WARNING; }; END

11.2.10 Event caching on endpoints and gateways In the event that an endpoint is unable to reach the gateway or the Tivoli event console server, event caching will take place, as described below. If a gateway is down or unreachable, then the events are cached at the endpoint into the following directory: 򐂰 For Windows: %LCF_DATDIR%\LCFNEW\Tmw2k\Tec\cache\ 򐂰 For UNIX: $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Tec/cache/ When the gateway is running again, the cached events are forwarded to the gateway and on to the Tivoli Enterprise Console. The events will be forwarded to the gateway once a connection has been established, and in some cases when a new event is generated. However, if the Tivoli Enterprise Console server is down or unreachable, then the events are forwarded to the gateway and cached in the following directory: 򐂰 For UNIX: /etc/Tivoli/Tec/ 򐂰 For Windows: winnt\system32\drivers\etc\tivoli\tec

404

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

When the Tivoli Enterprise Console server is running again, then the cached events are forwarded to the Tivoli Enterprise Console server when the next event is sent to the gateway. Usually the cache files should have the size of 0 bytes.

Chapter 11. TEC integration

405

406

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

12

Chapter 12.

Tivoli Business Systems Manager integration This chapter discusses integration of IBM Tivoli Monitoring 5.1 with Tivoli Business Systems Manager. This chapter will cover the following: 򐂰 Section 12.1, “Overview of Tivoli Business Systems Manager” on page 408 򐂰 Section 12.2, “Tivoli Business Systems Manager integration” on page 408 򐂰 Section 12.3, “Install/configure IBM Tivoli Monitoring Version 5.1 TBSM Adapter” on page 409 򐂰 Section 12.4, “IBM Tivoli Monitoring 5.1 TBSM Adapter” on page 415

© Copyright IBM Corp. 2002

407

12.1 Overview of Tivoli Business Systems Manager Tivoli Business Systems Manager is a management tool for monitoring business systems in an enterprise. Using Tivoli Business Systems Manager, one can perform distributed management, OS/390 management, or both. Tivoli Business Systems Manager enables monitoring of interconnected business systems across variety of platforms. The Tivoli Business Systems Manager graphical interface allows for viewing in two different formats: Physical resource view and logical resource view. A business component and its resources are termed as a Line of Business (LOB). The LOB concept is used to plan, define, control, and manage the various business components within the LOBs. A typical example of an LOB may consist of a mainframe hosting a huge database, distributed platforms hosting applications that access the database and process transactions, a Web server running the application that users might access to conduct transactions, and the networking infrastructure that connects all of these components. This entire business system could be viewed as a Line of Business and presented as a logical flow of business process to a systems manager. IBM Tivoli Monitoring events are sent to Tivoli Business Systems Manager by the Tivoli Business Systems Manager adapter, an IBM Tivoli Monitoring component that should be installed on all gateways in a region. The adapter feeds Tivoli Business Systems Manager with information about resources managed by Tivoli Monitoring by means of a bulk/delta discovery. For an overview of Tivoli Business Systems Manager, please refer to Tivoli Business Systems Manager A Complete End-to-End Management Solution, SG24-6202.

12.2 Tivoli Business Systems Manager integration Tivoli Business Systems Manager integration is performed in order to enable Tivoli Business Systems Manager to view the resources monitored by IBM Tivoli Monitoring 5.1. The integration between IBM Tivoli Monitoring 5.1 and Tivoli Business Systems Manager provides a complete availability solution that can be viewed at the business level. IBM Tivoli Monitoring 5.1 feeds the availability status of various resources to Tivoli Business Systems Manager and the configured LOB views present the Business Systems Manager critical data to manage the business objects. With this view, the Business Systems Manager is able to understand the business impact the availability status of a resource can have.

408

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The integration between IBM Tivoli Monitoring 5.1 and Tivoli Business Systems Manager is achieved by doing the following: 򐂰 Installing and configuring the CommonListener service on the Tivoli Business Systems Manager Management Server. 򐂰 Installing and configuring the TBSM Adapter on each of the gateways whose attached endpoints are to be monitored. By integrating Tivoli Business Systems Manager with IBM Tivoli Monitoring 5.1, the following functions are performed: 򐂰 TBSM Adapter performs endpoint registration and discovery when the wdmdiscovery command is issued on the gateway. 򐂰 IBM Tivoli Monitoring 5.1 events are sent directly to Tivoli Business Systems Manager through the TBSM Adapter on the endpoint gateway. 򐂰 IBM Tivoli Monitoring 5.1 Health Console can be launched from the Tivoli Business Systems Manager Java Console. The first step towards integrating Tivoli Business Systems Manager with IBM Tivoli Monitoring 5.1 is to install the CommonListener service on the Tivoli Business Systems Manager Management Server. To install the CommonListener, please follow the installation and configuration notes for the CommonListener available in the Patch 1.5-TBSM-0024 Release Notes. The next section describes installation and configuration of the TBSM Adapter for IBM Tivoli Monitoring 5.1. Note: Patch 1.5-TBSM-0024 must be installed on the Tivoli Business Systems Manager Management Server where the CommonListener service is running. Patch 1.5-TBSM-0024 must be installed on the Tivoli Business Systems Manager database server for launching the Health Console from the Tivoli Business Systems Manager Java Console.

12.3 Install/configure IBM Tivoli Monitoring Version 5.1 TBSM Adapter The following is the procedure to install and configure the TBSM Adapter for IBM Tivoli Monitoring 5.1: 1. Install IBM Tivoli Monitoring 5.1 on the gateway if it is not already present. Please refer to 4.2, “Installation and upgrade” on page 80. 2. Install IBM Java Version 1.30 Runtime Environment, shipped with IBM Tivoli Monitoring 5.1, as outlined in 12.3.1, “JRE Version 1.3.0 installation” on page 410.

Chapter 12. Tivoli Business Systems Manager integration

409

3. Install IBM Tivoli Monitoring 5.1 TBSM Adapter, as outlined in 12.3.2, “IBM Tivoli Monitoring 5.1 TBSM Adapter installation” on page 411. 4. Configure the IBM Tivoli Monitoring 5.1 TBSM Adapter, as outlined in 12.3.3, “IBM Tivoli Monitoring 5.1 TBSM Adapter configuration” on page 414.

12.3.1 JRE Version 1.3.0 installation JRE Version 1.3.0 software is shipped along with IBM Tivoli Monitoring 5.1 software. It is installed on each gateway where the TBSM Adapter is to be installed. The following procedure describes the installation process of JRE Version 1.30: 1. From the Tivoli Desktop, select Desktop -> Install -> Product. 2. Click the Select Media button and point the path to the location of jre130.image. 3. Click Set Media & Close. 4. Select IBM Tivoli Monitoring Version 5.1.0- JRE 1.3.0, which appears in the Select Product to Install box, as shown in Figure 12-1 on page 411.

410

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 12-1 JRE Install Product panel

5. Select the clients to install on and click Install & Close. 6. Once the dependency checks are completed, click Continue Install. 7. If the installation is reported successful, JRE Version 1.3.0 for IBM Tivoli Monitoring 5.1 installation is complete.

12.3.2 IBM Tivoli Monitoring 5.1 TBSM Adapter installation The IBM Tivoli Monitoring 5.1 TBSM Adapter is installed on each gateway that is required to send events from its endpoints to Tivoli Business Systems Manager.

Chapter 12. Tivoli Business Systems Manager integration

411

Note: Before installing the IBM Tivoli Monitoring 5.1 TBSM Adapter, add jre_root/jre/bin;/jre_root/jre/bin/classic (where jre_root is the directory where JRE is installed) to the PATH environment variable in both UNIX and Windows systems. The following procedure describes the TBSM Adapter installation. 1. From the Tivoli Desktop, select Desktop -> Install -> Install Product. The Install Product dialog shows three products to install. Select IBM Tivoli Monitoring TBSM Adapter Version 5.1.0, as shown in Figure 12-2.

Figure 12-2 IBM Tivoli Monitoring Version 5.1 TBSM Adapter Install Product

2. Select the clients to install on from the Available Clients panel.

412

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

3. Click Install Options and type the path to where JRE is installed in the dialog box, as shown in Figure 12-3.

Figure 12-3 Tivoli Distributed Monitoring 4.1 TBSM Adapter JRE install path

4. Click Set followed by Close. 5. Click Install & Close in the Install Product dialog box. 6. After the dependency checks are performed, click Continue Install. The installation of IBM Tivoli Monitoring TBSM Adapter Version 5.1.0 is complete.

Chapter 12. Tivoli Business Systems Manager integration

413

12.3.3 IBM Tivoli Monitoring 5.1 TBSM Adapter configuration The IBM Tivoli Monitoring 5.1 TBSM Adapter is configured using the wdmconfig command. The adapter has to be configured before IBM Tivoli Monitoring 5.1 can interact with Tivoli Business Systems Manager. The following parameters must be configured for this functionality: 򐂰 transport.server.mqe.port: The port number that is used by the CommonListener to listen; the default port number is 8082. 򐂰 transport.server.ip.address: The IP address of the Tivoli Business Systems Manager server where the CommonListener service is installed. 򐂰 tbsma.jre_root: The root path of the JRE installation, if this has not been specified at the time of installing the IBM Tivoli Monitoring 5.1 TBSM Adapter. Example 12-1 shows a typical wdmconfig command issued to configure the TBSM Adapter. For details on using the wdmconfig command please refer to IBM Tivoli Monitoring Version 5.1 Users Guide, SH19-4569. Example 12-1 wdmconfig root:/ ==>wdmconfig -m all -D transport.mqe.port=8082 -D transport.server.ip.address=192.168.1.101

Other key/value pairs that can be used for the TBSM Adapter configuration are shown below. All these key/value pairs and their values are contained in the file $DBDIR/dmml/.config. This file can be read but it must not be modified manually. If you wish to modify it, use the wdmconfig command. 򐂰 adapter.trace.enable Set this to true if you want to write to the file identified in trace.filename, all trace messages regarding the operations of the adapter. The default is false. 򐂰 adapter.trace.level Set this to low, medium, or high, according to the level of detail you require if you have enabled adapter trace messages. The default is low. 򐂰 adapter.working.dir Working directory that will be used by the adapter. The default, which you are recommended to use, is the Tivoli Monitoring middle layer directory ($DBDIR/dmml). 򐂰 subnet-mask:network_id Subnet mask of the CommonListener component. For example: subnet-mask:mynetworkid=255.255.255.0

414

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

򐂰 tbsma.jre_root This parameter is set during the installation of the Tivoli Business Systems Manager Adapter (see 12.3.2, “IBM Tivoli Monitoring 5.1 TBSM Adapter installation” on page 411), and you do not normally need to change it manually. However, if, for example, you want to install the adapter on a group of gateways using one instance of the install action or command, you will then need to change this parameter on any gateways in the group that have JRE installed at a location different from that supplied on the Install Options dialog. Set this parameter to the complete path of the root directory of Java Runtime Environment 1.3.0 (excluding the directory /bin). Note: In a Windows NT workstation, if the install target path contains a directory with spaces in the name, the directory name must be specified between single quotes, as in this example, D:\'Program Files'\jre. 򐂰 trace.filename File name to which will be written the trace messages from the adapter. The default is dm.trc. 򐂰 transport.mqe.usefiller Set this to true if the gateway on which the adapter is installed is running Windows NT 4.0 Service Pack 5; otherwise leave is as the default value of false. 򐂰 transport.trace.enable = false Set this to true if you want to write to the file identified in trace.filename, all messages regarding the transport of adapter-acquired data to the CommonListener. The default is false. 򐂰 transport.trace.level Set this to low, medium, or high, according to the level of detail you require if you have enabled transport trace messages. The default is low.

12.4 IBM Tivoli Monitoring 5.1 TBSM Adapter IBM Tivoli Monitoring 5.1 TBSM Adapter runs as tmnt_tbsm_eng.exe on Windows platforms and tmnt_tbsm_eng on UNIX platforms to interface with the DMML processor methods. The TBSM Adapter process can be started or stopped by using the wdmmn command. Please refer to IBM Tivoli Monitoring

Chapter 12. Tivoli Business Systems Manager integration

415

Version 5.1 User’s Guide, SH19-4569, for details on the wdmmn command. The following sections describe how the TSBM Adapter interacts with IBM Tivoli Monitoring 5.1. For troubleshooting Tivoli Business Systems Manager - IBM Tivoli Monitoring 5.1 integration, please refer to CROSSREF to troubleshooting. Once installed and configured, the adapter provides its services in a fully automatic way, not needing to be independently started or stopped. It carries out the following activities: 򐂰 Bulk discovery The wdmdiscovery command can be issued from the Tivoli desktop to carry out a bulk discovery. The adapter sends details of all systems it has in its cache to the Tivoli Business Systems Manager's CommonListener. 򐂰 Delta discovery The same wdmdiscovery command can be issued from the Tivoli desktop to carry out a delta discovery. The adapter sends details of all changes since the last discovery to the Tivoli Business Systems Manager's CommonListener. Figure 12-4 shows the data flow for bulk and delta discovery.

Figure 12-4 Data flow for bulk/delta discovery

416

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The wdmdiscovery command is issued at the Tivoli desktop (steps 1 and 2). At the target gateway, the Tivoli Business Systems Manager Adapter obtains the appropriate discovery information (bulk or delta) from the endpoint cache (step 3) and sends it to the Tivoli Business Systems Manager's CommonListener (also known as the Tivoli BSM Agent Listener) (step 4). Note: The Adapter expects that the CommonListener will respond to a discovery command within a configurable time-out (the default is 300 seconds). If the systems on which these components are running have misaligned time settings, the time-out value may seem to have been exceeded, with the discovery being rejected. The solution is either to align the systems' time settings or to increase the time-out value. As an illustrative example, we used the following simplified setup as our environment: 򐂰 geesun: (Solaris 8) Tivoli management region, gateway, and endpoint (geesun) 򐂰 geesun: (Solaris 8) endpoint 򐂰 b-pc1: (Windows 2000 Server) Tivoli Business Systems Manager database server 򐂰 b-pc1: (Windows 2000 Server) Tivoli Business Systems Manager console server 򐂰 itwin1.dev.tivoli.com: (Windows 2000) Tivoli management region, gateway, and endpoint (itwin1_ep) 򐂰 ITSOTIV5 & ITSOTIV6: (Windows 2000) endpoints 򐂰 tividc15.itsc.austin.ibm.com: (Windows 2000) Tivoli Business Systems Manager database server 򐂰 tividc14.itsc.austin.ibm.com: (Windows 2000) Tivoli Business Systems Manager console server 򐂰 The TBSM Adapter was installed on the gateway itwin1.dev.tivoli.com. It was configured to send heartbeat events to Tivoli Business Systems Manager. A Tivoli Distributed Monitoring (Advanced Edition) 4.1 profile with the TMW_Services resource model was distributed to all three endpoints with the Send event to TBSM option checked. The TBSM Adapter was installed on the gateway geesun. It was configured to send heartbeat events to Tivoli Business Systems Manager. An IBM Tivoli Monitoring 5.1 profile with a TMW_Process resource model was distributed to all endpoints with the Send TBSM option checked in the Profile settings, as shown in Figure 12-5 on page 418.

Chapter 12. Tivoli Business Systems Manager integration

417

Figure 12-5 Profile settings - Send events to TBSM

The Indications settings are shown in Figure 12-6 on page 419.

418

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 12-6 Profile settings - Indications to send TBSM events

12.4.1 Endpoint discovery The Tivoli Business Systems Manager Management Server is populated with the resources monitored by IBM Tivoli Monitoring 5.1 through the TBSM Adapter by using the wdmdiscovery command. When the wdmdiscovery command is issued the first time with the bulk discovery switch, the resource objects monitored by IBM Tivoli Monitoring 5.1 are automatically created in the TSBM server. When the wdmdiscovery command is issued, the TBSM Adapter retrieves the endpoint list from the epcache.db file in the $DBDIR/dmml directory and communicates the discovered resources to the Tivoli Business Systems Manager server through the CommonListener. If the wdmdiscovery command is issued with the bulk discovery option again, then existing objects that were populated by the

Chapter 12. Tivoli Business Systems Manager integration

419

command issued before are deleted before recreating objects for discovered resources. Hence, it is advisable to issue the command with the delta discovery switch if there are not significant changes in the IBM Tivoli Monitoring 5.1 monitoring environment. Issue wdmdiscovery, as shown in Example 12-2, on itwin1 in our environment populated with the Tivoli Business Systems Manager Resource view, as shown in Figure 12-7 on page 421. Example 12-2 Using the wdmdiscovery command ==>wdmdiscovery -m itwin1 -d -a Processing ManagedNode itwin1... -->

If the wdmmngcache command is now run at the gateway, the output is similar to what is shown in Example 12-3. Example 12-3 Using the wdmmngcache command bash$ wdmmngcache -m itwin1 -l -v Processing ManagedNode itwin1... Endpoint | HB status

| TBSM status

------------------------+---------------+-----------ITSOTIV5 HBOff Discovered

420

ITSOTIV6

HBOff

Discovered

itwin1_ep

HBOff

Discovered

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 12-7 Tivoli Business Systems Manager All Resources view

Tivoli Business Systems Manager automatically populated the new resources under Your Company Inc. This name can be changed in the Tivoli Business Systems Manager MS SQL database table CL_AutoPlacement for the field name CL_Instrumentation_to_ENT for your own environment. Double-clicking the newly populated Enterprise icon in the Tivoli Business Systems Manager resource view gave the listing of the three endpoints that were discovered by the TBSM Adapter, as shown in Figure 12-8 on page 422.

Chapter 12. Tivoli Business Systems Manager integration

421

Figure 12-8 Endpoints discovery in Tivoli Business Systems Manager

12.4.2 Event forwarding to Tivoli Business Systems Manager IBM Tivoli Monitoring 5.1 events on the endpoints can be forwarded to Tivoli Business Systems Manager using the IBM Tivoli Monitoring 5.1 TBSM Adapter. There are two kinds of events that can be sent to Tivoli Business Systems Manager, heartbeat events and IBM Tivoli Monitoring 5.1 events. Heartbeat events can be sent to Tivoli Business Systems Manager by configuring the TBSM Adapter, using the wdmconfig command, with the heartbeat.send_events_to_tbsm parameter set to true. The option to forward IBM Tivoli Monitoring 5.1 events to Tivoli Business Systems Manager is configured in the IBM Tivoli Monitoring 5.1 profile before distributing the profile to the endpoints. Please refer to “Profile settings - Send events to TBSM” on page 418 for configuring the IBM Tivoli Monitoring 5.1 profile to send events to Tivoli Business Systems Manager. When a profile with the appropriate setting is distributed to the endpoints and an event occurs, the endpoint invokes a send_event_to_tbsm() request to the Tivoli Business Systems Manager processor (tmnt_tbsm_eng.exe on Windows platform and tmnt_tbsm_eng on UNIX platforms) on the gateway. The Tivoli

422

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Business Systems Manager processor then forwards the event to the CommonListener on the Tivoli Business Systems Manager Management Server. The Tivoli Business Systems Manager Management Server updates the new event information accordingly. Cached event files named TMA-label.cached-events contain the events to be forwarded to Tivoli Business Systems Manager. In our environment, the heartbeat function was turned on at the gateway itwin1 and the TBSM Adapter was configured with the heartbeat.send_events_to_tbsm parameter set to true. The endpoint service was stopped on the endpoint itwin1_ep. This resulted in the Tivoli Business Systems Manager properties view of itwin1_ep showing an alert with the details of the event. Figure 12-9 shows the resulting Tivoli Business Systems Manager properties view of itwin1_ep.

Figure 12-9 Properties of Tivoli Distributed Monitoring 4.1 resource

Chapter 12. Tivoli Business Systems Manager integration

423

424

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

13

Chapter 13.

Data Warehouse This chapter describes the integration of IBM Tivoli Monitoring Version 5.1 with Tivoli Enterprise Data Warehouse. For Tivoli Enterprise Data Warehouse concepts and installation, refer to Introduction to Tivoli Enterprise Data Warehouse, SG24-6607. This chapter contains the following: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

13.1, “Introduction to Tivoli Enterprise Data Warehouse” on page 426 13.2, “IBM Tivoli Monitoring Version 5.1 data flow” on page 428 13.3, “Installing IBM Tivoli Monitoring Version 5.1 PAC” on page 429 13.4, “Configuring TEDW” on page 449 13.5, “Creating and configuring the data mart” on page 462 13.6, “Creating a report” on page 468

© Copyright IBM Corp. 2002

425

13.1 Introduction to Tivoli Enterprise Data Warehouse The Tivoli Enterprise Data Warehouse is an application used to collect and manage data from various Tivoli and non-Tivoli system management applications. The data is imported from the source applications, stored centrally, and further processed to fit the needs of the end users. We describe the basic components of the TEDW in the logical order of the data flow, as shown in Figure 13-1.

Tivoli Warehouse

Control Server: IBM DB2® DWC

Warehouse Metadata

Tivoli Reporting Services

Source Apps

ITM

ETL

Inventory

ETL

Tivoli Reporting Interface Data Marts Data Marts ETL

TEC

ETL

Central Data Warehouse

Data Marts Data Marts

Business Intelligence Tools

Data Marts Data Marts

Source App

ETL

IBM

Cognos

Brio

Business Objects

Figure 13-1 Tivoli Data Warehouse data flow

The first step to introducing TEDW is enabling the source applications. This means to provide all tools and customizations necessary to import the source operational data into the central data warehouse. All components needed for that task are collected in warehouse packs for each source application. An important part of the warehouse packs is the ETL programs. The abbreviation of ETL is for Extract, Transform, and Load data. In principle ETL programs process data in three steps. First they extract the data from a data source. Then the data is validated, transformed, aggregated, and/or cleansed so that it fits the format and needs of the data target. Finally the data is loaded into the target database.

426

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

In TEDW there are two types of ETLs. The central data warehouse ETL pulls the data from the source applications and loads it into the central data warehouse. The central data warehouse ETL is also known as source ETL or ETL1. The second type of ETL is the data mart ETL. The central data warehouse (CDW) is the database that contains all enterprise-wide historical data (with hour as the lowest granularity). This data store is optimized for the efficient storage of large amounts of data and has a documented format that makes the data accessible to many analysis solutions. The database is organized in a very flexible way, and you can store data from new applications without adding or changing tables. The data mart ETL extracts a subset of historical data from the central data warehouse that contains data tailored to and optimized for a specific reporting or analysis task. This subset of data is used to create data marts. Data mart ETL is also known as target ETL or ETL2. A data mart is a subset of the historical data that satisfies the needs of a specific department, team, or customer. A data mart is optimized for interactive reporting and data analysis. The format of a data mart is specific to the reporting or analysis tool you plan to use. Each application that provides a data mart ETL creates its data marts in the appropriate format. TEDW provides a Report Interface (RI) that creates static two-dimensional reports of your data using the data marts. The RI is a role-based Web interface that can be accessed with a simple Web browser without any additional software installed on the client. You can also use other tools to perform OLAP analysis, business intelligence reporting, or data mining. The Control server is the system that contains the control database, which contains metadata for Tivoli Enterprise Data Warehouse and from which you manage your data warehouse. The Control server controls communication between the Control server, the central data warehouse, the data marts, and the Report Interface. The Control server uses the Data Warehouse Center to define the ETL processes and the star schemas used by the data marts. You use the Data Warehouse Center to schedule, maintain, and monitor these processes. For more information about Tivoli Enterprise Data Warehouse, refer to Introduction to Tivoli Enterprise Data Warehouse, SG24-6607.

Chapter 13. Data Warehouse

427

13.2 IBM Tivoli Monitoring Version 5.1 data flow The data flow between IBM Tivoli Monitoring Version 5.1 and TEDW is described in Figure 13-2.

Figure 13-2 IBM Tivoli Monitoring Version 5.1 data flow to TEDW

428

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

The data flow is: 1. The data is logged on the local endpoint database. 2. Data is updated in the RIM object (spr_rim). This is done after running the tasks DMAE_ReadDataIn, DMAE_NewDataAggregation, and DMAE_RollupIntoDB. 3. Data already in the relational database is processed by the TEDW jobs AMW_c10_s010_loadDMData and AMW_c10_s010_process_DMData.

13.3 Installing IBM Tivoli Monitoring Version 5.1 PAC We now describe the installation steps to enable IBM Tivoli Monitoring Version 5.1 to record data into TEDW. To proceed with the following steps, we assume that you have already installed Tivoli Enterprise Data Warehouse and DB2.

13.3.1 Installing IBM Tivoli Monitoring Version 5.1 PAC in TEDW In this section we describe the installation procedures for IBM Tivoli Monitoring Version 5.1 PAC. All the steps must be executed on the TEDW box. IBM Tivoli Monitoring Version 5.1 PAC is located on the IBM Tivoli Monitoring Version 5.1 TEDW CD. To install IBM Tivoli Monitoring Version 5.1 PAC you can use the GUI or CLI. We installed it using the GUI, following steps in our ITSO environment (tividc13). 1. Start the TEDW installation, as shown in Figure 13-3 on page 430.

Chapter 13. Data Warehouse

429

Figure 13-3 Install welcome screen

2. Select Application installation only, as shown in Figure 13-4 on page 431.

430

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-4 Install type dialog (application only)

3. Confirmation of the system host name, as shown in Figure 13-5.

Figure 13-5 Confirmation of local host name

4. Enter the user name and password settings, as shown in Figure 13-6 on page 432.

Chapter 13. Data Warehouse

431

Figure 13-6 Specifying DB2 user name and password

5. Setting the directory for IBM Tivoli Monitoring Version 5.1 PAC, the directory must be in the leaf of the file twh_app_install_file.cfg, as shown in Figure 13-7.

Figure 13-7 Set directory name for IBM Tivoli Monitoring Version 5.1 PAC

6. Since we are installing only one PAC, we did not choose to install additional packages, as shown in Figure 13-8 on page 433.

432

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-8 Install additional packages dialog

Note: At this moment, you can choose to install multiple PACs; however, we do not recommend it. We suggest installing one PAC at a time. 7. You will receive a confirmation of the PAC installation, as shown in Figure 13-9.

Figure 13-9 Confirm installation options

8. The installation is in progress, as shown in Figure 13-10 on page 434.

Chapter 13. Data Warehouse

433

Figure 13-10 Installing application

9. When the PAC installation is finished you will see a screen like that shown in Figure 13-11.

Figure 13-11 Installation complete

434

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Important: Introduction to Tivoli Enterprise Data Warehouse, SG24-6607, mentions that the NT Services Warehouse Server and Warehouse logger must be reconfigured to run as the user db2admin. If you have not made this change, you will see failures when trying import data into the Data Warehouse. Please confirm that you have reconfigured and restarted these services. 10.After the PAC installation conclusion, you must reboot TEDW.

13.3.2 Installing IBM Tivoli Monitoring Version 5.1 Data Gathering In this section we describe the necessary steps for configuring the data gathering on the TMR and gateways. You should install the IBM Tivoli Monitoring Gathering Historical Data Component in the TMR and all the ManagedNodes/gateways you want to collect data from where endpoints to send to TEDW. This has as requirement IBM Tivoli Monitoring Version 5.1.0. Note: Before starting the installation, do the following step: Create a user at the Operating System Level in the RDBMS box with the same name as your DB2 instance to be used by RIM. In our environment, we created the account DB2, adding this account to the groups Administrators and Tivoli_Admin_Privileges. In our ITSO environment, we installed the IBM Tivoli Monitoring Gathering Historical Data Component on the TMR and all ManagedNodes. We installed this using the Tivoli desktop, as follows: 1. From the Tivoli desktop, select Desktop -> Install -> Install Product. 2. The install product dialog shows three products that are available to install (Figure 13-12 on page 436).

Chapter 13. Data Warehouse

435

Figure 13-12 Installing Data Gathering Component on TMR Server

3. Select IBM Tivoli Monitoring Gathering Historical Data Component, then select your Tivoli management region server and the gateways that you want to install IBM Tivoli Monitoring Version 5.1 on. Select Install and follow the normal installation dialogs. 4. RIM configuration is required to proceed with the installation, as shown in Figure 13-13 on page 437.

436

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

.

Figure 13-13 RIM options for Data Gathering installation

Note: In our ITSO environment, the installation has not created the RIM object, which must have the label spr_rim, so we created the RIM using the CLI: wcrtrim -v DB2 -h tividc13 -d AMW -u db2 -H e:/sqllib -s tcpip -I db2 spr_rim

After the installation, you must run the following scripts: 1. RIM host – Set the Tivoli environment variables using the DB2 command line. – $BINDIR/TME/Tmw2k/TDS/rdbcfg/cr_rollup_db.sh. 2. TMR Server $BINDIR/TME/Tmw2k/TDS/rdbcfg/twh_enable.sh.

Chapter 13. Data Warehouse

437

3. RIM host – Set the Tivoli environment variables using the DB2 command line. – $BINDIR/TME/Tmw2k/TDS/rdbcfg/run_query.sh twh_enabl. The installation creates a new PolicyRegion tividc-11_SPR_Region (see Figure 13-14).

Figure 13-14 PolicyRegion SPR_Region

This contains one profile manager (SPR_ProfileMgr) and one Task Library (SPR_TaskLib), as shown in Figure 13-15 on page 439.

438

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-15 PolicyRegion tividc11_SPR_Region

The TaskLibrary SPR_TaskLib contains the tasks/jobs, as shown in Figure 13-16.

Figure 13-16 TaskLibrary SPR_TaskLib

Chapter 13. Data Warehouse

439

13.3.3 Explanation of installed tasks/jobs This section describes the tasks and jobs that are installed as part of the IBM Tivoli Monitoring Version 5.1 PAC.

DMAE_ReadDataInDB Table 13-1 provides the default job configuration details. Table 13-1 Job DMAE_ReadDataInDB configuration Job name

DMAE_ReadDataInDB

Associated task name

DMAE_ReadDataInDB

Associated script

$BINDIR/TME/Tmw2k/TDS/spr/inv_read_ data.sh

Output file

/tmp/dmae_dmquery.log

Schedule

01:00 daily

Execution

Staged

Timeout

3600 seconds

Staging count

12

Staging interval

70 seconds

Target for execution

@ProfileManager:SPR_ProfileMgr

Actions on endpoint The following are actions on an endpoint. 򐂰 Creates /tmp/$EPLabel/$ProfileName/$ModelName-$MetricName for SPR-related profiles. 򐂰 Logs data into /tmp/dmae_spp_guide.log. 򐂰 Runs dmquery to extract "yesterday's" data, creating files in the directory structure just mentioned (info, MIN.log, MAX.log, AVG.log). 򐂰 Runs Perl script ($LCF_DATDIR/cache/bin/$INTERP/TME/Tmw2k/copy_log_at_ep.pl).

Additional information If the subsequent job (DMAE_NewDataAggregation) succeeds, the files mentioned will be deleted. if something fails during the execution of DMAE_NewDataAggregation, these backup files will be not deleted and they will be used to aggregate data on following days using the tasks DMAE_AggregateOnDate and DMAE_RollupIntoDBOnDate.

440

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

An execution example with extra logging turn on is within.\DMAE_ReadDataInDB_extra_logging.txt.

DMAE_NewDataAggregation Table 13-2 provides the default job configuration details. Table 13-2 Job DMAE_NewDataAggregation configuration Job name

DMAE_NewDataAggregation

Associated task name

DMAE_NewDataAggregation

Associated script

$BINDIR/TME/Tmw2k/TDS/tasks/wrappe r_for_aggreg.sh

Output file

/tmp/dmae_out.log

Schedule

02:00 daily

Execution

Staged

Timeout

68 seconds

Staging count

12

Staging interval

70 seconds

Target for execution

@ProfileManager:SPR_ProfileMgr

Tip: In our environment, the DMAE_NewDataAggregation job was configured to run against the profile manager SPR_ProfileMgr. This job should actually be configured to run on the TMR Server.

Actions The following are actions: 򐂰 Endpoint Simply checks for /tmp to exist; if it does not, it is created. (Since DMAE_ReadDataInDB already performs this check, this is essentially a “do nothing” task, which is why this job should be reconfigured to only run on the TMR Server.) 򐂰 ManagedNode Runs TME job name (DMAE_DataAggregation) from SPR_TaskLib. – Reads on each endpoint the files created in /tmp/ep_name/profile_name/rm_name metric_name.

Chapter 13. Data Warehouse

441

– Arranges the data in a unique file containing all the data (AVG, MAX, and MIN) for all the endpoints. It is called dmae_out.log.date and it is created in the directory /tmp (or installation_drive:/tmp on NT/W2K) on the TMR Server. Runs Perl script ($BINDIR/TME/Tmw2k/TDS/tasks/move_log.pl), which renames /tmp/dmae_out.log to /tmp/dmae_out.log.YYMMDD.

DMAE_RollupIntoDB Table 13-3 provides the default job configuration details. Table 13-3 Job DMAE_RollupIntoDB configuration Job Name

DMAE_RollupIntoDB

Associated task name

DMAE_RollupIntoDB

Associated script

$BINDIR/TME/Tmw2k/TDS/spr/inv_rdb.s h

Output file

/tmp/dmae_spp_guide.log

Schedule

03:30 daily

Execution

Parallel

Timeout

3600 seconds

Target for execution

TMR Server

Actions on TMR Server The actions on the TMR Server are: 򐂰 Runs the Perl script ($BINDIR/bin/rolup2db.pl) with the parameters: – – – – – –

$1 - tasklib (SPR_TaskLib) $2 - taskname (DMAE_NewDataAggregation) $3 - jobname (DMAE_NewDataAggregation) $4 - rim (spr_rim) $5 - $1 (anything passed from CLI to DMAE_RollupIntoDB task itself) $6 - Warehouse enabled flag (-twh)

򐂰 Reads the file dmae_out.log.date. 򐂰 Inserts data into the RIM (spr_rim).

Additional information If something fails during the execution of DMAE_RollupIntoDB, the file dmae_out.log.date will be not deleted. It can be used to store data in RDBMS on following days using the task DMAE_RollupIntoDBOnDate.

442

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

DMAE_DataAggregation Table 13-4 provides the default job configuration details. Table 13-4 Job DMAE_DataAggregation configuration Job name

DMAE_DataAggregation

Associated task name

DMAE_DataAggregation

Associated script

$BINDIR/TME/Tmw2k/TDS/spr/inv_aggr. sh

Output file

/tmp/dmae_out.log

Schedule

launched by DMAE_NewDataAggregation

Actions The action is that it runs Perl script ($LCF_DATDIR/cache/bin/$INTERP/TME/Tmw2k/dmae_aggreg.pl). 򐂰 Reads on each endpoint the files MIN, MAX, and AVE.log.date created in /tmp/ep_name/profile_name/rm_name-metric_name. 򐂰 Arranges the data in a unique file containing all the data (AVG, MAX, and MIN) for all the endpoints. It is called dmae_out.log.date and it is created in the directory /tmp (or installation_drive:/tmp on NT/W2K) on the TMR Server.

Schedule of jobs The jobs DMAE_ReadDataIn, DMAE_NewDataAggregation, and DMAE_RollupIntoDB are scheduled to run every day, as shown in Figure 13-17 on page 444.

Chapter 13. Data Warehouse

443

Figure 13-17 Scheduled jobs

The profile manager SPR_ProfileMgr contains the Tmw2kProfiles, as shown in Figure 13-18 on page 445.

444

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-18 Tmw2kProfiles inside SPR_ProfileMgr

The Tmw2kProfile SPR_NtProfile is created with a default cycle time of 30 seconds (as shown in Figure 13-19 on page 446) and contains three parameters (as shown in Figure 13-20 on page 447).

Chapter 13. Data Warehouse

445

Figure 13-19 Tmw2kProfile SPR_NtProfile settings The Tmw2kProfile SPR_NtProfile contains three parameters, as shown in Figure 13-20 on page 447.

446

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-20 Tmw2kProfile SPR_NtProfile parameters The Tmw2kProfile SPR_UNIXProfile is created with a default cycle time of 180 seconds (as shown in Figure 13-21 on page 448) and contains two parameters (as shown in Figure 13-22 on page 449).

Chapter 13. Data Warehouse

447

Figure 13-21 Tmw2kProfile SPR_UNIXProfile settings

The Tmw2kProfile SPR_UNIXProfile contains two parameters, as shown in Figure 13-22 on page 449.

448

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-22 Tmw2kProfile SPR_UNIXProfile parameters

Note: You must distribute either SPR_NTProfile or SPR_UnixProfile to every endpoint on which you desire historical data gathering.

13.4 Configuring TEDW In order to configure TEDW to analyze IBM Tivoli Monitoring Version 5.1 data, perform the following steps in the TEDW system: 1. Create a new ODBC Data Source labeled AMW_RIM, pointing to the RIM database (in our environment, the database is AMW). 2. For this ODBC Data Source, we have configured the same user ID and password as the RIM setting, as shown in Figure 13-23 on page 450 and Figure 13-24 on page 450.

Chapter 13. Data Warehouse

449

Figure 13-23 Connecting to AMW database

Figure 13-24 Setting the user ID and password in the ODBC

3. Since we created the DB2 account in the operating system, we also created the account DB2 in the DB2 Data Warehouse.

450

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Using the DB2 Data Warehouse Center, click Administration -> Warehouse Users and Groups, and right-click in Warehouse Users. Then choose Define, as shown in Example 13-25.

Figure 13-25 Creating the user DB2 in DB2 Data Warehouse

Figure 13-26 on page 452 shows the required data for the account creation.

Chapter 13. Data Warehouse

451

Figure 13-26 Creating an account in DB2 Data Warehouse Center

4. As the database AMW was created with the account DB2, we granted the permission to the user db2admin. Using the DB2 Control Center, click Systems -> hostname -> Instances -> DB2 -> Databases. Then right-click on the database used by RIM, choosing Authorities, as shown in Figure 13-27 on page 453.

452

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-27 Getting the authorities from the database

Figure 13-28 on page 454 shows the user db2admin with all rights in the database.

Chapter 13. Data Warehouse

453

Figure 13-28 Permissions granted to db2admin

5. Using the DB2 Data Warehouse Center, set the user ID and password for all the warehouse sources and warehouse targets. Figure 13-29 on page 455 shows where to access properties.

454

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-29 Getting to AMW_RIM_Source properties

Figure 13-30 on page 456 shows the settings for the warehouse source AMW_RIM. For this specific warehouse source we used the account DB2. This same procedure should be repeated for all of the warehouse sources and warehouse targets, however, setting the account as db2admin. In our environment this meant setting six total sources/targets (three sources and three targets).

Chapter 13. Data Warehouse

455

Figure 13-30 Setting AMW_Rim_Source user ID and password

6. Using the DB2 Data Warehouse Center, schedule the process AMW_c10_loadDMData_Process. Click Subject Areas -> AMW_ApplicationEnablement_v5.1.0_Subject_Areas -> Processes, and right-click AMW_c10_loadDMData_Process. Then choose Schedule, as shown in Figure 13-31 on page 457.

456

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-31 Schedule option for AMW_c10_loadDMData_Process

Selecting Schedule will open up a dialog box, as shown in Figure 13-32 on page 458.

Chapter 13. Data Warehouse

457

Figure 13-32 Configuring daily schedule for AMW_c10_loadDMData_Process

7. Using the DB2 Data Warehouse Center, change the scheduled process to Production, as shown in Figure 13-33 on page 459.

458

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-33 Promoting schedule to Production mode

8. We found that a specific reconfiguration was required in order for us to successfully run the process AMW_c10_loadDMData_Process. The following steps should be issued: a. Using the DB2 Control Center, open Databases, choose the database created and used by RIM (in our environment the database is AMW) and click Tables. b. Check the Schema field for the tables DM_METRICS and TWH_DM_METRICS, as shown in Figure 13-34 on page 460. In our environment the schema for these tables is DB2.

Chapter 13. Data Warehouse

459

Figure 13-34 Using DB2 Control Center to view the AMW database schema

c. Using the DB2 Data Warehouse Center, click Subject Areas -> AMW_ApplicationEnablement_v5.1.0_Subject_Areas -> Processes -> AMW_c10_loadDMData_Process. Then right-click the table DM_METRICS, choosing Properties, as shown in Figure 13-35 on page 461.

460

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-35 Getting DM_METRICS properties

d. Change the Table schema field, to the schema that the table belongs to, as shown in Figure 13-36. In our environment the table schema is DB2.

Figure 13-36 Setting the right schema for the table DM_METRICS

Chapter 13. Data Warehouse

461

e. You can confirm your settings by right-clicking DM_METRICS and choosing Sample Contents. If this action produces a table (no error messages), you have your source table configured properly. f. Repeat the previous step using the source table TWH_DM_METRICS.

13.5 Creating and configuring the data mart The following highlights the steps necessary to create and configure your new data mart containing IBM Tivoli Monitoring Version 5.1 data. 1. Connect to the IBM Console by pointing a Web browser at: http://YourWarehouseServer/IBMConsole

Note: We logged in using the default super account of superadmin with a default password of password. As noted in Introduction to Tivoli Enterprise Data Warehouse, SG24-6607, this account has all privileges. As a result, you may see multiple/duplicate menus. This is unique to this particular account. You should always select the top-most menu option to run that option under the most privileged context

2. Under My Work select Manage Data Marts, as shown in Figure 13-37 on page 463.

462

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-37 Data Warehouse main page

3. Click the double-arrow (>>) next to Root and choose Create, as shown in Figure 13-38.

Figure 13-38 Begin creation of new data mart

4. Provide general information for your new data mart, including the name, description, database user ID/password, and database connection string, as shown in Figure 13-39 on page 464.

Chapter 13. Data Warehouse

463

Note: The Database Driver field should not be modified. In our environment, the database connection string was: jdbc:db2:TWH_MART

Your environment may differ slightly, but it must include jdbc:db2:dbname if you are using DB2.

Figure 13-39 Filling out general information for new data mart

5. Next click the User Groups item, just below General toward the middle of your screen. The resulting screen can be seen in Figure 13-40 on page 465.

464

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-40 Configuring user groups within data mart

6. Click the Add button to add a user group. 7. Select the default user group TWHAdmin, as shown in Figure 13-41.

Figure 13-41 Selecting the TWHAdmin user group

8. Click OK to confirm your user group selection. 9. Next click the Star Schema item, just below User Groups. The resulting screen can be seen in Figure 13-42 on page 466.

Chapter 13. Data Warehouse

465

Figure 13-42 Adding a star schema to the data mart

10.Click Add to add a new start schema. 11.Select AMW Daily Performance Star Schema, as shown in Figure 13-43.

Figure 13-43 Selecting the AMW star schema

12.Click OK to select the star schema. This brings you back to the dialog shown in Figure 13-44 on page 467.

466

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-44 Finalizing creation of data mart

13.Click OK at the bottom of the window to complete the creation of your data mart (Figure 13-44). 14.You will return to the Manage Data Marts screen. However, your newly created data mart will probably not be visible (Figure 13-45).

Figure 13-45 Before refreshing data mart view

15.In order to see your new data mart, simply click Refresh toward the top of your window. The result should be similar to Figure 13-46 on page 468.

Chapter 13. Data Warehouse

467

Figure 13-46 After refreshing data mart view - Creation complete

13.6 Creating a report The following example walks you through the creation of your first report utilizing IBM Tivoli Monitoring Version 5.1 data within the Tivoli Enterprise Data Warehouse. In this example we will create a report that displays the minimum, maximum, and average disk utilization of the root (/) file system on our UNIX endpoints. 1. Log in to the IBM Console (http://YourWarehouseServer/IBMConsole). Note: We used the default user of superadmin (default password = password).

2. Under My Work select Create Report. Note: As mentioned previously, because we are running as superadmin, there are two Create Report options. Choose the top-most option.

3. Select your data mart, as shown in Figure 13-47 on page 469.

468

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-47 Creating a report - Selecting data mart

4. Provide general report information, including name, description, and whether this is a public report (viewable by all users). See Figure 13-48 on page 470.

Chapter 13. Data Warehouse

469

Figure 13-48 Creating a report - Filling out general information

5. Click the Add button to add metrics to your report, as shown in Figure 13-49.

Figure 13-49 Creating a report - Adding metrics

470

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

6. Using the arrow buttons surrounding Page x of y, scroll through the list of available metrics and select those you wish to report. In our example we selected availKBytes avg, availKBytes min, and availKBytes max as shown in Figure 13-50 (availKBytes max is not shown in this figure; it came from the page immediately following the one in Figure 13-50—you can select metrics from multiple pages, as we did).

Figure 13-50 Creating a report - Selecting metrics

7. Once you have selected your metrics, click the Next button to proceed to the Specify Aggregations screen shown in Figure 13-51 on page 472.

Chapter 13. Data Warehouse

471

Figure 13-51 Creating a report - Specify aggregations

8. Choose the appropriate aggregation for each metric (as shown in Figure 13-51) then select Next to see the dialog box shown in Figure 13-52 on page 473. Tip: The name of the metric gives you a good clue as to the aggregation type to select. 򐂰 avg -> Average 򐂰 min -> Minimum 򐂰 max -> Maximum

472

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-52 Creating a report - Specify group by and order by

9. Specify any desired filtering (optional) as well as the group by (required) and order by (required). In Figure 13-52 we grouped and ordered by the host name. 10.Click the Finish button to bring up the confirmation dialog box, as shown in Figure 13-53.

Figure 13-53 Creating a report - Confirming metrics and aggregation

Chapter 13. Data Warehouse

473

11.You can confirm your metrics and aggregations. Click Time, as indicated by the cursor in Figure 13-53 on page 473, to bring up the dialog box shown in Figure 13-54.

Figure 13-54 Creating a report - Specify general time frame

12.Specify the time frame for this report. In our example, we configured the report to span the last 365 days (though we did not have historical data that old). Click OK and your report will be created. If everything was successful, you should see a window similar to Figure 13-55 on page 475.

474

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-55 Creating a report - Confirmation

13.Click OK and your report will be now manageable. 14.Return to the Manage Reports and Report Output screen (under My Work), then select Reports, as shown in Figure 13-56.

Figure 13-56 Running the report - Selecting from the Manage Reports view

Chapter 13. Data Warehouse

475

15.Click the double-arrow (>>) next to your report and choose Run, which will open the Time Details dialog shown in Figure 13-57.

Figure 13-57 Running the report - Specify general time frame

16.Specify the time frame; it will default to the value you chose when creating this report. In our example we left it at the default, as shown in Figure 13-57. Selecting Next will bring up the Report Output Name dialog box, as shown in Figure 13-58 on page 477.

476

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Figure 13-58 Running the report - Specify report name and description

17.Provide a report output name and description. We left the defaults. 18.Click the Run button to execute your report. Your report is generated as in Figure 13-59.

Figure 13-59 Running the report - My First Report completed

Chapter 13. Data Warehouse

477

19.From here you can save or print your report, or you can simply close the report window (your report will not be saved).

478

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Part 5

Part

5

Troubleshooting

© Copyright IBM Corp. 2002

479

480

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

14

Chapter 14.

Troubleshooting This chapter will give you information on how to generate and obtain logs and traces that are available for resolving problems in your IBM Tivoli Monitoring 5.1 environment. Beyond the overview of the logs, we show you a real example of troubleshooting the profile distribution process and the running engine. The following topics are described in this chapter: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

14.1, “Logs and traces format” on page 482 14.2, “TMR Server logs and traces” on page 483 14.3, “Gateway logs and traces” on page 484 14.4, “Endpoint logs” on page 490 14.5, “Tools” on page 498 14.6, “Known problems and their resolutions” on page 503 14.7, “Problem determination” on page 503

© Copyright IBM Corp. 2002

481

14.1 Logs and traces format When there is a problem or strange behavior, you should check the logs. If the problem still persists after further analysis, send the traces to Tivoli Customer Support. IBM Tivoli Monitoring 5.1 provides a new format for logs and traces. Currently log files are available only for non-Windows endpoints. For Windows endpoints a trace file is available. The log record format: Date1Date2ProductIDComponentServerProcessIDMessageIdLog Text

The trace record format: Date1Date2ProductIDComponentServerProcessIDTraceLevFile NameMethodThreadLogTextException

Where:

482

Date1

Time the log was produced specified in milliseconds; since January 1st 1970, for example, 1015343592000.

Date2

Time the log was produced specified in GMT. It includes the date, time, and time zone; for example, Tue Apr 05 15:53:12 2002 GMT.

ProductID

The three-letter code assigned to the product for identifying its messages uniquely, AMW, for IBM Tivoli Monitoring 5.1.

Component

Represents a runtime grouping of a product’s parts. If a product has multiple applications, the component name will reflect the name of the application.

Server

Host name (ManagedNode label or endpoint label).

ProcessID

The process ID of the process that produced the log message. Only present for Java processes.

MessageId

The unique numeric message identifier.

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

TraceLev

The level of detail the trace represents. MAX: Deep level of tracing MIN: Minimum level of tracing MID: A level between MIN and MAX OTHER: Highest level of tracing; in our environment we only observed these level in traces of UNIX endpoints

FileName

Name of the file or class to which the trace refers.

Method

The method in the class to which the trace refers.

Thread

Thread to which the trace refers. Thread element refers to the platform-specific notion of a thread.

LogText

A description of the error.

Exception

Exception element will depend upon the specific language/platform. For Java, it will be a stack trace.

Important: We noticed that the log and trace timestamp in almost all the ManagedNode logs and traces are based on a fixed GMT time zone. In our environment the time zone was CST6, generating a timestamp inconsistency between the files. To bypass this issue, we changed the time zone to GMT, then the logs and traces have a consistent timestamp.

14.2 TMR Server logs and traces In TMR servers, IBM Tivoli Monitoring 5.1 provides a distribution log in addition to the Tivoli Framework log files.

14.2.1 Profile distribution process The profile distribution process records information about distribution and status on the TMR Server, so you should analyze a distribution problem gathering logs from the TMR Server. The log file maintained in the TMR contains information about profile distribution to any of the subscribers. Name of the log file

msg_Tmw2kProfilename.log

Location of the log file

$DBDIR/AMW/logs

Chapter 14. Troubleshooting

483

Configuration

To configure the log when distributing by means of the command wdmdistrib, use the options - e, - i, and - w. There are no configuration options when the GUI is used to perform a distribution. The default is that all options are specified.

The profile distribution process uses MDist2, so it might be necessary to check the MDist2 logs. 򐂰 򐂰 򐂰 򐂰

TMR Distribution Manager: $DBDIR/distmgr.log Gateway Repeater: $DBDIR/gatelog ManagedNode Repeater: $DBDIR/rpt2log Endpoint: $LCF_DATDIR/lcfd.log Tip: You can use the wmdist -D command to change the level of detail for distmgr.log. Valid debug numbers are from 0 (least information) through 9 (most information). The default value is 3.

For more information about MDist2 troubleshooting, refer to Tivoli Software Distribution Version 4.1: New Features and Scenarios, SG24-6045.

14.3 Gateway logs and traces At the gateway, the product maintains eight traces: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

14.3.1, “Heartbeat engine traces” on page 485 14.3.2, “Task engine trace” on page 486 14.3.3, “Tivoli Business Systems Manager engine traces” on page 486 14.3.4, “Tivoli Business Systems Manager adapter trace” on page 487 14.3.5, “Tivoli Business Systems Manager transport trace” on page 487 14.3.6, “Endpoint upcall traces” on page 488 14.3.7, “Request manager trace” on page 488 14.3.8, “Profile core trace” on page 489

All ManagedNode components produce traces. Traces can be configured using the command: wdmconfig -m node –D component_name.trace_level=value wdmconfig -m node –D component_name.trace_size=value

Where component_name is the name of the component that produces the trace:

484

heartbeat

For the heartbeat engine

task

For the task engine

tbsma

For the Tivoli Business Systems Manager engine

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

core

For the profile core engine

request-manager

For the request manager engine

gw

For the endpoint upcall

dmml

For the distributed middle layer

The trace_level of the components can be configured as follows: 򐂰 0 No trace information is generated. 򐂰 1 MID trace information. 򐂰 2 MAX trace information. Note: We noticed that changing the parameters dmml.trace_level and dmml.trace_size will also configure the trace settings for the following components: Heartbeat engine, task engine, tbsm engine, and request manager engine.

14.3.1 Heartbeat engine traces A trace is maintained at the gateway of the activities carried out when the heartbeat engine is running. It records all messages output by the process tmnt_hb_eng. The details are as follows: Process name

tmnt_hb_eng, executable location $BINDIR/TME/Tmw2k

Trace name

trace_tmnt_hb_engn.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

$DBDIR/AMW/logs

Configuration

To configure the log, use the command: wdmconfig -m node –D heartbeat.trace_level= 0 - 2 wdmconfig -m node –D heartbeat.trace_size=value

After changing the trace level or size, the heartbeat engine should be restarted to reflect the changes. The heartbeat engine should be stopped using the command: wdmmn -stop -m node -h

Chapter 14. Troubleshooting

485

The heartbeat engine should be started using the command: wdmheartbeat -m node -s frequency

14.3.2 Task engine trace A trace is maintained at the gateway of the activities carried out when the task engine is running to perform tasks on attached endpoints as determined by resource model definitions. It records all messages output by the process tmnt_task_eng. The details are as follows: Process name

tmnt_task_eng, executable location $BINDIR/TME/Tmw2k

Trace name

trace_tmnt_task_engn.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

$DBDIR/AMW/logs

Configuration

To configure the log, use the command: wdmconfig -m node –D task.trace_level= 0 - 2 wdmconfig -m node –D task.trace_size=value

After changing the trace level or size, the task engine should be restarted to reflect the changes. The task engine should be stopped using the command: wdmmn -stop -m node -t

The task engine should be started using the command: wdmmn -start -m node -t

14.3.3 Tivoli Business Systems Manager engine traces Two traces are maintained at the gateway of the activities carried out when the Tivoli Business Systems Manager engine is running. They record all messages output by the process tmnt_tbsm_eng, which implements the CORBA methods. The details are as follows:

486

Process names

tmnt_tbsm_eng, tmnt_tbsm_wrapper, executable location $BINDIR/TME/Tmw2k

Trace names

trace_tmnt_tbsm_engn.log trace_tmnt_tbsm_wrappern.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

$DBDIR/AMW/logs

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Configuration

To configure the log, use the command: wdmconfig -m node –D tbsma.trace_level=0 - 2 wdmconfig -m node –D tbsma.trace_size=value

After changing the trace level or size, the Tivoli Business Systems Manager engine must be restarted to reflect the changes. The TBSM engine should be stopped using the command: wdmmn -stop -m node -b

The TBSM engine should be started using the command: wdmmn -start -m node -b

14.3.4 Tivoli Business Systems Manager adapter trace A log is maintained at the gateway of the activities carried out by the Tivoli Business Systems Manager adapter, as follows: Log name

User-defined; default is dm.trc (when the log is full it is renamed as filename.old, deleting any existing file with that name, and a new log file is created).

Location

As defined in the wdmconfig configuration variable adapter.working.dir (default is $DBDIR/dmml).

Configuration

To configure the log, use the wdmconfig command to modify the variables trace.filename, adapter.trace.enable, and adapter.trace.level, see 12.3.3, “IBM Tivoli Monitoring 5.1 TBSM Adapter configuration” on page 414.

14.3.5 Tivoli Business Systems Manager transport trace A trace is maintained at the gateway of the activities carried out by the Tivoli Business Systems Manager adapter when sending events or messages to the Tivoli Business Systems Manager CommonListener, as follows: Trace name

User-defined; default is dm.trc (when the trace is full it is renamed as filename.old, deleting any existing file with that name, and a new trace file is created).

Location

As defined in the wdmconfig configuration variable adapter.working.dir (default is $DBDIR/dmml).

Chapter 14. Troubleshooting

487

Configuration

To configure the trace, use the wdmconfig command to modify the variables trace.filename, transport.trace.enable, and adapter.transport.level, see 12.3.3, “IBM Tivoli Monitoring 5.1 TBSM Adapter configuration” on page 414, for more details.

14.3.6 Endpoint upcall traces A trace is maintained at the gateway of the upcall messages sent to the gateway from the endpoints. It contains details of the following: 򐂰 Endpoint registration upcalls 򐂰 Events and indications sent from the endpoint component 򐂰 Task upcalls It records all messages output by the process tmnt_gtw_eng, which receives the upcalls. The details are as follows: Process name

tmnt_gtw_eng, executable location $BINDIR/TME/Tmw2k

Trace name

trace_tmnt_gtw_engn.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

UNIX or Linux: /tmp/AMW/logs Windows: $DBDIR/tmp/AMW/logs

Configuration

To configure the log, use the command: wdmconfig –D gw.trace_level = wdmconfig –D gw.trace_size =

Note: In our environment, the process tmnt_gtw could not generate trace information on the file trace_tmnt_gtw_engn.log, even when the trace_level was configured to the higher trace level.

14.3.7 Request manager trace A trace is maintained at the gateway of the activities carried out when the request manager process is running. It records all messages output by the process tmnt_rm_eng. The details are as follows:

488

Process name

tmnt_trm_eng, executable location $BINDIR/TME/Tmw2k

Trace name

trace_tmnt_rm_engn.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

$DBDIR/AMW/logs

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Configuration

To configure the log, use the command: wdmconfig -m node –D request-manager.trace_level= 0 - 2 wdmconfig -m node –D request-manager.trace_size= value

After changing the trace level or size, the request manager should be restarted, to reflect the changes. The request manager should be stopped using the command: wdmmn -stop -m node -r

The request manager should be started using the command: wdmmn -start -m node -r

14.3.8 Profile core trace A log is maintained at the gateway of the activities carried out when the profile core engine is running. It records all messages output by the process tmw2k_profile_core. The details are as follows: Process name

tmw2k_profile_core, executable location $BINDIR/TME/Tmw2k

Trace name

trace_tmnt_profile_coren.log (where n is a number in the range 1–9; as each file becomes full the number is incremented, cycling back to 1 when file 9 is full)

Location

$DBDIR/AMW/logs

Configuration

To configure the log, use the command wdmconfig to change the following variables: wdmconfig -m node –D core.trace_level= 0 - 2 wdmconfig -m node –D core.trace_size=value

Table 14-1 summarizes the configuration of the available trace components. Table 14-1 Gateway trace components summary Component name

Process name

Location

Trace name

Heartbeat engine

tmnt_hb_eng

$DBDIR/AMW/logs

trace_tmnt_hb_en gn.log

Task engine

tmnt_task_eng

$DBDIR/AMW/logs

trace_tmnt_task_e ngn.log

Tivoli Business Systems Manager engine traces

tmnt_tbsm_eng / tmnt_tbsm_wrappe r

$DBDIR/AMW/logs

trace_tmnt_tbsm_e ngn.log / trace_tmnt_tbsm_ wrappern.log

Chapter 14. Troubleshooting

489

Component name

Process name

Location

Tivoli Business Systems Manager adapter/transport trace

Trace name

dm.trc

Endpoint upcall traces

tmnt_gtw_eng

/tmp/traces %DBDIR%/TMP/tr aces

trace_tmnt_gtw_en gn.log

Request manager trace

tmnt_rm_eng

$DBDIR/AMW/logs

trace_tmnt_rm_en gn.log

Profile core trace

tmw2k_profile_cor e

$DBDIR/AMW/logs

trace_tmnt_profile_ coren.log

14.4 Endpoint logs Different log files are available depending on the platform of the operating system. Logs and traces maintained at Windows endpoints are different from those at non-Windows endpoints.

14.4.1 Common endpoint logs This section reviews common endpoint logs.

Profile distribution endpoint logs The product uses lcfd.log as a profile distribution endpoint log. The IBM Tivoli Monitoring 5.1 distribution process records the MDist2 activities of the engine update, as follows: Process name

Tmw2k_ep

Log name

lcfd.log

Location

$LCF_DATDIR

Configuration

To configure the log you can edit the last.cfg, then change the parameter log_threshold to 3 or issue the following command: wadminep endpoint reexec_lcfd -D log_threshold=3

490

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

14.4.2 Windows endpoint logs The product maintains an endpoint engine log at Windows endpoints, and there are also logs maintained by WMI.

Windows endpoint engine log The main trace log generated by the IBM Tivoli Monitoring 5.1 engine at Windows endpoints records the activities of the endpoint engine, as follows: Process name

Tmw2k, executable location %LCF_DATDIIR%\LCFNEW\Tmw2k\bin Tmw2k_ep, executable location %LCF_DATDIIR%\cache\bin\w32-ix86\TME\Tmw2k

Trace name

Tmw2k.log (when the log is full the oldest 20 percent of messages are deleted)

Location

%LCF_DATDIR%/LCFNEW/Tmw2k

Configuration

To configure the trace, issue the command wdmtrceng from the server or ManagedNode, identifying the endpoint at which you want to configure the log. You can set any of the following parameters: Trace Tmw2k.log Trace level, default is 0, you can set to the following modes: 0: Errors 1: Warnings and errors 2: All steps of the monitoring process 3: Verbose output, all operations of the monitoring process Maximum file size, default is 5000000 (5.0 MB)

Note: We recommend not changing the trace level for performance reasons. The wdmtrceng command changes the following registry hives: HKEY_LOCAL_MACHINE -> SOFTWARE -> Tivoli -> Tmw -> DisplayThreshold and HKEY_LOCAL_MACHINE -> SOFTWARE -> Tivoli -> Tmw -> MaxFileLogSize. Each line in the log contains the following columns: 򐂰 Date

Chapter 14. Troubleshooting

491

򐂰 Trace level 򐂰 Component/classes – ActionManager: Executes corrective actions (built-in tasks) – Adapters: Sends events to TEC and TBSM – Analyzer: Collects and analyses the performance data – ComponentManager: Provides the default implementation for creation/profile push/start/stop/delete – Dispatcher: Responsable for forwarding the events or operations to the subscribed components (this is because the upcall is not thread safe; therefore, events or operations that need to perform upcall need to pass through the dispatcher) – EventAggregator: Receives indications from decision trees and processes those based on the occurrences and holes – EventCorrelator: Consolidates and correlates events generated between different resource models – InstallSharedDll: Installs the shared dll – Logger: Stores the data locally on the endpoint – MsgQueueManager – MofCompiler – TemporarySink: It is a pure virtual utility that provides a wrapping to WMI for facilitating the implementation of WMI event listeners. It also provides support for checking the correct behavior, ensuring that WMI does not unload the listeners – TMNT_Upcall – TMWMethodProv: The WMI method provider; responsible for implementing the CIM class methods – TMWService: An object that allows the scripts implementing the best practices to configure and set the default values for parameters and thresholds to specify the data sources that will be used; also useful to use some IBM Tivoli Monitoring 5.1 services like event sending, tracing, running monitors and scripts, and data logging; basically, a way the scripts interface the engine and the underlying CIM implementation – WbemConnector 򐂰 Thread ID 򐂰 Message

492

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

After changing the trace settings, the endpoint monitoring engine should be restarted to reflect the changes. The monitoring engine should be stopped using the command: wdmcmd -stop -e endpoint_label

The monitoring engine should be started using the command: wdmcmd -restart -e endpoint_label

Tip: When issuing the wdmtrceng command on a Windows target, you must use the forward slash (/) in the directory path. For example: wdmtrceng -e itsotiv1 c:/progra~1/tivoli/lcf/dat /1/LCFNEW/ Tmw2k/Tmw2k.log 3 800000

WMI You can use the Workbench to see the functionality of the Windows resource model, then use the CIM Studio to browse the CIM repository. CIM Studio provides the command wbemdump, which is a CLI tool that can query the CIM repository. These kinds of tools can be useful to determine the efficiency of the probes. For more information about Workbench and CIM Studio, check Chapter 10, “Workbench” on page 245. The WMI log files record the activities of WMI in collecting the data required by the resource models. The WMI log files are located in the directory %SystemRoot%/system32/wbem/logs. For details see the WMI documentation available in Windows Help, or you can see also a WMI Tutorial available at: http://www.microsoft.com/downloads/release.asp?releaseid=12570

14.4.3 Non-Windows endpoints logs The product maintains four logs at the endpoint: 򐂰 򐂰 򐂰 򐂰

Endpoint engine update log Endpoint engine log and trace Endpoint native trace Endpoint JMX log

Chapter 14. Troubleshooting

493

Note: The endpoint engine update log, endpoint engine log, endpoint native trace, and endpoint JMX log use the same configuration file for tracing. So changing one trace level of one log affects the other logs’ settings. The command wdmtrceng changes the files $LCF_DATDIR/LCFNEW/Tmw2k/Unix/data/log_level and $LCF_DATDIR/LCFNEW/Tmw2k/Unix/data/log_size. In our environment, to change all trace levels to 3 and the trace size to 8000000, we issued the following command: wdmtrceng -e itsodev3 /Tivoli/lcf/dat/1/LCFNEW/Tmw2k/Unix/data/log_level 3 8000000

Endpoint engine update log This log maintains details of the activities of the engine update process, which is the process that launches and controls the endpoint engine, as follows: Process name

tmw2k_ep; this process is started and finished very fast, so it is not possible to see the running process.

Log name

trace_dmxeu.log (when the log is full it is renamed as dmxeu.old, deleting any existing file with that name, and a new log file is created).

Location

$LCFDATDIR/LCFNEW/AMW/logs.

Configuration

To configure the trace, issue the command wdmtrceng from the server or ManagedNode, identifying the endpoint at which you want to configure the log. You can set any of the following parameters: Trace trace_dmxeu.log Trace level; default is 0; you can set to the following modes: 0: Errors 1: Warnings and errors - Trace level MID 2: All steps of the monitoring - Trace level MAX 3: Verbose output, all operations of the monitoring process - Trace level MAX and OTHER

494

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Endpoint engine log and trace This log maintains details of the activities of the engine, which is the process that runs the resource models and sends events and indications to the gateway, as follows: Process name

java com.tivoli.dmunix.ep.agent.Main

Log name

msg_dmxengine.log (when the log is full it is renamed as dmxengine.old, deleting any existing file with that name, and a new log file is created)

Trace name

trace_dmxengine.log (when the log is full it is renamed as dmxengine.old, deleting any existing file with that name, and a new log file is created)

Location

$LCF_DATDIR/LCFNEW/AMW/logs

Configuration

To configure the trace, issue the command wdmtrceng from the server or ManagedNode, identifying the endpoint at which you want to configure the log. You can set any of the following parameters: Trace trace_dmxeu.log Trace level; default is 0; you can set to the following modes: 0: Errors 1: Warnings and errors - Trace level MID 2: All steps of the monitoring - Trace level MAX 3: Verbose output, all operations of the monitoring process - Trace level MAX and OTHER

Endpoint native trace This log maintains details of the activities of the native processes which obtain the resource information required by the resource models, as follows: Process name

java com.tivoli.dmunix.ep.agent.Main

Log name

trace_dmxntv.log (when the log is full it is renamed as dmxntv.old, deleting any existing file with that name, and a new log file is created)

Location

$LCFDATDIR/LCFNEW/AMW/logs

Chapter 14. Troubleshooting

495

Configuration

To configure the trace, issue the command wdmtrceng from the server or ManagedNode, identifying the endpoint at which you want to configure the log. You can set any of the following parameters: Trace trace_dmxeu.log Trace level; default is 0; you can set this to the following modes: 2: All steps of the monitoring - Trace level MAX 3: Verbose output, all operations of the monitoring process - Trace level MAX and OTHER

Endpoint JMX log This log maintains details of the activities of the JMX process, which is a Tivoli implementation of Java Management Extension. It is only written when the trace level is set to 3. The details are as follows: Process name

Tmx4j

Log name

Tmx4j_1.log (when the log is full it is renamed as Tmnx4j_2.log, deleting any existing file with that name, and a new log file is created)

Location

$LCF_DATDIR/LCFNEW/Tmw2k/UNIX

Configuration

To configure the log, issue the command wdmtrceng from the server or ManagedNode, identifying the endpoint at which you want to configure the log. You should note that this command maintains a common configuration for all logs at a non-Windows endpoint. You can set any of the following parameters: Trace level: 2 (verbose)

Note: For the log’s endpoint native trace and endpoint JMX, we only noticed some information being registered in the log after setting the log_level to a minimum value of 2.

14.4.4 Web Health Console logs and traces The Web Health Console has a facility for both standard message logging and advanced debug tracing. Message logging and minimum level debug tracing are always on and writing to their own files. These files can be found under the location /usr/Tivoli/AMW/logs.

496

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Modifying Web Health Console tracing parameters Tracing can be adjusted by modifying the tracing parameters for the Web Health Console application. Edit the file WHC_INSTALL_DIR\installedApps\dm.ear\dm.war WEB-INF\classes\com\ibm \dm\web\util\PDLog.properties. You can change the line: tmeLogger.trc.level=DEBUG_MIN tmeLogger.trc.level=DEBUG_MID tmeLogger.trc.level=DEBUG_MAX

depending on how much tracing you want, MIN, MID or MAX. MID provides a good amount of Web Health Console operation, MAX provides a great deal of detailed internal operation. You can also adjust the following lines to change the number of trace files written and the max size of the files before it roles over to a new file. file.maxFiles=3 file.maxFileSize=1024

Once these changes are made, you should stop and start the WebSphere Application Server to enable the changes, with the following commands: 򐂰 UNIX export DISPLAY=machineName:0.0 WHC_INSTALL_DIR/bin/stopServer.sh WHC_INSTALL_DIR/bin/startServer.sh

򐂰 Windows WHC_INSTALL_DIR/bin/stopServer.bat WHC_INSTALL_DIR/bin/startServer.bat

Note: The Web Health Console will run slower while in MID or MAX tracing. This should be turned back to MIN as soon as possible.

WebSphere tracing You can set also WebSphere Application Server tracing. Then you will be able to see all the requests and operations in the ApplicationServer perspective. To enable WebSphere tracing, you do the following steps: $WHC_INSTALL_DIR/bin/DrAdmin.sh -serverPort 7000 -setRingBufferSize 2048 $WHC_INSTALL_DIR/bin/DrAdmin.sh -serverPort 7000 -setTrace "com.ibm.*=all=enabled"

After these commands, the trace log can be located under $WHC_HOME/logs/default_server_stdout.log and $WHC_HOME/logs/default_server_stderr.log.

Chapter 14. Troubleshooting

497

To disable the WebSphere tracing, you can run the command: $WHC_INSTALL_DIR/bin/DrAdmin.sh -serverPort 7000 -setTrace "com.ibm.*=all=disabled"

For further reference to WebSphere troubleshooting, please refer to IBM WebSphere Version 4.0 Advanced Edition Handbook, SG24-6176.

14.5 Tools In this section we describe some tools provided with the product and other available tools.

14.5.1 Tool to generate XML file The formatter program creates an XML-based file from the log or trace generated by IBM Tivoli Monitoring 5.1. It is located on the IBM Tivoli Monitoring 5.1 Tools CD in the directory LogToXML. It accepts three parameters: 򐂰 The first parameter defines whether the product is dealing with a log file or a message file. 򐂰 The second parameter is the name of the source file (either a log or a message file). 򐂰 The third parameter is the name of the file to be created in XML. Here is an example: prepareLog LOG trace_x.log trace_x.xml

Note: Before running the prepare log program, the Java Virtual Machine 1.3.0 path must be set.

14.5.2 Autotrace Autotrace is a process tracing software from The Kernel Group Inc. (TKG), and is available on Solaris, HP-UX, Windows, and AIX platforms. It is used to collect information at an endpoint that is stored in a configurable memory buffer. You choose when to write a snapshot of the buffer to a file, and you then send the file to Tivoli Customer Support for analysis. The information written to the trace file consists of the input and output parameters for each process call.

498

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Autotrace consists of two elements: 򐂰 A trace collector enabled and controlled by you at the endpoint and Tivoli management region server 򐂰 A trace analyzer operated by the Tivoli Customer Support staff at the Tivoli Customer Support For installation and use of Autotrace see Example 14-5 on page 517, and refer to IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569.

14.5.3 Serviceability tasks IBM Tivoli Monitoring 5.1 provides three serviceability tasks, as described in the following sections.

DMCollectEpLog This task collects (in a tar file created at the endpoint) all the endpoint logs and information about the size and dates of the binaries, as well as the current and universal time the logs were created. The task accepts the name of the tar file as an argument. The tar file can be found at the endpoint directory $LCF_DATDIR. For UNIX/Linux platforms, the following files are collected: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

$LCF_DATDIR/lcfd.log $LCF_DATDIR/lcfd.bk $LCF_DATDIR/last.cfg $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Tmx4j1.log $LCF_DATDIR/LCFNEW/Tmw2k/Unix/Tmx4j2.log $LCF_DATDIR/LCFNEW/AMW/logs/trace_xxxxx.log $LCF_DATDIR/LCFNEW/AMW/logs/msg_xxxxx.log $LCF_DATDIR/LCFNEW/Tmw2k/Unix/data/dmxout.log (this is a file that traces errors at Java engine startup)

For Windows platforms, the following files are collected: 򐂰 򐂰 򐂰 򐂰

%LCF_DATDIR%/lcfd.log %LCF_DATDIR%/lcfd.bk %LCF_DATDIR%/last.cfg %LCF_DATDIR%/LCFNEW/Tmw2k/Unix/Tmw2k.log

Any core dumps from the engine are not included in the tar file to avoid impacting the task performance. Core dumps can be found in the directory $LCF_DATDIR \LCFNEW\Tmw2k\Unix.

Chapter 14. Troubleshooting

499

DMCollectMnLog This task collects (in a tar file created at the ManagedNode in the $DBDIR directory) all the ManagedNode logs and traces, including event logs for Windows platforms. The task accepts the name of the tar file as an argument. The tar file can be find at the ManagedNode directory $DBDIR. For UNIX/Linux platforms, the following files are collected: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

$DBDIR/oservlog $DBDIR/gatelog /tmp/traces/trace_tnmt_gtw_engn.log $DBDIR/AMW/logs/trace_xxxx.log $DBDIR/ $DBDIR/

For Windows platforms, the following files are collected: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰

%DBDIR%/oservlog %DBDIR%/gatelog %DBDIR%/tmp/traces/trace_tnmt_gtw_engn.log %DBDIR%/AMW/logs/trace_xxxx.log %DBDIR%/ %DBDIR%/

DMCollectEpEnv This task collects information about the environment at the endpoint. The data collected is written to a file using the Execute Task dialog (Save to File option). This task does not accept arguments. For UNIX/Linux platforms, the following information is collected: 򐂰 򐂰 򐂰 򐂰 򐂰

Operating system version Disk space statistics and file system installation at the endpoint Memory statistics (available and used) Environment variable settings List of system patches installed

For Windows platforms, the output from the winmsd command is collected. 򐂰 For Windows 2000 the report is created in %LCF_DATDIR%/winmsdreport.txt. 򐂰 For Windows NT the report is created in %LCF_DATDIR%/.txt. Example 14-1 DMCollectEpEnv task output tividc11:/>wruntask -t DMCollectEpEnv -l "Tivoli Distributed Monitoring (Advanced Edition) Tasks" -h vmlinux5 ############################################################################

500

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

Task Name: DMCollectEpEnv Task Endpoint: vmlinux5 (Endpoint) Return Code: 0 ------Standard Output--------> Platfom Linux ---> Operating System level 2.2.16 ---> Disk space and inode information for file system /opt/tivoli/dat/1 Filesystem 1k-blocks Used Available Use% Mounted on /dev/dasdc1 708636 146452 526188 22% /opt ---> Statistics for memory Amount of idle memory (kB): 8272 Amount of memory used as buffers (kB): 35736 ---> Environment variables BASH=/bin/sh BASH_VERSINFO=([0]="2" [1]="04" [2]="0" [3]="1" [4]="release" [5]="s390-suse-linux") BASH_VERSION='2.04.0(1)-release' COLORTERM=1 DIRSTACK=() ENDPOINT='vmlinux5 (Endpoint)' ENDPOINT_OID=1931340152.16.517+#TMF_Endpoint::Endpoint# EUID=0 FROM_HEADER=vmlinux5.itso.ibm.com GNOMEDIR=/opt/gnome GROUPS=() HISTCONTROL=ignoredups HOME=/root HOSTNAME=vmlinux5 HOSTTYPE=s390 IFS='' INFODIR=/usr/local/info:/usr/share/info:/usr/info INFOPATH=/usr/local/info:/usr/share/info:/usr/info INTERP=linux-s390 KDEDIR=/opt/kde LCFROOT=/opt/tivoli LCF_BINDIR=/opt/tivoli/bin/linux-s390/mrt LCF_CACHEDIR=/opt/tivoli/dat/1/cache LCF_DATDIR=/opt/tivoli/dat/1 LCF_LIBDIR=/opt/tivoli/lib/linux-s390 LCF_TEMPDIR=/tmp/ LC_CTYPE=en_US LD_LIBRARY_PATH=/opt/tivoli/dat/1/cache/lib/linux-s390:/opt/tivoli/dat/1:/opt/t ivoli/lib/linux-s390:/usr/lib

Chapter 14. Troubleshooting

501

LESS='-M -S -I' LESSCHARSET=latin1 LESSKEY=/etc/lesskey.bin LESSOPEN='|lesspipe.sh %s' LOGNAME=root LS_COLORS='no=00:fi=00:di=01;34:ln=01:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01 :ex=01;31:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.tar=00; 31:*.tgz=00;31:*.rpm=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z= 00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xb m=01;35:*.xpm=01;35:*.tif=01;35:*.png=01;35:' LS_OPTIONS='-a -N --color=tty -T 0' MACHTYPE=s390-suse-linux MAIL=/var/spool/mail/root MANPATH=/usr/local/man:/usr/share/man:/usr/man:/usr/X11R6/man:/usr/openwin/man MINICOM='-c on' NLSPATH=/opt/tivoli/generic/msg_cat/%L/%N.cat:/opt/tivoli/generic/msg_cat/%l/%N .cat:/opt/tivoli/generic/msg_cat/C/%N.cat NNTPSERVER=news OPTERR=1 OPTIND=1 OSTYPE=linux PAGER=less PATH=/opt/tivoli/bin/linux-s390/tools:/bin:/usr/bin:/usr/ucb:/usr/sbin:/sbin:/u sr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/ usr/games/bin:/usr/games:/opt/gnome/bin:/opt/kde/bin PIPESTATUS=([0]="0") POVRAYOPT=-l/usr/lib/povray/include PPID=23900 [email protected] PRINTER=lp PS4='+ ' PWD=/opt/tivoli/dat/1 QTDIR=/usr/lib/qt RC_LANG=en_US RC_LC_COLLATE=POSIX REMOTEHOST=tividc11 SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=2 SUSE_DOC_HOST=localhost TERM=vt100 TEXINPUTS=':~/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX' TISDIR=/opt/tivoli/dat/1 UID=0 USER=root WINDOWMANAGER=/usr/X11R6/bin/kde XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB XNLSPATH=/usr/X11R6/lib/X11/nls_='---> Environment variables' curdir=/opt/tivoli/dat/1

502

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

freemem=8272 ignoreeof=0 no_proxy=localhost os=Linux usedmem=35736 ---> List of the patches installed ------Standard Error Output-----############################################################################

14.6 Known problems and their resolutions IBM Tivoli Monitoring 5.1 has some known issues that are reported in Appendix C of the IBM Tivoli Monitoring User’s Guide Version 5.1, SH19-4569 .

14.7 Problem determination In this section we describe, with examples, the necessary steps to troubleshoot a profile distribution and running engine on a Windows/UNIX endpoint.

Windows engine In this example we use the following components: 򐂰 Endpoint: itsotiv1 򐂰 Tmw2kProfile: tmw2k.windows.pf 򐂰 Resource model: TMW_ParamServices – Parameter - Schedule – Cycle Time = 60s – Indication Services Stopped Service • Ocurrences=6, holes=0 • Send TEC event • ActionList = dm_mn_send_notice, Restart Service

Windows endpoint profile distribution Before starting the distribution, we changed the debug level of the following components: TMR 򐂰 MDist 2 wmdist -D 9

Chapter 14. Troubleshooting

503

򐂰 Gateway wgateway gateway set_debug_level 6 򐂰 Endpoint last.cfg - log_threshold=3 The first profile distribution compiles the MOF files to create the associations in the CIM repository, as shown in Example 14-2. Example 14-2 Profile distribution in Windows environment # Distributing the Tmw2kProfile tividc11:/>wdmdistrib -p tmw2k.windows.pf itsotiv1 AMW0162I - Operation successfully submitted. Distribution ID is 1931340152.249

# Checking distribution status tividc11:/>wmdist -l -i 1931340152.249 Name Distribution ID Targets Completed Successful Failed tmw2k.windows.pf(install) 1931340152.249 1 1(100%) 1(100%)

# Checking log TMR - $DBDIR/distmgr.log. Verifying distribution ID # 1931340152.249 2002/05/09 11:24:18 +06: Registering new distribution, ID = 1931340152.249 2002/05/09 11:24:20 +06: DBStatusUpdate started for ID= 2002/05/09 11:24:20 +06: retrieving repeater nodes via nodeElements() for distri bution 2002/05/09 11:24:21 +06: DBStatusUpdate completed for ID= 2002/05/09 11:24:22 +06: DBStatusUpdate started for ID= 2002/05/09 11:24:23 +06: retrieving repeater nodes via nodeElements() for distribution 2002/05/09 11:24:23 +06: retrieving repeater nodes via nodeElements() for distribution 2002/05/09 11:24:24 +06: DBStatusUpdate completed for ID= 2002/05/09 11:24:25 +06: DBStatusUpdate started for ID= 2002/05/09 11:24:25 +06: retrieving repeater nodes via nodeElements() for distribution 2002/05/09 11:24:26 +06: DBStatusUpdate completed for ID= 2002/05/09 11:24:33 +06: DBStatusUpdate started for ID= 2002/05/09 11:24:33 +06: retrieving target nodes via nodeElements() for distribution 2002/05/09 11:24:34 +06: retrieving repeater nodes via nodeElements() for distribution 2002/05/09 11:24:35 +06: DBStatusUpdate completed for ID=

504

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

# Checking log TMR - $DBDIR/AMW/logs/msg_tmw2k.windows.pf.log 1020961460000Thu May 9 11:24:20 2002GMTAMWcoremanaged_node21 028AMW --> 1931340152.249 - tmw2k.windows.pf(install) - AMW0162I Operation successfully submitted. Distribution ID is '1931340152.249'. 1020961475000Thu May 9 11:24:35 2002 GMTAMW coreitsotiv1 12526AMW MAX../../../../src/objects/TMNTUpcall/platform/NTTask_Engine_meth_imp.cpp< F >t_imp_DMMiddleLayer_Processor_TMNT_Task_start536992792MAX../../../../src/objects/TMNTUpcall/platform/NTTask_Engine_meth_imp.cpp< F>t_imp_DMMiddleLayer_Processor_TMNT_Task_terminate537003944return data = 0None. 1020963214000Thu May

514

9 16:53:34 2002GMTAMWtasktividc1111878
MAX../../../../src/objects/TMNTUpcall/platform/NTTask_Engine_meth_imp.cpp< F>t_imp_DMMiddleLayer_Processor_TMNT_Task_terminate537003944wdmdistrib -p tmw2k.unix.pf -J /backup/Tools/Jre itsodev3 AMW0162I - Operation successfully submitted. Distribution ID is 1931340152.263 tividc11:/>wmdist -l -i 1931340152.263 Name Distribution ID Targets Completed Successful Failed tmw2k.unix.pf(install) 1931340152.263 1 1(100%) 1(100%) 0( 0%). # Checking log TMR - $DBDIR/distmgr.log. Verifying distribution ID # 1931340152.263

Chapter 14. Troubleshooting

515

2002/05/14 11:40:30 : Registering new distribution, ID = 1931340152.263 2002/05/14 11:40:32 : DBStatusUpdate started for ID= 2002/05/14 11:40:32 : retrieving repeater nodes via nodeElements() for distribution 2002/05/14 11:40:33 : DBStatusUpdate completed for ID= 2002/05/14 11:43:01 : DBStatusUpdate started for ID= 2002/05/14 11:43:01 : retrieving target nodes via nodeElements() for distribution 2002/05/14 11:43:02 : retrieving repeater nodes via nodeElements() for distribution 2002/05/14 11:43:02 : DBStatusUpdate completed for ID= # Checking log TMR - $DBDIR/AMW/logs/msg_tmw2k.unix.pf.log 1021376431000Tue May 14 11:40:31 2002 GMTAMWcoremanaged_node6510AMW --> 1931340152.263 tmw2k.unix.pf(install) - AMW0162I - Operation successfully submitted. Distribution ID is '1931340152.263'. 1021376583000Tue May 14 11:43:03 2002 GMTAMW core itsodev3 12744AMW >>> ENTRY May 14 11:31:41 Q mdist2_operation_receiver init_passing_data >>>> ENTRY May 14 11:31:41 Q mdist2_operation_receiver load_status_file >>>> ENTRY

516

IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring

May 14 11:31:41 Q mdist2_operation_receiver Status file '/Tivoli/lcf/dat/1/LCFNEW/Tmw2k/dist.st' does not exist. New distribution!!! May 14 11:31:41 Q mdist2_operation_receiver return data = 0 May 14 11:31:41 Q mdist2_operation_receiver load_status_file