In the first part of this blog series, you were introduced to three critical elements involved in expediting your service resolution: documentation, preparation and conversation. In case you missed the documentation tips, you can read about them in Part 1.
Log Collection Preparation
After collecting the server’s configuration documentation, it would be good to prepare a collection of log data. There are some logs that are specific to the type of OS that is running on the failing logical partition (LPAR) and I’ll list some of those a little later. There are also logs that can be collected no matter what type of OS is in operation on your server. Use the following tips to gather this content:
Launch ASM Interface using Opera (browser) for Log Gathering
From the HMC, perform the “Launch ASM Interface” steps listed in the IBM Knowledge Center and review the Error/Event Logs found under the System Aids option. There is a way to copy these listed logs for transmission to another location, but the steps are too lengthy for this blog entry. Suffice to say, even just viewing these logs can be useful in identifying a problem source related to the overall server.
Useful HMC Commands for Log Gathering:
| Command | Example of Syntax or Use of Flags | Definition |
|---|---|---|
| tail | Reads the final few lines of any text given to it as an input and writes them to standard output. | |
| tail –f /var/log/messages | Concurrently look at the messages being logged on the HMC. | |
| tail –n 50 –f /var/log/messages | ||
| lssvcevents | Check from the HMC command line for new service events. | |
| lssvcevents –t hardware –i 10 | The –i 10 flag looks for new errors in the last 10 minutes. | |
| pedbg | Provides debug tools for Product Engineer/Support Personnel. This command requires PE authority to run, i.e., only hscpe user can access this command. | |
| pedbg –c –q 1 | Pull the network connection log for the HMC. |
As mentioned earlier, there are logs that can be collected for the various OSes that can be installed on the POWER platform. The following are logs that can/should be collected for those OSes:
Useful AIX Commands for Log Gathering:
| Command | Example of Syntax or Use of Flags | Definition |
|---|---|---|
| errpt | Generates a report of logged errors. | |
| errpt | pg | Generates a list of reported errors by page. | |
| errpt –a | List error log in detailed format. | |
| errpt –aN hdisk# | List detailed report by specified resource name. | |
| errpt –s mmddhhmmyy | List errors posted since date entered (month, day, hour, minute, year). | |
| snap | Gathers system configuration information. | |
| snap –a | Gathers complete system configuration information and writes it to /tmp/ibmsupt directory and subdirectories. | |
| snap –gbc | Gathers most commonly selected information and writes it to the /tmp/ibmsupt/snap.pax.Z file for transfer to the support center or to the decode website for decoding. |
Useful SuSE Linux Enterprise Server (SLES) Edition and Red Hat Enterprise Linux (RHEL) Linux Commands (Available if IBM Linux Toolkit is installed) for Log Gathering:
| Command | Example of Syntax or Use of Flags | Definition |
|---|---|---|
| servicelog | Query the service log. | |
| servicelog –h | View options available when running the servicelog command. | |
| servicelog –serviceable=all | View all serviceable events in the service log. | |
| log_repair_action | Indicate when a repair action has taken place in the system. |
If you are running on an IBM i OS LPAR, you could review the Product Activity Log (PAL), the Service Action Log (SAL) and the problem log for additional insights.
Along with the hardware logs, there are also logs that can be collected for the various applications that can be run on the POWER server platform. The method of collection for these logs can vary, which is why it’s important to ask the support center representative to be specific in the logs that are being requested. Engage in an open conversation where (if you’ve got the time to pull additional logs) you query the support center representative about the full breadth of logs that may be required for the analysis of your issue. This allows you to minimize the preparation and attempts needed for any log traps to be set and collection to take place (in short, attempt to get all the necessary logs on the first pass of log collection). This leads us to the next tip.
Questions to Ask During Conversation
Engaging in an open conversation with your support center representative is a good way to ensure that the correct information related to an issue is collected during the first pass of log gathering. During my years in the service field, I’ve held several different service-related positions. In one of my roles I assisted clients with maneuvering through the maze of the various departments and levels of support that may be required in order to provide a resolution to a given issue. What I realized is that it seemed normal for a support center representative to request one type of log from the POWER Server only to get that information and then realize that additional log information may be necessary for problem resolution.
One thing that can be done during your initial contact or at least during your follow-up contact with a vendor support center is to repeatedly ask the support center representative if there is any additional log gathering that would be helpful for resolution. Let them know that you may not have the ability to set up multiple, separate, system windows for data collection. You may only have one window of opportunity available to collect all logs that might be needed for problem resolution. During this type of conversation don’t be shy. Remember, the support center is there to help resolve your issue, and all you’re wanting to accomplish is to reach the resolution in the shortest amount of time and get your server back to a 100 percent operational state so that all service level agreements (SLAs) are, once again, met.
Having the above information ready when interacting with your service provider will only serve to speed the resolution of the problem. Just remember documentation, preparation and conversation!
About the Author
Martin Clayton has been working in the IT field for 35 years, including 28 years within IBM’s service division. He attended Wayne State University where he studied electrical and computer engineering. In addition to his work as a field engineer supporting the IBM mainframe, Power Systems, storage and System x platforms, Martin has also been an instructor and developer.
Related Learning Resources:
PowerVM Virtualization Essentials
Courses
Linux on Power Performance (LX52G)
Power Systems for AIX III: Advanced Administration and Problem Determination (AN15G)
Power Systems for AIX IV: Performance Management (AN51G)
Tips and Tricks for Expediting Service Resolution in Your POWER Server Environment Series
- Tips and Tricks for Expediting Service Resolution in Your POWER Server Environment: Part 1
- Tips and Tricks for Expediting Service Resolution in Your POWER Server Environment: Part 2