Upgrading Physical Machine Firmware

Update BIOS, BMC, and storage firmware whenever upgrading Avance or performing Physical Machines (PM) maintenance: For instructions:

1. In the Avance Management Console, click Physical Machines.
2. Click Help and follow the instructions for Upgrading Firmware.

To view current firmware versions:

1. In the Console, select a PM.
2. Click the Details tab.

Recovering a Physical Machine

If a PM becomes unreachable or repeatedly crashes until Avance removes it from service, use the Recover button to re-image the PM and return it to service.

1. In the Avance Management Console, follow the instructions in Recovering Physical Machines in the online help.
2. When prompted, select Yes (Recover w/Reformat).

If the problem persists, run your hardware vendor’s diagnostic tools. If no hardware issue is detected, contact your Avance service provider.

Repairing a Physical Machine

These hardware faults let you gracefully shut down a running PM for repair

• Memory DIMMs exhibiting correctable (non-fatal) errors
• Fans
• NICs

These can be repaired with the PM running (see “Adding/Upgrading disks with the PM Running“):

• Physical disks
• Power supplies
• Business Network Cables

In all cases, you can repair the PM without interrupting the virtual machines (VMs).

Restrictions

• You cannot change the motherboard or RAID controllers.
• You cannot change existing RAID volume types or order.

Shutting Down a PM

1. In the Avance Management Console, click Physical Machines.
2. Select the PM to repair. Click Work On.
3. When the PM displays running (in maintenance), click Shutdown. Click OK.
4. Repair the PM as required.
5. Reconnect all network cables.
6. In the Avance Management Console, select the repaired PM. Click Power On. Click OK. Avance powers on and boots the PM. When the PM is running, Avance begins synchronizing the PM’s storage ( appears).
7. On the Networks page, click the Fix button if it is highlighted. This might occur if any network cables were moved on the repaired PM.
8. Select the repaired PM on the Physical Machines page. Click Finalize. Click OK.
9. When synchronization ends ( disappears), continue with normal operation.

---

To avoid data loss, do not power down the primary PM while the disks are synchronizing.

---

Adding Resources to a Physical Machine

You can add these resources to a PM without interrupting applications (VMs):

• Higher speed or more CPUs
• More memory
• New or upgraded (larger/faster) disks
• RAID volumes
• Ethernet ports

For instructions, see “Upgrading or Adding Components,” below. In addition, you can add a single-disk RAID0 or non-RAID drives, or upgrade physical disks, without shutting down PMs. See “Adding/Upgrading disks“, below.

Restrictions

• All upgrades must follow the Avance Compatibility Matrix for your Avance software version.
• You cannot change the motherboard or RAID controllers.
• You cannot change existing RAID volume types or order, but you can add new RAID volumes/disks.
• Downgrades are not supported.

Adding/Upgrading disks with the PM Running

When adding/upgrading physical disks, plug each disk pair into the same slot number on both PMs. Paired disks are recommended to be the same size and speed.

1. In the Avance Management Console, click Physical Machines.
2. Select the secondary PM (not primary). Click Work On.
3. If upgrading a disk:Remove the old disk.After a moment, Avance recognizes the disk is gone. If the PM reboots, wait until it completes.
4. Insert the new disk.Avance automatically recognizes the new disk and brings it into service, unless the disk contains data. In this case, Avance marks the disk as foreign and leaves it out of service.
   

   ---

   Activating a disk marked foreign erases all its data.

   ---

5. If the disk is marked foreign: make sure the data is unneeded. Click Activate Diskto erase and bring it into service.When the disk is in service, Avance begins synchronizing the PM’s storage ( appears).
6. Click Finalize. Wait until synchronization is complete ( disappears).
7. Repeat for the primary PM.

Upgrading or Adding Components

1. In the Avance Management Console, click Physical Machines.
2. Select the secondary PM (not primary). Click Work On.
3. When the PM displays running (in maintenance), click Shutdown. Click OK.
4. Upgrade the PM as required. If adding disks to multi-disk RAID arrays, see “Adding disks to an existing or new multi-disk RAID array,” below.
5. Reconnect all network cables. Do not add cables to any new network ports at this point.
6. In the Avance Management Console, select the PM. Click Power On. Click OK.Avance powers on and boots the PM. When the PM is running,Avance begins synchronizing the PM’s storage ( appears).
7. On the Networks page, click the Fix button if it is highlighted. This might occur if any network cables were moved on the upgraded PM.
8. Select the repaired PM on the Physical Machines page. Click Finalize. Click OK.
9. Wait until synchronization is complete ( disappears).
   

   ---

   To avoid data loss, do not power down the primary PM while the disks are synchronizing.

   ---

10. Select the primary PM. Click Work On. Avance then migrates VMs to the upgraded PM, and makes that PM primary.
11. Repeat steps 3 through 9 for the PM in maintenance.
12. If you added NIC ports, see “Connecting Additional Business Networks” on page 10.

Adding disks to an existing or new multi-disk RAID array When adding new physical disks to both PMs, plug each disk pair into the same slot number on both PMs. Paired disks are recommended to be the same size and speed.

1. Insert the new disks in the PM, starting at the lowest available slot number. Do not skip slots.
2. Attach a keyboard and monitor to the PM.
3. Reboot the PM.
4. When prompted, enter the RAID BIOS configuration utility:
   

   Server type	Press
   Dell	Ctrl+R
   HP	F8
   IBM, ICO, Intel, Lynx, Primeline, Seneca, Tarox and Wortmann	Ctrl+H

5. Configure the new disks to an existing or new RAID configuration. See your server documentation for instructions. Save the configuration.
6. Power off the PM.
7. Continue with step 5 of “Upgrading or adding other components,” above.

Adding power supplies or fans to Intel server board based systems If additional power supplies or fans are inserted after firmware and FRUSDR flashing on Intel server board based systems then the the Field Replaceable Unit/Sensor Data Record (FRSDR) on the server must also be updated in order for the power and fan sensors to be monitored by the BMC. Refer to the document “How to Update the FRUSDR for Optimum Server Performance” on the Intel website for details. If this is not done then failures in these components might not be detected.

Upgrading to New Physical Machines

You can upgrade to new PMs having compatible CPUs and chipsets without interrupting applications (VMs). Avance warns you if the new PM has incompatible CPUs or chipsets, and if so prompts you to shut down the VMs before the upgrade completes.

Restrictions

• All upgrades must follow the Avance Compatibility Matrix for your Avance software version. If the PM is supported in a newer Avance release, you must upgrade the Avance software prior to upgrading the PM.
• New PMs must include the same RAID volume types and order as the original PM, and can have additional RAID volumes/disks.
• New PMs must have processors from the same processor family in order to support live migration. If the PMs are from different processor families then the VMs must be stopped in order for them to migrate from one PM to the other.
• New/upgraded PMs must not have fewer physical resources than the original PMs:
  • Number of processor cores
  • Number of drives. Also, drives must be at least as large as the corresponding drives in the original PM, and in the same slot positions.
  • Total memory
  • Total network ports. Each port must support at least the speed of the existing ports. All add-on NICs within a particular PM must have the same vendor/model number.

  Although Avance warns you if a new PM has incompatible or insufficient resources, it’s more efficient to correct issues beforehand.

Procedure

1. Upgrade Avance software if required to support the new PM. Refer to the Avance Release Notes and the help on the Avance Upgrade Kits page of the Avance Management Console.
2. Upgrade the secondary PM (not primary). See “Appendix A: Replacing a Physical Machine” .
3. Repeat for the primary PM. Avance then migrates the VMs to the other PM.
4. If you added additional NIC ports, see “Connecting Additional Business Networks“.

Replacing Physical Machines or Motherboards

Several classes of hardware faults can hang or crash a PM, such as:

• Motherboard or mid-plane failures
• CPU failures
• Storage controller failures

If this occurs, repair using the Avance PM Replace function. This does the following:

1. Deletes the defective PM from the Avance unit’s database.
2. Waits for repaired/replaced PM to power on.
3. Images the repaired PM and synchronizes its storage from the running primary PM (because the hardware failure may have corrupted the PM’s storage).

Restrictions

• These components must be the same on the replacement PM as on the original:
  • Vendor and model number
  • Compatible CPUs
  • Storage and network controllers (models/versions)
• Any RAID volumes (type and order) must be configured identically.
• The replacement PM cannot have fewer physical resources than the original PM:
  • Number of processor cores
  • Number of drives. Also, drives must be at least as large as the corresponding drives in the original PM, and in the same slot positions.
  • Total memory
  • Total network ports. Each port must support at least the speed of the existing ports. All add-on NICs within a particular PM must have the same vendor/model number.

  Although Avance warns you if a new PM has incompatible or insufficient resources, it’s more efficient to correct issues beforehand. For instructions, see “Appendix A: Replacing a Physical Machine” on page 8.

Appendix A: Replacing a Physical Machine

1. Prepare the new PM by following steps 2 through 5 (“Configure Networks” through “Configure BIOS”) in the Avance Installation Guide.
2. In the Avance Management Console, click Physical Machines.
3. Select the PM to be replaced. Click Work On.
4. When the PM shows running (in maintenance), click Replace. Avance deletes the PM from the Avance unit’s database, then prompts you to replace the PM.
5. Press to power off the old PM.
6. Disconnect and remove the existing PM.
7. Install the new PM.
8. Reconnect only the private network (GB1 port for Dell servers, port 1 for others).
9. Reconnect any 10G networks.
10. Press  to power on the new PM. To monitor the process of bringing the new PM into service, click Physical Machines.After about 30 minutes, the new PM is imaged and shows running (in Maintenance). Avance then begins synchronizing the new PM’s storage (displays ), which can take several hours.
    

    ---

    To avoid data loss, do not power down the primary PM while the disks are synchronizing.

    ---

11. While storage is synchronizing, reconnect the business links to the new PM.
12. On the Networks page, click the Fix button if it is highlighted. This might occur if any network cables were moved on the replaced PM.
13. Select the new PM. Click Finalize.
14. If the Dashboard warns that the upgraded PM has insufficient resources, repeat the replacement process starting at step 3 above. Correct the PM configuration when Avance completes step 4.
15. If Avance warns that the upgraded PM is not compatible with the running PM (different architecture), and prevents it from leaving maintenance state:
    1. Select the primary PM. Click Work On.
    2. When VMs shut down, select the upgraded PM. Click Finalize. The VMs restart on the upgraded PM, which becomes the primary.
16. Check the Avance unit periodically to ensure storage synchronization completes ( appears on the dashboard).

---

To avoid IP address conflicts, do not attach replaced PMs to the network until you have removed or overwritten the Avance software installation.

---

Appendix B: Connecting Business Networks

Connecting Additional Business Networks

If you added network ports to the PMs, connect network cables as follows:

1. In the Avance Management Console, go to the Networks page.
2. Connect additional network cables to both nodes, a pair at a time. Ideally in the same NIC slot and port number in each server: For each connection:
   1. Wait for the new Shared Network to appear. If it does not appear within a minute or so, it means your cables are on different subnets or the NIC ports between the nodes are not compatible (e.g 10Gb on Node 0 and 1Gb on Node1).
   2. Double click the shared network name and re-name it based on the based on its L3 Lan connectivity: “10.83.0.x” or “Engineering Lab”
3. Verify the newly formed shared network displays a green check.