Upgrading AHV Firmware with Ansible Automation Platform

Last updated: March 4, 2026

---

Table of Contents

1. Why Automate Firmware Upgrades?

2. Nutanix Upgrade Concepts – LCM vs Manual

3. Understanding AHV Firmware Upgrade Scope

4. Ansible Automation Platform Setup

5. Inventory and Credentials

6. Pre-Upgrade Health Check Playbook

7. Trigger LCM Firmware Upgrade via Prism Central API

8. Poll Upgrade Progress

9. Post-Upgrade Validation Playbook

10. Full Pipeline: Job Template Chain in AAP

11. Rollback Considerations

12. What I Learned the Hard Way

---

Why Automate Firmware Upgrades?

Nutanix's Life Cycle Manager (LCM) does a good job of managing firmware and software upgrades through the Prism Central UI. You click through a wizard, LCM inventories available updates, you select what to apply, and it handles the node-by-node rolling upgrade process.

But there are real operational reasons to drive this through Ansible and AAP instead:

• Consistent change window enforcement — AAP job templates enforce that upgrades only run during approved maintenance windows via schedules and approval gates

• Audit trail — every upgrade run is logged in AAP with who triggered it, what the before/after versions were, and the full output

• Pre/post gating — I want automated health checks before and after every firmware change, not manual ones I might skip at 11pm

• Multi-cluster coordination — I manage three CE clusters. A single AAP workflow upgrades all three sequentially with health validation between stages

This article shows the playbooks and AAP configuration I use to automate AHV and node firmware upgrades on my CE clusters.

---

Nutanix Upgrade Concepts – LCM vs Manual

Life Cycle Manager (LCM)

LCM is Nutanix's built-in upgrade orchestration framework, accessible via:

Prism Central → Operations → LCM

LCM handles:

Component	What LCM Can Upgrade
AOS	Nutanix operating system on CVMs
AHV	Hypervisor version on each node
Firmware	BIOS, BMC, HBA, NIC, SSD firmware
Prism Central	PC appliance software
NCC	Nutanix Cluster Check health framework
Foundation	Imaging tool on nodes

LCM operates by:

1. Inventory — scans available updates from Nutanix portal or a dark-site bundle

2. Pre-upgrade checks — validates cluster health, version compatibility, disk space

3. Rolling upgrade — upgrades one node at a time, live-migrates VMs off (for AHV upgrades), then upgrades that node, brings it back

Manual Upgrade (for reference)

Without LCM:

# AHV upgrade via acli on one CVM
nutanix@cvm:~$ host.upgrade ahv_installer_package.tar.gz target_host=<ahv-host-ip>

Manual upgrades are rarely needed but useful to understand — LCM wraps similar calls internally.

---

Understanding AHV Firmware Upgrade Scope

When I say "AHV firmware upgrade" in this article, I mean both:

1. AHV hypervisor version upgrade (e.g., 20230302.x → 20240304.x) — manages the KVM/QEMU layer version across nodes

2. Node hardware firmware (BIOS, BMC/iDRAC, NIC, SSD) — applied by LCM during a firmware update cycle

Both are managed through the LCM API via Prism Central. The Ansible playbooks in this article call the LCM REST API to:

1. Run an LCM inventory (discover available updates)

2. Create an LCM update plan for the target components

3. Execute the plan

4. Poll until completion

5. Run NCC post-upgrade checks

---

Ansible Automation Platform Setup

I run AAP 2.4 on a small VM (4 vCPU, 8 GB RAM) inside my Nutanix CE cluster — yes, it lives on the same cluster it manages, which is fine for CE but something to think carefully about for production.

AAP Version and Collections Required

# On your AAP Execution Environment or control node:
pip install ansible-runner
ansible-galaxy collection install ansible.builtin
ansible-galaxy collection install ansible.posix
ansible-galaxy collection install community.general
# nutanix.ncp for VM/snapshot operations (optional for this article)
ansible-galaxy collection install nutanix.ncp

No dedicated Nutanix collection exists for LCM — LCM operations use the standard Prism Central REST API via ansible.builtin.uri.

Execution Environment

I created a custom Execution Environment image with Nutanix-specific Python dependencies:

# ee/Dockerfile
FROM quay.io/ansible/creator-ee:latest

RUN pip install requests urllib3 PyYAML

Build and push to your private registry, then register in AAP:

AAP → Administration → Execution Environments → + Add
Name: nutanix-ee
Image: registry.lab.local/ansible/nutanix-ee:latest

---

Inventory and Credentials

Inventory

# inventory/nutanix_clusters.ini
[prism_central]
pc.lab.local ansible_host=10.0.10.20

[cvms]
cvm-node1 ansible_host=10.0.10.31
cvm-node2 ansible_host=10.0.10.32
cvm-node3 ansible_host=10.0.10.33

[cvms:vars]
ansible_user=nutanix
ansible_ssh_private_key_file={{ lookup('env','CVM_SSH_KEY') }}

[prism_central:vars]
pc_port=9440
pc_user=admin

Credentials in AAP

Create two credential types in AAP:

Prism Central API Credential (Custom type):

# Input fields
fields:
  - id: pc_host
    type: string
    label: Prism Central Hostname/IP
  - id: pc_user
    type: string
    label: Username
  - id: pc_password
    type: string
    label: Password
    secret: true

# Injectors
extra_vars:
  pc_host: "{{ pc_host }}"
  pc_user: "{{ pc_user }}"
  pc_password: "{{ pc_password }}"

CVM SSH Credential (Machine type):

Type: Machine
Username: nutanix
SSH Private Key: <paste CVM SSH private key>

---

Pre-Upgrade Health Check Playbook

Run this playbook before any firmware or AHV upgrade. It fails immediately if the cluster is not in a healthy state, preventing upgrades on a degraded cluster.

# playbooks/nutanix_pre_upgrade_check.yml
---
- name: Nutanix Pre-Upgrade Health Check
  hosts: prism_central
  gather_facts: false
  vars:
    pc_base_url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v3"
    pc_lcm_url: "https://{{ pc_host }}:{{ pc_port }}/lcm/v1.r0"
    ncc_check_url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v1"

  tasks:

    - name: Get cluster list
      ansible.builtin.uri:
        url: "{{ pc_base_url }}/clusters/list"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          kind: cluster
          length: 100
        status_code: 200
      register: cluster_list

    - name: Print cluster names
      ansible.builtin.debug:
        msg: "Cluster: {{ item.status.name }} | State: {{ item.status.resources.nodes.hypervisor_server_list | length }} nodes"
      loop: "{{ cluster_list.json.entities }}"

    - name: Run NCC health check (trigger)
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v1/health_checks/runs"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          check_type: all
        status_code: [200, 201, 202]
      register: ncc_run

    - name: Wait for NCC to complete
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v1/health_checks/runs/{{ ncc_run.json.task_uuid }}"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: ncc_result
      until: ncc_result.json.status in ['SUCCEEDED', 'FAILED']
      retries: 40
      delay: 30

    - name: Fail if NCC reports critical issues
      ansible.builtin.fail:
        msg: "NCC health check failed. Aborting upgrade. Review Prism alerts before proceeding."
      when: ncc_result.json.status == 'FAILED'

    - name: Check cluster fault tolerance
      ansible.builtin.uri:
        url: "{{ pc_base_url }}/clusters/list"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          kind: cluster
          length: 100
        status_code: 200
      register: clusters

    - name: Assert cluster can tolerate failure (RF2 = at least 3 nodes)
      ansible.builtin.assert:
        that:
          - item.status.resources.nodes.hypervisor_server_list | length >= 3
        fail_msg: "Cluster {{ item.status.name }} has fewer than 3 nodes. Upgrade aborted."
        success_msg: "Cluster {{ item.status.name }} has {{ item.status.resources.nodes.hypervisor_server_list | length }} nodes. OK."
      loop: "{{ clusters.json.entities }}"

    - name: Run LCM inventory to discover available updates
      ansible.builtin.uri:
        url: "{{ pc_lcm_url }}/operations/inventory"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body: {}
        status_code: [200, 202]
      register: lcm_inventory_task

    - name: Wait for LCM inventory to complete
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v3/tasks/{{ lcm_inventory_task.json.data.task_uuid }}"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: lcm_inv_result
      until: lcm_inv_result.json.status in ['SUCCEEDED', 'FAILED', 'ABORTED']
      retries: 20
      delay: 30

    - name: Get LCM available entities
      ansible.builtin.uri:
        url: "{{ pc_lcm_url }}/resources/entities/list"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: lcm_entities

    - name: Print entities with available updates
      ansible.builtin.debug:
        msg: >
          Entity: {{ item.entity_model }}
          Current: {{ item.version }}
          Available: {{ item.available_version_list | map(attribute='version') | list }}
      loop: "{{ lcm_entities.json.data.entities }}"
      when: item.available_version_list | length > 0

---

Trigger LCM Firmware Upgrade via Prism Central API

The LCM update flow is:

1. GET entity list → identify UUIDs and target versions for the components you want to upgrade

2. POST /lcm/v1.r0/operations/update → create the update plan

3. Poll the returned task UUID until completion

# playbooks/nutanix_lcm_upgrade.yml
---
- name: Trigger LCM Firmware / AHV Upgrade
  hosts: prism_central
  gather_facts: false
  vars:
    pc_lcm_url: "https://{{ pc_host }}:{{ pc_port }}/lcm/v1.r0"
    # Set these as extra_vars when launching from AAP:
    # upgrade_component: "AHV"   (or "AOS", "Firmware", "NCC")
    # target_version: "20240304.x"  (leave empty to take latest)

  tasks:

    - name: Get LCM entities
      ansible.builtin.uri:
        url: "{{ pc_lcm_url }}/resources/entities/list"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: lcm_entities

    - name: Build update spec for target component
      ansible.builtin.set_fact:
        update_entities: >-
          {{
            lcm_entities.json.data.entities
            | selectattr('entity_model', 'equalto', upgrade_component)
            | selectattr('available_version_list', 'ne', [])
            | map(attribute='uuid')
            | list
          }}
        update_versions: >-
          {{
            lcm_entities.json.data.entities
            | selectattr('entity_model', 'equalto', upgrade_component)
            | selectattr('available_version_list', 'ne', [])
            | map(attribute='available_version_list')
            | map('last')
            | map(attribute='version')
            | list
          }}

    - name: Fail if no entities found for component
      ansible.builtin.fail:
        msg: "No upgradeable entities found for component: {{ upgrade_component }}"
      when: update_entities | length == 0

    - name: Print entities to be upgraded
      ansible.builtin.debug:
        msg: "Will upgrade {{ upgrade_component }} entities: {{ update_entities }} to versions: {{ update_versions }}"

    - name: Confirm upgrade (requires manual approval in AAP workflow)
      ansible.builtin.pause:
        prompt: |
          About to upgrade {{ upgrade_component }} on {{ update_entities | length }} entities.
          Target versions: {{ update_versions }}
          Press ENTER to continue or Ctrl+C to abort.
      when: require_confirmation | default(false) | bool

    - name: Trigger LCM update
      ansible.builtin.uri:
        url: "{{ pc_lcm_url }}/operations/update"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          entity_update_spec_list: >-
            {{
              update_entities | zip(update_versions) | list
              | map('combine', {})
              | map('dict2items')
              | list
            }}
        status_code: [200, 202]
      register: lcm_update

    - name: Store LCM task UUID
      ansible.builtin.set_fact:
        lcm_task_uuid: "{{ lcm_update.json.data.task_uuid }}"

    - name: Print LCM task UUID
      ansible.builtin.debug:
        msg: "LCM upgrade task started: {{ lcm_task_uuid }}"

Note on the entity_update_spec_list format: The LCM API expects a list of objects with uuid and to_version. The approach above is illustrative; for production use, build the list explicitly with a set_fact loop over known entity UUIDs and target versions instead of relying on zip transforms.

Explicit update spec example (recommended):

- name: Build explicit update spec
  ansible.builtin.set_fact:
    entity_update_spec_list: []

- name: Add each entity to spec
  ansible.builtin.set_fact:
    entity_update_spec_list: "{{ entity_update_spec_list + [{'uuid': item.uuid, 'to_version': item.available_version_list[-1].version}] }}"
  loop: "{{ lcm_entities.json.data.entities }}"
  when:
    - item.entity_model == upgrade_component
    - item.available_version_list | length > 0

- name: Trigger LCM update with explicit spec
  ansible.builtin.uri:
    url: "{{ pc_lcm_url }}/operations/update"
    method: POST
    user: "{{ pc_user }}"
    password: "{{ pc_password }}"
    force_basic_auth: true
    validate_certs: false
    body_format: json
    body:
      entity_update_spec_list: "{{ entity_update_spec_list }}"
    status_code: [200, 202]
  register: lcm_update

---

Poll Upgrade Progress

LCM upgrades for AHV can take 30–90 minutes (node-by-node, with VMs live-migrated off each node). This playbook polls until the task completes.

# playbooks/nutanix_lcm_poll.yml
---
- name: Poll LCM Upgrade Progress
  hosts: prism_central
  gather_facts: false
  vars:
    max_retries: 180    # 180 × 60s = 3 hours max
    poll_interval: 60   # seconds between polls

  tasks:

    - name: Poll LCM task until completion
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v3/tasks/{{ lcm_task_uuid }}"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: task_result
      until: task_result.json.status in ['SUCCEEDED', 'FAILED', 'ABORTED']
      retries: "{{ max_retries }}"
      delay: "{{ poll_interval }}"

    - name: Print final task status
      ansible.builtin.debug:
        msg: >
          LCM Task {{ lcm_task_uuid }}
          Status: {{ task_result.json.status }}
          Progress: {{ task_result.json.percentage_complete }}%

    - name: Fail playbook if upgrade did not succeed
      ansible.builtin.fail:
        msg: "LCM upgrade task {{ lcm_task_uuid }} ended with status: {{ task_result.json.status }}. Check Prism Central LCM logs."
      when: task_result.json.status != 'SUCCEEDED'

    - name: Log upgrade completion
      ansible.builtin.lineinfile:
        path: /var/log/nutanix-upgrades.log
        line: "{{ ansible_date_time.iso8601 }} | LCM upgrade {{ lcm_task_uuid }} SUCCEEDED | Component: {{ upgrade_component }}"
        create: true
      delegate_to: localhost

---

Post-Upgrade Validation Playbook

After LCM completes, validate that:

1. All CVMs are UP

2. All node services are running

3. NCC health check passes

4. AHV version matches the expected target

# playbooks/nutanix_post_upgrade_check.yml
---
- name: Nutanix Post-Upgrade Validation
  hosts: cvms
  gather_facts: false
  vars:
    expected_ahv_version: "{{ target_version | default('') }}"

  tasks:

    - name: Check CVM cluster status
      ansible.builtin.command: cluster status
      register: cluster_status_output
      changed_when: false

    - name: Assert all required services are UP
      ansible.builtin.assert:
        that:
          - "'Zeus    UP' in cluster_status_output.stdout"
          - "'Stargate UP' in cluster_status_output.stdout"
          - "'Cassandra UP' in cluster_status_output.stdout"
          - "'Prism    UP' in cluster_status_output.stdout"
        fail_msg: "One or more CVM services are not UP on {{ inventory_hostname }}"
        success_msg: "All core services UP on {{ inventory_hostname }}"


- name: Verify AHV version via Prism Central API
  hosts: prism_central
  gather_facts: false
  vars:
    pc_base_url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v3"

  tasks:

    - name: Get hosts list
      ansible.builtin.uri:
        url: "{{ pc_base_url }}/hosts/list"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          kind: host
          length: 100
        status_code: 200
      register: hosts_list

    - name: Print AHV versions per node
      ansible.builtin.debug:
        msg: >
          Host: {{ item.status.name }}
          AHV Version: {{ item.status.resources.hypervisor.hypervisor_full_name }}
      loop: "{{ hosts_list.json.entities }}"

    - name: Verify expected AHV version (if target_version set)
      ansible.builtin.assert:
        that:
          - expected_ahv_version in item.status.resources.hypervisor.hypervisor_full_name
        fail_msg: >
          Node {{ item.status.name }} is on AHV version
          {{ item.status.resources.hypervisor.hypervisor_full_name }},
          expected to contain {{ expected_ahv_version }}
        success_msg: "Node {{ item.status.name }} AHV version OK"
      loop: "{{ hosts_list.json.entities }}"
      when: expected_ahv_version != ''

    - name: Run post-upgrade NCC check
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v1/health_checks/runs"
        method: POST
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        body_format: json
        body:
          check_type: all
        status_code: [200, 201, 202]
      register: post_ncc_run

    - name: Wait for post-upgrade NCC
      ansible.builtin.uri:
        url: "https://{{ pc_host }}:{{ pc_port }}/api/nutanix/v1/health_checks/runs/{{ post_ncc_run.json.task_uuid }}"
        method: GET
        user: "{{ pc_user }}"
        password: "{{ pc_password }}"
        force_basic_auth: true
        validate_certs: false
        status_code: 200
      register: post_ncc_result
      until: post_ncc_result.json.status in ['SUCCEEDED', 'FAILED']
      retries: 40
      delay: 30

    - name: Fail if post-upgrade NCC has critical issues
      ansible.builtin.fail:
        msg: "Post-upgrade NCC health check failed. Review Prism Central alerts."
      when: post_ncc_result.json.status == 'FAILED'

    - name: Summary
      ansible.builtin.debug:
        msg: "Post-upgrade validation PASSED. Cluster healthy after LCM upgrade."

---

Full Pipeline: Job Template Chain in AAP

In AAP, chain the playbooks into a Workflow Job Template with approval gates:

Workflow: Nutanix LCM Upgrade Pipeline
│
├── [Job] Pre-Upgrade Health Check
│     playbook: nutanix_pre_upgrade_check.yml
│     On Success ──►
│
├── [Approval] Gate: Confirm Upgrade
│     Message: "Pre-checks passed. Approve to proceed with {{ upgrade_component }} upgrade."
│     On Approve ──►
│
├── [Job] LCM Upgrade Trigger
│     playbook: nutanix_lcm_upgrade.yml
│     extra_vars: upgrade_component, target_version
│     On Success ──►
│
├── [Job] LCM Poll Progress
│     playbook: nutanix_lcm_poll.yml
│     On Success ──►
│
├── [Job] Post-Upgrade Validation
│     playbook: nutanix_post_upgrade_check.yml
│     On Success ──► Workflow SUCCESS
│     On Failure ──► Workflow FAILURE (alert notification)

AAP Schedule for Maintenance Windows

AAP → Job Templates → Nutanix LCM Upgrade Pipeline → Schedules → + Add
Name: Monthly Firmware Maintenance
Timezone: Asia/Yangon
Start: First Saturday of each month, 02:00
Prompt on Launch: upgrade_component=Firmware

Notification on Failure

AAP → Notifications → + Add
Type: Webhook (or Email, Slack)
Name: Nutanix Upgrade Failure Alert
URL: https://hooks.slack.com/services/xxx/yyy/zzz

Attach to: Nutanix LCM Upgrade Pipeline → Notifications → Failure

---

Rollback Considerations

Nutanix LCM does not have a one-click rollback for AHV or firmware upgrades. Here is what I do instead:

AHV Version Rollback

AHV upgrade is managed by LCM and is typically forward-only once applied. However:

1. Nutanix Support can provide a downgrade path for AHV — contact them with your ticket before attempting

2. For CE, the fastest recovery from a bad AHV upgrade is to re-image the node using Foundation

Pre-Upgrade Snapshot of CVMs

Before triggering LCM, snapshot all CVM VMs as a precaution (though CVM snapshots are rarely needed — Nutanix recommends against CVM snapshots in production, but for CE this is acceptable insurance):

nutanix@cvm:~$ acli vm.snapshot_create <cvm-vm-name> snapshot_name=pre-ahv-upgrade-$(date +%Y%m%d)

Firmware Rollback

For BMC/BIOS firmware changes:

• Nutanix LCM stores the previous firmware version metadata in its internal DB

• In some versions, LCM supports rollback for specific components; check lcm/v1.r0/resources/entities/list for rollback_version in entity metadata

• Hardware vendor tools (e.g., iDRAC firmware rollback) remain an option if LCM rollback is not available

---

What I Learned the Hard Way

1. Never run LCM upgrades with degraded storage

I once triggered a firmware upgrade while a disk was showing Warning in Prism. LCM started but paused mid-way when the node it was upgrading couldn't confirm RF2 compliance. The cluster entered a partially upgraded state that took an hour to resolve. The pre-upgrade NCC check should catch this — do not skip it.

2. VM live migration must work before AHV upgrade

LCM drains each node before upgrading AHV — it live-migrates all VMs off the node. If your CPUs between nodes are different generations (e.g., mixed NUC 12 and NUC 13), live migration may fail for VMs without CPU masking. Test live migration manually before LCM runs:

nutanix@cvm:~$ acli vm.migrate <vm-name> host=<target-ahv-ip>

Fix CPU compatibility issues (enable EVC/CPU masking) before triggering any AHV upgrade.

3. LCM task API vs v3 task API

LCM uses https://pc:9440/lcm/v1.r0/ for its own operations, but the task polling endpoint for LCM tasks uses the standard v3 tasks API at https://pc:9440/api/nutanix/v3/tasks/<uuid>. This inconsistency caught me off guard — I was polling the LCM endpoint for task status and getting 404 errors until I found this in the LCM API docs.

4. CE LCM requires internet access for inventory

CE clusters need outbound HTTPS to download.nutanix.com for LCM to discover available updates. In a fully air-gapped CE setup, you need to set up a dark-site LCM bundle. In AAP, make sure your execution environment can reach the Prism Central IP, and that PC can reach Nutanix download servers during inventory.

---

Next Steps

• Automating Nutanix Backup with Ansible

• GitLab CI/CD Integration with Nutanix Blueprint

• Implementing HCI with Nutanix

---

Tags: Nutanix, AHV, LCM, Life Cycle Manager, Ansible, Ansible Automation Platform, AAP, Firmware Upgrade, AOS Upgrade, Prism Central API, Community Edition