# Babuk ESXi VMDK Recovery Runbook for Operators and AI Agents ## Purpose This runbook describes a conservative method for recovering data from VMware ESXi VMDK files damaged by Babuk-style ransomware. It is intended for system administrators, incident responders, and AI agents operating under human supervision. The method described here is usually **reconstruction**, not cryptographic decryption. It works when ransomware has encrypted only selected regions—often the beginning of a virtual disk—while later filesystem data and backup metadata remain intact. There is no universal command that decrypts every `.babyk` file. A genuine decryptor requires the correct ransomware variant and cryptographic key. Never claim that reconstruction is decryption. ## Non-negotiable safety rules 1. Never repair, mount read-write, rename, truncate, or otherwise modify the original encrypted VMDK. 2. Never execute the ransomware sample to “test” it. 3. Preserve the encrypted descriptor, flat extent, ransom notes, logs, malicious scripts, cron entries, and ransomware footer before cleanup. 4. Work from a clone or a disposable copy-on-write layer. 5. Attach evidence disks to the recovery VM in VMware `independent-nonpersistent` mode. 6. Set the Linux block device read-only immediately after every boot. 7. Run `xfs_repair -n` before any modifying XFS command. 8. Never use `xfs_repair -L` on the original disk or a direct read-only reconstruction. 9. Confirm exact paths and device sizes before every command. Do not rely only on `/dev/sdX` letters. 10. Record every command, offset, hash, device-mapper table, and result. ## Recovery architecture ```text Encrypted flat VMDK (immutable) | v Clean VMDK descriptor excluding ransomware footer | v VMware independent-nonpersistent disk | v Linux read-only block device | v Read-only LVM/LV device-mapper reconstruction | v Disposable capped device-mapper COW snapshot | v XFS repair inside COW only | v Read-only recovered mount -> copy data elsewhere ``` Every layer above the disposable COW must remain read-only. ## Phase 1: Contain and preserve Before recovery: - Disconnect compromised workloads from public networks. - Power off encrypted VMs gracefully when possible. - Prevent the malicious cron job or startup script from running again. - Preserve suspicious files instead of deleting them immediately. - Restrict ESXi management access to trusted addresses. - Record the incident time, timezone, ESXi version, VM IDs, datastore UUIDs, and disk paths. - Take hashes of small evidence files and ransomware footers. Do not assume that an attacker is gone merely because encryption stopped. ## Phase 2: Inventory the VMDK on ESXi Identify both parts of each virtual disk: ```text vm-name.vmdk.babyk vm-name-flat.vmdk.babyk ``` The small `.vmdk` is normally a text descriptor. If ransomware encrypted it, it may appear as binary data. The large `-flat.vmdk` contains the virtual disk sectors. Collect read-only facts: ```sh ls -l /vmfs/volumes/DATASTORE/VM/ stat /vmfs/volumes/DATASTORE/VM/vm-name-flat.vmdk.babyk hexdump -C -n 128 /vmfs/volumes/DATASTORE/VM/vm-name-flat.vmdk.babyk tail -c 128 /vmfs/volumes/DATASTORE/VM/vm-name-flat.vmdk.babyk | hexdump -C vmkfstools -D /vmfs/volumes/DATASTORE/VM/vm-name-flat.vmdk.babyk ``` Check for: - Random-looking bytes where a partition table, LVM label, or filesystem signature should exist. - A small random footer appended after the original virtual-disk size. - An encrypted binary descriptor. - Active VMware locks. ### Preserve an appended footer If analysis confirms a 64-byte footer: ```sh tail -c 64 SOURCE-flat.vmdk.babyk > SAFE_LOCATION/SOURCE.babyk.footer64 chmod 400 SAFE_LOCATION/SOURCE.babyk.footer64 sha256sum SAFE_LOCATION/SOURCE.babyk.footer64 ``` Do not remove the footer from the original file. ## Phase 3: Calculate the real virtual-disk size If the encrypted flat file is the original disk plus a 64-byte footer: ```text real_bytes = encrypted_file_bytes - 64 sectors = real_bytes / 512 ``` Both divisions must be exact. Stop if the result is not sector-aligned. ### Worked 5 TiB example ```text Encrypted flat size: 5,497,558,138,944 bytes Footer: 64 bytes Real disk size: 5,497,558,138,880 bytes Descriptor sectors: 10,737,418,240 ``` ## Phase 4: Create a clean descriptor Create a new tiny descriptor; do not replace the encrypted descriptor. ```text # Disk DescriptorFile version=1 encoding="UTF-8" CID=fffffffd parentCID=ffffffff createType="vmfs" # Extent description RW 10737418240 VMFS "vm-name-flat.vmdk.babyk" # The Disk Data Base #DDB ddb.adapterType = "lsilogic" ddb.geometry.cylinders = "668373" ddb.geometry.heads = "255" ddb.geometry.sectors = "63" ddb.virtualHWVersion = "14" ``` The `RW` value is the important field. It must expose only the real disk sectors and exclude the ransomware footer. Validate the descriptor without opening it in a VM: ```sh vmkfstools -e /vmfs/volumes/DATASTORE/VM/vm-name-recovery.vmdk ``` Expected result: ```text Disk chain is consistent. ``` ## Phase 5: Check datastore capacity Never begin a full clone unless enough space exists for the actual allocation behavior of the operation. For a 5 TiB source and only 2–3 TiB free, do not create a full clone. Use: - A read-only descriptor. - VMware independent-nonpersistent mode. - A small, capped repair COW layer. - Selective file copying to off-host storage. Thin provisioning does not eliminate the risk of datastore exhaustion. ## Phase 6: Attach safely to a recovery VM Use a dedicated SystemRescue or forensic Linux VM. Power it off before changing VMX disk mode. Add the recovery descriptor to a free SCSI slot, then ensure the VMX contains: ```text scsi0:3.mode = "independent-nonpersistent" ``` Reload the VM configuration and verify the mode before powering on. After Linux boots, identify the disk by its exact byte size and immediately set it read-only: ```sh lsblk -b -o NAME,SIZE,TYPE,RO,FSTYPE,LABEL,MOUNTPOINTS blockdev --setro /dev/sdX blockdev --getro /dev/sdX ``` The last command must print `1`. Repeat this check after every reboot. ## Phase 7: Determine whether data is renamed or encrypted A `.babyk` extension alone does not prove file contents are encrypted. Check: ```sh blkid -p /dev/sdX wipefs -n /dev/sdX fdisk -l /dev/sdX parted -s /dev/sdX unit s print dd if=/dev/sdX bs=64 count=1 status=none | od -An -tx1 ``` Interpretation: - Valid partition/LVM/filesystem signatures suggest renaming or limited encryption. - Random data at sector zero with valid later metadata suggests partial encryption. - Random data across sampled regions may indicate broader encryption. - Do not conclude that the entire disk is unrecoverable based only on its first sector. ## Phase 8: Recover LVM geometry from the OS disk If the affected data disk used LVM, look for backups on a recovered system/root filesystem: ```text /etc/lvm/backup/ /etc/lvm/archive/ ``` An LVM backup can preserve: - PV UUID. - VG and LV names. - Physical extent size. - PV data start. - LV starting extent. - LV extent count. Important units: LVM text metadata normally records `extent_size`, `pe_start`, and `dev_size` in 512-byte sectors. For a linear LV: ```text lv_length_sectors = extent_count * extent_size lv_start_sector = pe_start + (start_extent * extent_size) ``` ### Worked LVM example ```text extent_size = 8192 sectors # 4 MiB pe_start = 2048 sectors # 1 MiB start_extent = 0 extent_count = 1,310,719 lv_length = 1,310,719 * 8,192 = 10,737,410,048 sectors lv_start = 2,048 sectors ``` Create a read-only linear reconstruction: ```sh dmsetup create vg_data-lv_data-recovery --readonly --table \ "0 10737410048 linear /dev/sdX 2048" blockdev --getro /dev/mapper/vg_data-lv_data-recovery ``` The read-only check must return `1`. ## Phase 9: Locate surviving XFS secondary superblocks Run only the no-modify check first: ```sh xfs_repair -n /dev/mapper/vg_data-lv_data-recovery ``` A promising result looks like: ```text bad primary superblock attempting to find secondary superblock found candidate secondary superblock verified secondary superblock ``` This proves that later XFS allocation-group metadata survives. If the tool does not print the offset, trace read seeks without modifying data: ```sh strace -f -e trace=lseek -o /tmp/xfs-repair.trace \ xfs_repair -n /dev/mapper/vg_data-lv_data-recovery ``` Inspect large aligned seeks: ```sh cat /tmp/xfs-repair.trace ``` Verify candidate blocks directly. For a 4096-byte XFS block size: ```sh dd if=/dev/mapper/vg_data-lv_data-recovery \ bs=4096 skip=CANDIDATE_BLOCK count=1 status=none | dd bs=1 count=64 status=none | od -An -tx1 ``` A valid XFS superblock begins with hexadecimal: ```text 58 46 53 42 ``` This is ASCII `XFSB`. ## Phase 10: Create a capped disposable repair snapshot Do not repair the read-only reconstruction directly. Load device-mapper snapshot support: ```sh modprobe dm_snapshot dmsetup targets | grep snapshot ``` Create a capped COW file on the recovery system disk. Ensure that filesystem has enough free space: ```sh truncate -s 8G /persistent/data5tb-repair.cow losetup --find --show /persistent/data5tb-repair.cow ``` Assuming the returned loop device is `/dev/loop2`: ```sh dmsetup create data5tb-repair --table \ "0 10737410048 snapshot /dev/mapper/vg_data-lv_data-recovery /dev/loop2 P 128" ``` Verify all layers: ```sh blockdev --getro /dev/sdX blockdev --getro /dev/mapper/vg_data-lv_data-recovery blockdev --getro /dev/mapper/data5tb-repair dmsetup status data5tb-repair du -h /persistent/data5tb-repair.cow ``` Expected: - Source disk: read-only (`1`). - Linear origin: read-only (`1`). - Disposable snapshot: writable (`0`). - COW consumption well below its cap. If the COW fills, the snapshot becomes invalid. Discard it and start again with a larger COW; the original remains unchanged. ## Phase 11: Repair only the disposable snapshot First run without log destruction: ```sh xfs_repair /dev/mapper/data5tb-repair ``` If XFS reports that the internal log contains metadata changes, first try a read-only mount with log replay disabled: ```sh mkdir -p /mnt/data-recovery mount -o ro,norecovery,nouuid /dev/mapper/data5tb-repair /mnt/data-recovery ``` If the mount fails and the log cannot be replayed, `-L` may be necessary. Use it only on the disposable snapshot: ```sh xfs_repair -L /dev/mapper/data5tb-repair ``` `-L` destroys the XFS journal. It can lose the most recent uncommitted metadata changes. It is acceptable only after the original is protected and the repair target is disposable. Monitor COW usage while repair runs: ```sh dmsetup status data5tb-repair du -h /persistent/data5tb-repair.cow df -h /persistent ``` ## Phase 12: Mount and assess recovered data After a successful repair: ```sh mount -o ro,norecovery,nouuid \ /dev/mapper/data5tb-repair /mnt/data-recovery findmnt /mnt/data-recovery df -h /mnt/data-recovery xfs_info /mnt/data-recovery ``` If the original root directory inode was encrypted, XFS may reconnect surviving directory trees under: ```text /mnt/data-recovery/lost+found/ ``` Numeric names are normally inode numbers, not encrypted filenames. Child directory and file names can still be intact. Classify recovered objects without reading all file contents: ```sh find /mnt/data-recovery/lost+found \ -mindepth 1 -maxdepth 1 -printf '%y\n' | sort | uniq -c ``` Create a size index: ```sh du -x -B1 --max-depth=1 /mnt/data-recovery/lost+found | sort -nr > /persistent/recovered-top-level-bytes.txt ``` This can take time on filesystems with millions of inodes. ## Phase 13: Copy recovered files elsewhere Copy from the read-only recovery mount to separate storage. Do not copy back to the affected datastore unless capacity is verified. Example pull from a trusted destination: ```sh rsync -aHAX --numeric-ids --info=progress2 \ root@RECOVERY_HOST:/mnt/data-recovery/lost+found/RECOVERED_TREE/ \ /safe/destination/RECOVERED_TREE/ ``` For untrusted or cross-organization destinations, use a dedicated restricted SSH account instead of root. After copying: - Compare file counts and byte totals. - Generate SHA-256 manifests for critical files. - Open representative documents, images, archives, databases, and media. - Run application-specific integrity checks. - Scan recovered content for malware before returning it to production. Data blocks located inside the ransomware-encrypted region may remain corrupt even when their filenames and metadata are recoverable. ## Making SystemRescue persistent A live ISO normally loses network configuration, SSH keys, firewall rules, read-only flags, loop devices, and device-mapper mappings after reboot. For long recoveries, install SystemRescue on a small dedicated virtual disk or configure its persistent backing store. Keep evidence disks separate. Use an autorun script to: - Restore static networking. - Install authorized SSH public keys; never store passwords. - Enforce key-only SSH. - Set evidence disks read-only by exact byte size. - Recreate read-only LVM mappings. - Reattach the persistent repair COW. - Mount the recovered filesystem read-only. Official documentation: - - - ## AI-agent operating contract Give the recovery agent an explicit contract similar to this: ```text You are assisting with ransomware recovery. Treat every original VMDK and evidence file as immutable. Do not mount original disks read-write. Do not run repair tools without no-modify mode first. Do not use xfs_repair -L except on a verified disposable COW target. Before every modifying command, report: 1. the exact target, 2. whether it is original, clone, or COW, 3. the read-only state of every lower layer, 4. maximum additional storage consumption, 5. rollback procedure. Identify disks by exact byte size and recorded UUIDs, not only /dev/sdX. Keep a command log and verify every calculated sector offset. Never expose passwords, private keys, customer filenames, or recovered data. Stop if device identity, units, free space, or write isolation is uncertain. ``` ## Status format for AI agents Agents should report recovery status using a consistent structure: ```text Original source: unchanged / read-only VMware mode: independent-nonpersistent Descriptor size: N sectors LVM reconstruction: table and offsets verified Filesystem: XFS UUID and geometry Repair target: disposable COW only COW used/capacity: X GiB / Y GiB Repair phase/result: phase N / exit code Recovered mount: path and mount options Recovered allocated bytes: value Unresolved risks: list Next reversible action: command or procedure ``` ## Common mistakes - Running `xfs_repair -L` directly on the encrypted VMDK. - Attaching evidence in persistent VMware mode. - Forgetting that Linux read-only flags reset after reboot. - Treating a large sparse REDO or COW logical size as actual consumed space. - Creating a 5 TiB clone on a datastore with less than 5 TiB free. - Using a descriptor sector count that includes the ransomware footer. - Confusing LVM extent units with bytes. - Assuming a destroyed partition table means later filesystem metadata is gone. - Mounting XFS read-only without `norecovery`; XFS may otherwise attempt log replay. - Rebooting before saving reconstruction tables and COW paths. - Deleting attacker artifacts before hashing and preserving them. ## What success looks like A successful reconstruction has all of these properties: - The original VMDK hashes and timestamps remain unchanged. - The original block device and linear reconstruction are read-only. - All repair writes are isolated in a capped disposable COW. - XFS repair completes on the COW with exit code zero. - The recovered filesystem mounts `ro,norecovery,nouuid`. - Directory trees or `lost+found` objects are accessible. - Important files are copied to independent storage and verified. - Recovery steps can be recreated after reboot from recorded scripts and metadata. Even after these conditions are met, rebuild production systems from clean installation media. Treat recovered data as untrusted input, not as a trustworthy operating system image.