Reproducing the Multpath I/O Issue With Ubuntu 22.04

-

Problem Statement

In the previous two articles, I described the issue I was running into when installing Ubuntu 22.04 on a Dell VRTX server.  After some research, I suspect it's related to the incorrect configuration of the multipath I/O function.

How to fixed it?

Reproducing the Issue with VMware

To reproduce and fix the multipath issue, I decided to set up a testing environment with a VM running inside VMware ESXi.  If we can create a VM with two virtual SCSI HBA cards, and connect the same virtual hard drive file to them.  The Linux VM should be able to see the same hard drive twice via two SCSI paths.

Multipath I/O Config In VMware ESXi

First Try

With VMware ESXi, when adding a SCSI card to a VM, there is one option to configure if the SCSI bus is sharing.  The default configuration is None.  In order to share the same virtual hard drive to the VM twice via those two SCSI cards, I set them to "Physical".  Ideally, setting to "Virtual" should also work.

When adding the virtual hard drive to the VM, there is an option "Sharing".  To share the hard drive through multiple SCSI cards, I set it to "Multi-writer sharing".

The second hard drive was added with the same .vmdk file and the same settings for "Sharing".

After the VM was created, I started to install Ubuntu 22.04 into that VM.  Surprisingly, the issue did not happen.  Ubuntu actually supports multipath I/O!!!  So multipath I/O itself does not seem to be the root cause of the problem.

If running the multipath command, it actually correctly detected the multipath capability for the hard drives.

# multipath -l 

mpatha (360000c29a131cd4fcc993dffd7b994035) dm-0 VMware, Virtual disk

size=32G feature'0' hwhandler='0' wp=rw

|-+- policy='service-time 0' prio=0 status=active

| `- 33:0:0:0 sda 8:0 active under running

`-+- policy='service-time 0' prio=0 status=enabled

  `-34:0:0:0 sdb 8:32 active undef running

Why did it happen to me earlier then?

Second Try

How come my VM setup is not able to reproduce the issue?  With that questions in mind, I started to think what's the difference between my VM and the Dell VRTX server?

Hey, the VRTX server has two hard drives detected by Ubuntu 22.04's installation program.  One is the logical drive, and the other is the IDSDM Storage Device.  When installing Ubuntu 22.04, I installed the /boot and /boot/efi into the IDSDM Storage Device.  While the rest of the OS is installed into the logical drive.

But with the VM I set up, everything is installed on one single hard drive.

How about adding a 3rd hard drive to the VM, then installing /boot and /boot/efi to the 3rd drive?

After the VM is created, I started to install Ubuntu to the VM again.  As usual, the installation was completed successfully.  After a reboot, the issue indeed happened!  The system dropped into the (initramfs) prompt again.


Comments

Post a Comment

Popular posts from this blog

Secure Home Network with OPNsense

-
Image
I was working on some smart home project as I bought a new home earlier this year.  With more and more devices connected to my home network then to Internet.  I do have some concerns regarding the security of those devices.  For two reasons: Quite of the smart home devices need to have Internet connection and being controlled by some SaaS platform Any devices in my home network can communicate with the smart home devices Smart home devices can freely talk to other devices in my home network In order to provide some protection to the home network and those smart home devices, I decided to bring some open source software firewall to segregate the traffic for different types of the devices. After doing some research, I decided to go with OPNsense firewall.  According to the article on Wikipedia,  OPNsense is an open source, FreeBSD-based firewall and routing software developed by Deciso. It is a fork of pfSense, which in turn was forked from m0n0wall built o...
Read more

Dropping Into "initramfs" Prompt After Installing Ubuntu 22.04 LTS on to Dell VRTX Server

-
Image
Background During one of the recent projects, I want to install a Linux OS onto a Dell VRTX server. Without checking the hardware compatibility list and with limited knowledge about the Dell VRTX server platform, I started to install Ubuntu 20.04 LTS on one of the server blades in the chassis. And running into some interesting issues, which forces me to learn more about the Dell VRTX platform and Linux OS. Though spending quite a lot of effort in sorting out the issue, I am happy to share it with others who are interested to know more about it. Server blades inside the Dell VRTX chassis have no hard hard drive installed. But the chassis enclosure has 25 2.5 in SAS hard drive installed. Before starting the whole process, I picked up two of the SAS hard drive and created a RAID-1 logical drive, and assigned it to the server blade. I thought the system would be able to boot from the new logical drive. The Start of the Problem After inserting the USB disk with Ubuntu 22.04, the server blad...
Read more

Ubuntu 22.04 Multipath I/O Problem with Dell VRTX

-
Image
High-Level Architecture of Dell PowerEdge VRTX In the previous article , I explain the issue I am running into when installing Ubuntu 22.04 on a Dell VRTX server.  To understand a little bit more about what went wrong and potentially fix the problem, I tried to know how the Dell VRTX server was designed. After some googling, I found the  Technical Guide from Dell.  It revealed a pretty interesting design of the system.  The Dell VRTX system is designed to be a Cost-Effective Hyperconverged server system for a small office environment or remote office environment, where there are some compute power is needed but not on a large scale.  One Dell VRTX chassis can host up to 4 server blades, up to 25 x 2.5 in. NLSAS, SAS, or SAS SSD hot-plug drives (not including the hard drive bays on the server blades), and 8 flexible PCIe slots.  All those hard drives, PCIe slots, and network ports belong to the Dell VRTX chassis.  The chassis management syste...
Read more