Ubuntu 22.04 Multipath I/O Problem with Dell VRTX

-

High-Level Architecture of Dell PowerEdge VRTX

In the previous article, I explain the issue I am running into when installing Ubuntu 22.04 on a Dell VRTX server.  To understand a little bit more about what went wrong and potentially fix the problem, I tried to know how the Dell VRTX server was designed.

After some googling, I found the Technical Guide from Dell.  It revealed a pretty interesting design of the system.  The Dell VRTX system is designed to be a Cost-Effective Hyperconverged server system for a small office environment or remote office environment, where there are some compute power is needed but not on a large scale.  One Dell VRTX chassis can host up to 4 server blades, up to 25 x 2.5 in. NLSAS, SAS, or SAS SSD hot-plug drives (not including the hard drive bays on the server blades), and 8 flexible PCIe slots.  All those hard drives, PCIe slots, and network ports belong to the Dell VRTX chassis.  The chassis management system can dynamically allocate resources to the server blades installed.  Unlike traditional ways how the resources are shared via Ethernet connectivity in a data center environment, the Dell VRTX chassis share the resources via the PCIe switch.  (Fabric B and Fabric C in the block diagram below.)

According to the block diagram, there are two of the Dell Shared PowerEdge RAID Controller (PERC) 8 connected to the PCIe switch fabrics. One to each of the fabrics.  Each server blade has two PCIe connections, one to each of the PCIe switch fabrics too!

Dell Shared PowerEdge RAID Controller (PERC) 8 For Dell PowerEdge VRTX Systems

According to the User's Guide published by Dell,

The Dell Shared PowerEdge RAID Controller (PERC) 8 card installed on the Dell PowerEdge VRTX system supports Single Root Input Output Virtualization (SR-IOV). This feature allows multiple systems to share available hardware resources.  The PERC 8 card allows four server modules to access a local shared storage. Each server module's operating system loads a Virtual Function (VF) driver that allows the server module to communicate with the Shared PERC 8 firmware. Virtual disks located on the shared storage can then be mapped to a server module.

In the Fault Tolerant Shared PERC 8 card configuration, the PowerEdge VRTX system has two Shared PERC 8 cards in an active-passive mode. In this configuration, each host server can access the same virtual disk through both cards. In the absence of a multipath driver, the operating system identifies the single disk as two identical disks. The multipath driver aggregates the two disks into a single disk with redundant access that is then presented to the operating system. The operating system’s multipath functionality determines the path chosen for any I/O sent to the virtual disk. Any I/O directed to the passive card is internally redirected to the active card for completion.

 Now, it explains why our OS is seeing the same hard drive twice.

Multipath I/O

The multipath mentioned in Dell's document reminded me of the multipath we were using while I was managing the Oracle Database on Sun servers.  Because the data stored in the database were so critical, there were a lot of performance and high-availability considerations built into the system.  Database files were stored in some SAN storage systems connected via the SAN network.  To address the single point of failure situation, there is more than one SAN switch in the network.  Each sun server has two fiber connections to each of the SAN switches.  The SAN storage system has similar connections.  Each LUN defined in the SAN storage system are accessible via both fiber path.   Both LUNs were mapped to the hard drives on the Sun server side.  Then the multipath driver or software controls which hard drive to use when there is an I/O request for that drive.  At any one time, only one of the paths was being used.  If one path failed, it will use the 2nd path.  So that there is no single point of failure for the storage system.


Linux OS does offer similar multipath functions.  There must be something not done correctly.

Comments

Post a Comment

Popular posts from this blog

Secure Home Network with OPNsense

-
Image
I was working on some smart home project as I bought a new home earlier this year.  With more and more devices connected to my home network then to Internet.  I do have some concerns regarding the security of those devices.  For two reasons: Quite of the smart home devices need to have Internet connection and being controlled by some SaaS platform Any devices in my home network can communicate with the smart home devices Smart home devices can freely talk to other devices in my home network In order to provide some protection to the home network and those smart home devices, I decided to bring some open source software firewall to segregate the traffic for different types of the devices. After doing some research, I decided to go with OPNsense firewall.  According to the article on Wikipedia,  OPNsense is an open source, FreeBSD-based firewall and routing software developed by Deciso. It is a fork of pfSense, which in turn was forked from m0n0wall built o...
Read more

Dropping Into "initramfs" Prompt After Installing Ubuntu 22.04 LTS on to Dell VRTX Server

-
Image
Background During one of the recent projects, I want to install a Linux OS onto a Dell VRTX server. Without checking the hardware compatibility list and with limited knowledge about the Dell VRTX server platform, I started to install Ubuntu 20.04 LTS on one of the server blades in the chassis. And running into some interesting issues, which forces me to learn more about the Dell VRTX platform and Linux OS. Though spending quite a lot of effort in sorting out the issue, I am happy to share it with others who are interested to know more about it. Server blades inside the Dell VRTX chassis have no hard hard drive installed. But the chassis enclosure has 25 2.5 in SAS hard drive installed. Before starting the whole process, I picked up two of the SAS hard drive and created a RAID-1 logical drive, and assigned it to the server blade. I thought the system would be able to boot from the new logical drive. The Start of the Problem After inserting the USB disk with Ubuntu 22.04, the server blad...
Read more