0

Problem Overview

I recently upgraded my remote server contract with IONOS, increasing my hard drive space from 8GB to 80GB. I have an Ubuntu OS running bash.

I then went about extending my working partition, following a tutorial here: https://www.ryadel.com/en/resize-extend-disk-partition-unallocated-disk-space-linux-centos-rhel-ubuntu-debian/

All was OK, I wrote a new partition map, then rebooted my system. I waited a minute or two and then attempted to ssh into my server as usual. Problem. My ssh connection hangs, until eventually exiting with a time out.

Solution Attempts

At first, I reasoned the process of rebooting after a partition map change may take some time, and this was the cause of the timeout. After several more ssh attempts, this did not seem likely.

I used a 'KVM Console' provided in my IONOS console - here, the shell is in a state of (initramfs).

In attempting to diagnose the issue, I have tried the following:

  1. Running: fsck /dev/sda1 Result: /dev/sda1: clean, 312/124672 files, 26890/124672 blocks
  2. Running: fsck /dev/sda1 Result: fsck: error 2 (No such file or directory) while executing fsck.ext2 for /dev/sda2
  3. Running: blkid Result:

/dev/sda1: UUID="longString" TYPE="ext4" PARTUUID="520f1760-01"

/dev/sda2: PARTUUID="520f1760-02"

  1. Running all of the following commands returns sh: command name: not found. These are:
  • vgdisplay -v vg00
  • parted -l /dev/sda
  • free -m
  • cfdisk
  • lvdisplay -v
  • fdisk /dev/sda
  • pvresize /dev/sda2
  1. The output of cat proc/partitions is:
major minor #blocks name
8 0 83886080 sda
8 1 498688 sda1
8 2 83386368 sda2
11 0 1048575 sr0

From the above, I am confused why (2) returns no such file or directory - the entry sda2 is listed under the directory dev.

  1. The output of cat /proc/cmdline is: BOOT_IMAGE=/vmlinuz-5.4.0-132-generic root=/dev/mapper/vg00-lv01 ro apparmor=0
  2. After entering lvm and then vgscan -ccc, the output is:
....
Start of output not visible in terminal window due to no scrolling
....
filter caching bad /dev/loop5
Opened /dev/loop6 RO O_DIRECT
/dev/loop6: size is 0 sectors
Closed /dev/loop6
/dev/loop6: Skipping: Too small to hold a PV
filter caching bad /dev/loop6
Opended /dev/loop7 RO O_DIRECT
/dev/loop7: size is 0 sectors
Closed /dev/loop7
/dev/loop7: Skipping: Too small to hold a PV
filter caching bad /dev/loop7
Will scan 3 devices skip 0
Checking fd limit for num_devs 3 want 35 soft 1024 hard 4096
Scanning 3 devices for VG info
Scanning submitted 3 reads
Processing data from device /dev/sda 8:0 fd 4 block 0x55b511a17cd0
Scan filtering /dev/sda
/dev/sda: using cached size 167772160 sectors
/dev/sda: Skipping: Partition table signature found
filter caching bad /dev/sda
/dev/sda: Not processing filtered
Processing data from device /dev/sda1 8:1 fd 5 block 0x55b511a17d10
Scan filtering /dev/sda1
/dev/sda1: using cached size 997376 sectors
/dev/sda1: Device is a partition, using primary device sda for mpath component detection
/dev/sda1: using cached size 997376 sectors
filter caching good /dev/sda1
/dev/sda1: No lvm label detected
Processing data from device /dev/sda2 8:2 fd 6 block 0x55b511a17d50
Scan filtering /dev/sda2
/dev/sda2: using cached size 166772736 sectors
/dev/sda2: Device is a partition, using primary device sda for mpath component detection
/dev/sda2: using cached size 166772736 sectors
filter caching good /dev/sda2
Label checksum incorrect on /dev/sda2 - ignoring
/dev/sda2: No lvm label detected
Scanned devices: read errors 0 process errors 0 failed 0
Found VG info for 0 VGs
Obtaining the complete list of VGs to process
No volume groups found
Unlocking /run/lock/lvm/P_global
_undo_flock /run/lock/lvm/P_global
Dropping VG info
lvmcache has no info for vgname "#orphans_lvm2" with VGID #orphans_lvm2.
lvmcache has no info for vgname "#orphans_lvm2".
lvmcache: Initialised VG #orphans_lvm2.
Completed: vgscan -vvv
  1. The directory /etc/lvm/backup exists and contains: vg00 The directory /etc/lvm/archive exists and contains: vg00_00000-1647277590.vg vg00_00001-1228658393.vg

(3) and (5) give me hope - the location seems to be recognised, what would this suggest ?

Specific Steps Before Reboot

In summary, the steps I took before rebooting my system were:

  1. ran fdisk /dev/sda and noted the start and end points of the file systems by entering p.
  2. Deleted the file system map by entering d and then selecting sda2 with 2
  3. Created a new partition map by entering n. Setting the partition type to primary.
  4. I then entered the start and end locations for the new partition, as noted in step (1).
  5. I changed the partition type, by entering t, and selecting the 2nd partition by entering 2.
  6. I specified the partition type to be 'Linux LVM' by entering the HEX code 8e.
  7. Before writing to the disk, I ensured start and end points were correctly listed by entering p. The start point matched that of the original partition. The end point matched that of the disk end point.
  8. I wrote the partition map to disk by entering w.
  9. I reboot the system with reboot.

The result of running lvm p prior to partition map changes was:

lvm p output

At this point I am not sure how to proceed - I have encountered a file system issue before and was troubled at the prospect of loosing all my files. Ultimately, in that case, the files were still present. From that experience I am restraining my assumption all is lost.

Does anyone have any suggestions, or tips to offer in terms of debugging this situation ? Please feel free to ask if you would like extra information regarding my setup.

Update

I have been able to boot into a knoppix CD on my remote server. Here, I have run fdisk -l which outputs:

Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram2: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram3: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram4: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram5: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram6: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram7: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram8: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram9: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram10: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram11: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram12: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram13: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram14: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/cloop0: 1.83 GiB, 1960312832 bytes, 3828736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/cloop1: 9.63 GiB, 10335027200 bytes, 20185600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 1.45 GiB, 1560817664 bytes, 381059 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sda: 80 GiB, 85899345920 bytes, 167772160 sectors
Disk model: Virtual disk    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x520f1760

Device     Boot  Start       End   Sectors  Size Id Type
/dev/sda1  *      2048    999423    997376  487M 83 Linux
/dev/sda2       999424 167772159 166772736 79.5G 8e Linux LVM

I feel the final output lines, displaying the partition map for sda1 and sda2, are of interest. I believe the type of sda2 is correct as 8e (a Linux LVM), and the Start value correctly falls after the End of sda1.

Update II

Before attempting the below steps, I created a snapshot for backing up the system to its current state. I have now returned to this snapshot.

Attempting to restore from the /etc/lvm/backup/vg00 file (initramfs), first I ran pvcreate --restorefile /etc/lvm/backup/vg00 --uuid R5VWXg-jamB-5dWM-PpwY-7a49-LRz7-Vrvdl2 /dev/sda2. This returned:

WARNING: Couldn't find device with uuid `R5VWXg-jamB-5dWM-PpwY-7a49-LRz7-Vrvdl2. 
Failed to clear hint file.
Physical volume "/dev/sda2" successfully created.

Then, I ran vgcfgrestore --file /etc/lvm/backup/vg00 which returned:

No command with matching syntax recognised.
Nearest similar syntax command has syntax:
vgfcgrestore -f:--file String VG
Restore VG metadata from specified file.

There seems to be an issue here.

Improve this question
9
  • cat /proc/partitions? lvm vgs? boot a rescue system and check / chroot from there? Commented Feb 1, 2024 at 20:14
  • I have added the output of cat /proc/partitions to my question. Entering lvm vgs returns nothing. Booting into recovery mode prints Begin: Running /scripts/local-top ... Volume group "vg00" not found Cannot process volume group vg00 followed by ` Gave up waiting for suspend/resume device. I am then returned to the (initramfs)` shell - do you suggest I can chroot from here ? Commented Feb 2, 2024 at 16:44
  • not familiar with ionos - does your hoster not offer a rescue system for servers? you should check if the partition offset is really correct, or the partition got wiped, or what. hexdump -C /dev/sda2 | less and see if you find LVM metadata (ascii). but it's tiresome to do this with initramfs shell, so a proper rescue system would be good Commented Feb 2, 2024 at 17:34
  • Try lvm vgscan, then lvm pvs. It should show sda2 as a LVM physical volume - if not, you could read dmesg | less to see if there are any error messages about partition sda2. You might also try lvm vgscan -vvv, but be warned: the output will be quite long. If you can capture it and put it into an accessible pastebin somewhere, that would be great. Commented Feb 2, 2024 at 17:41
  • @frostschutz reading on the IONOS site, it seems they offer 'Dedicated Servers' with the ability to boot into a rescue system but for my product, a 'Virtual Private Server' it seems I need to contact customer support and have them boot the system into what they refer to as 'Repair Mode'. I am waiting to get through to customer service and will update when I make it through. Unfortunately (initramfs) does not feature hexdump or less. Commented Feb 2, 2024 at 19:21

1 Answer 1

Reset to default
0

You should examine the LVM VG metadata backup file /etc/lvm/backup/vg00 and find the original PV UUID of /dev/sda2 from there. It is a text file, and the PV UUID should be in a location like this: ([...] indicates some lines omitted for brevity)

[...]
vg00 {
    [...]
    physical_volumes {
    
        pv0 {
            id = "xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx"
            device = "/dev/sda2"   # Hint only

Once you know the PV UUID, you can use the backup file and the UUID to restore the PV UUID like this: (commands prefixed with lvm for use in initramfs environment; if you have extracted the VG metadata backup file from initramfs and do this in Knoppix, you can omit the lvm prefixes)

lvm pvcreate --restorefile /etc/lvm/backup/vg00 --uuid xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx /dev/sda2

Once the PV UUID is restored, you can restore the rest of VG metadata with:

lvm vgcfgrestore --file /etc/lvm/backup/vg00 vg00

After this, the VG should be good for activation:

lvm vgchange -ay vg00

If the VG activates successfully, and the filesystem within it can be mounted (with e.g. mount /dev/mapper/vg00-lvol1 /mnt), you should now be able to boot normally.

Once the system is running normally, you'll need two commands as root to achieve your original goal:

pvresize /dev/sda2

After this, pvs should indicate the sda2 PV is now successfully resized and vgs should indicate there is now plenty of unallocated space in vg00. To finally make use of it:

lvextend -r -l +100%FREE /dev/mapper/vg00-lvol1

and now df should indicate the root filesystem has plenty of free space again.

There is the command growpart (part of the cloud-guest-utils package in Debian, might be packaged separately as cloud-utils-growpart or just growpart in other distributions) which is specifically made to extend partitions safely and quickly, usually with no rebooting required.

In this specific case, the extension could have been achieved with just three commands:

growpart /dev/sda 2
pvresize /dev/sda2
lvextend -r -l +100%FREE /dev/mapper/vg00-lvol1
Improve this answer
4
  • Amazing - thankyou for the direction. I have found the PV UUID of sda2. I have followed the steps you list above and have added their result to my post above. Commented Feb 3, 2024 at 20:38
  • Sorry, it looks like vgcfgrestore needs the name of the VG to restore in addition to the backup file. Edited. Commented Feb 3, 2024 at 21:48
  • Absolute legend - boot successful and, following your tips on resizing the partition, the full 80GB is now in use! Many thanks Commented Feb 4, 2024 at 9:52
  • Now just wait a day or two (in case others have something to fill in), then click on the hollow green checkmark next to the beginning of the answer to mark it as the accepted answer if you're happy with it. When restoring corrupted LVM metadata from backups, the use of pvcreate first is a bit un-intuitive, but it's described in the vgcfgrestore man page, if you read it carefully. Commented Feb 4, 2024 at 10:59

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.