Bootable Linux USB stick: Difference between revisions

From CCP4 wiki
Jump to navigation Jump to search
No edit summary
 
(51 intermediate revisions by 2 users not shown)
Line 3: Line 3:
processing, structure solution and coot visualization. What we found:
processing, structure solution and coot visualization. What we found:
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.
# we use Fedora 23 (and higher) because its hardware support is very good. We always use the 64bit distro.
# we use 64bit Fedora (FC23 and higher) because its hardware support is very good. We tried to prepare a Ubuntu 18.04 LTS stick that boots on (U)EFI and Bios machines but that - other than the FC28-based stick - did not book on Macs.
# The stick can be booted on MacBooks as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.
# For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.
# if the WiFi does not work out-of-the-box on MacBook Pro, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl  --best --allowerasing</code>. After installing, reboot into the new kernel.  
# The stick can be booted on MacBooks (but typically not on the very newest ones) as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. If the WiFi does not work out-of-the-box on MacBook, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl  --best --allowerasing</code>. After installing, reboot into the new kernel. Essentially this is the same procedure as at [https://medium.com/@wizofe/how-i-installed-fedora-27-on-my-high-sierra-macbook-pro-15-80616909f294], except that rpmfusion-nonfree may already be installed. If the keyboard/touchpad does not work on "late 2016" MBP, you need (once) an external USB keyboard; see [https://gist.github.com/roadrunner2/1289542a748d9a104e7baec6a92f9cd7]. Also see [https://github.com/Dunedan/mbp-2016-linux/blob/master/README.md] and [https://nixaid.com/linux-on-macbookpro/].
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.
Line 13: Line 13:
'''Some more details'''
'''Some more details'''


# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a biosboot partition (1MB; required for booting bios system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition then does not fit and cannot be used on that stick, but the operating system will then work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a BIOS-boot partition (1MB; required for booting BIOS system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB or a 64GB stick; in case of 16GB the /data partition does not fit and cannot be used on that stick, but the operating system will still work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, and depending on the actual hardware, the stick may be /dev/sdb or /dev/sdc or ... !'''
# (if the installation not already does it for you) '''you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, it may be named /dev/sdb or /dev/sdc or ... !'''






== How to create a bootabled GPT-partitioned USB-stick with Fedora 23 ==
== How to create a bootabled GPT-partitioned USB-stick with Fedora ==
 
This will create a USB stick capable to
Adjust the BIOS to have UEFI enabled, and LegacyBoot / CSM disabled.
 
The steps below will then create a USB stick capable to
* EFI boot on Macs
* EFI boot on Macs
* UEFI boot on PCs
* UEFI boot on PCs
Line 28: Line 30:
* BIOS boot on medium aged hardware
* BIOS boot on medium aged hardware
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)
* not boot on really old BIOS hardware which doesn't support booting from GPT (see below)


=== GPT partition the stick using <code>gdisk</code> ===
=== GPT partition the stick using <code>gdisk</code> ===
The example shows a 32 GB Sandisk Extreme:
The example shows a 32 GB Sandisk Extreme:
  Part 1: +1G VFAT            (0700) (Windows needs it to be the first partition)  
  Part 1: +1G VFAT            (0700) (before Windows 10 version 1709, VFAT must be the first partition)  
  Part 2: +1M BIOSBOOT        (ef02)
  Part 2: +1M BIOSBOOT        (ef02)
  Part 3: +250M EFI          (ef00)
  Part 3: +250M EFI          (ef00)
Line 38: Line 41:
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.
For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.


=== install Fedora23 on an UEFI machine (UEFI enabled ; LegacyBoot / CSM disabled) ===
=== install Fedora  ===
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...).  
Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...).  
  sdX3: efi       /boot/efi
  sdX3: efi       /boot/efi
Line 44: Line 47:
  sdX5: ext4            /mnt/data
  sdX5: ext4            /mnt/data


Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the hfsplus-tools to enable Mac users to access their data on harddisk. Make sure to install the binutils RPM to get the "strings" and other useful GNU commands.
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the  
* hfsplus-tools RPM to access the HFS+ filesystem on the Mac; https://superuser.com/questions/961401/mounting-hfs-partition-on-arch-linux/1088110#1088110 explains how
* fuse-exfat RPM to access exFAT SD cards/disks,
* binutils RPM to get the "strings" and other useful GNU commands,
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor
* tcsh RPM to allow installation of CCP4.
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils fuse-exfat exfat-utils
An experimental apfs-fuse RPM can be installed to (read-only) access the new APFS filesystem on the Mac
dnf install https://forensics.cert.org/cert-forensics-tools-release-27.rpm && dnf install apfs-fuse
after which the Mac's APFS partition can be read-only mounted with e.g. <code>mkdir /mnt/sda2 && /usr/bin/apfs-fuse /dev/sda2 /mnt/sda2</code>


=== adjust the USB-stick for UEFI/EFI boot (before reboot from chroot environment) ===
=== adjust the USB-stick for UEFI/EFI boot ===
 
chroot to the installed system:
chroot /mnt/sysimage


install Fedora updates:
install Fedora updates:
  dnf -y update    # dnf on FC23 is successor to yum
  dnf -y update    # Fedora's dnf is successor to yum
 


configure grub2 for (U)EFI systems:
configure grub2 for (U)EFI systems:
Line 62: Line 70:
  grub2-mkconfig -o /etc/grub2-efi.cfg
  grub2-mkconfig -o /etc/grub2-efi.cfg
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)
(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)


Performance tuning (not strictly required):
Performance tuning (not strictly required):
Line 80: Line 87:
=== Install grub2 for BIOS boot (from chroot environment) ===
=== Install grub2 for BIOS boot (from chroot environment) ===


boot Fedora Live DVD on a BIOS machine and chroot to the stick:
Next, add the files to allow non-UEFI (legacy) boot: boot Fedora Live DVD and enter in a terminal:
  mount /dev/sdX4 /mnt
  mount /dev/sdX4 /mnt
  mount -o bind /dev /mnt/dev
  mount -o bind /dev /mnt/dev
Line 88: Line 95:


install grub2 for BIOS boot and configure it:
install grub2 for BIOS boot and configure it:
dnf -y install grub2
  rm /boot/grub2/grubenv
  rm /boot/grub2/grubenv
  grub2-install /dev/sdX
  grub2-install /dev/sdX
  grub2-mkconfig -o /etc/grub2.cfg
  grub2-mkconfig -o /etc/grub2.cfg
(on UEFI systems /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv (from package grub2-efi.rpm). The symlink has to be removed before grub2-install can finish successfully, it automatically creates a new file /boot/grub2/grubenv (as in package grub2.rpm)
explanation: on UEFI systems, /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv created by package grub2-efi. The symlink has to be removed before grub2-install can finish successfully; it automatically creates a new file /boot/grub2/grubenv.
 
The following is probably not necessary: check /etc/grub2.cfg and if needed, change at all positions:
“linuxefi” to “linux16”, and
“initrdefi” to “initrd16”


=== generating images and copies of the stick ===


check /etc/grub2.cfg and if needed, change at all positions:
Save a compressed disk image - we use the parallel gzip program called pigz:
“linuxefi” to “linux16”, and
pigz -c < /dev/sdX > usbstick.img.gz
“intrdefi” to “intrd16”
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)
 
write compressed image back to a stick:
unpigz -c usbstick.img.gz > /dev/sdX
(speed: ~ 98MB/s)
 
For creation/maintenance a bunch of sticks, a multiple port USB-HUB is a very useful tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 SanDisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions); all 12 sticks are recognised by the OS (CentOS7).


=== Problems on old BIOS hardware ===
== Hardware considerations ==
Some old computers still might not boot since they don't support booting from GPT.
This can be fixed to boot on such old hardware, however the stick afterwards cannot boot on EFI/UEFI systems anymore:


*set the boot flag (active) on the single 0xEE partition in the protective MBR by using fdisk version <= 2.22 (versions 1.23 and newer have GPT support and thus don't make changes to the protective MBR but to the GPT) (I used fdisk (util-linux-ng 2.17.2) on a ScientificLinux 6.7 machine)
Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using [https://crystalmark.info/software/CrystalDiskMark/index-e.html CrystalDiskMark] (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).
fdisk /dev/sdX
a
part 1
w


*recompute CHS values using option “h” in gdisk's expert menu:
The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read [https://en.wikipedia.org/wiki/Trim_(computing) this].
gdisk /dev/sdX
x
h
m
q


this can be reversed:
=== types of USB media ===
- toggle bootflag off in old fdisk
- recompute CHS values in gdisk


(using a sgdisk backup of the GPT should also be fine)
The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.


helpful links:
* very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. Some Mushkin USB sticks are also fast and support TRIM. USB sticks which support TRIM also usually support [https://en.wikipedia.org/wiki/S.M.A.R.T. S.M.A.R.T.], which is useful for detecting problems.
* good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the [https://en.wikipedia.org/wiki/Secure_Digital#Speed_class_rating Class 1 (A1) application performance class] since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.
* acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.
* inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.


http://www.rodsbooks.com/gdisk/bios.html#bios --> option h of gdisks experts menu did the trick!
The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to
# remove the microSD card from the USB3 adapter
# insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system
# run the <code>fstrim -v /mount/point</code> command (this takes a few seconds)
# unmount the card, and revert steps 2. and 1.


https://fedoraproject.org/wiki/GRUB_2?rd=Grub2#Updating_GRUB_2_configuration_on_BIOS_systems
This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, <code>fstrim</code> works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.


=== generating images and copies of the stick ===
=== TRIM ===
==== USB sticks ====  
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported".


Save a compressed disk image - we use the parallel gzip program called pigz:
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in some cases (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).
pigz -c < /dev/sdX > usbstick.img.gz
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)
write compressed image back to a stick:
unpigz -c usbstick.img.gz > /dev/sdX
(speed: ~ 98MB/s)


For creation/maintainance a bunch of sticks, a multiple port USB-HUB is a very usefull tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 Sandisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions) and all 12 stick are recognized by the OS (CentOS7).
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].


=== for techies: taking care of the stick ===
==== (micro)SD cards ====
These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the <code>lsblk -D</code> command reveals that the card can be TRIMmed, and <code>fstrim -v /dev/mmcXXXXX</code> works.


The following is about performance and durability of the USB stick, and reading it is not required for the functionality described above. If you don't know what TRIM is and how it relates to flash media, this section is probably not for you.
=== fill empty space of a partition with “zeroes” ===
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it.  
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should rather be TRIMmed because this does not wear out the storage cells.


To fill empty part of all partitions on the stick with “zeros”: for each partition, do (as root)
Other media: as root
  dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M  # this will stop when filesystem is full
  dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M  # this will stop when filesystem is full
  rm /mountpoint_of_partition/delete.me
  rm /mountpoint_of_partition/delete.me
However, this stresses the flash cells; they sustain only a limited number of write cycles.


This could be done after deleting large amounts of data from the USB stick, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” should instead be TRIMmed (see below) because this does not wear out the storage cells.
=== Low-level formatting of the media ===
 
Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.
==== Initializing the stick ====
==== Low-level formatting of "very good" USB sticks ====  
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:
<code>hdparm</code>'s (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)
* check if <code>hdparm -I /dev/sdX</code> reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with <code>pm-suspend</code> and wake it up again)
Line 157: Line 171:
  hdparm --user-master u --security-unlock Eins /dev/sdX</code>
  hdparm --user-master u --security-unlock Eins /dev/sdX</code>


==== TRIMming the stick ====
==== Low-level formatting of (micro)SD cards ====
TRIMming informs the USB stick firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported"; if that command also returns "Deterministic read data after TRIM" then zeros are returned upon reads of TRIMmed blocks.
In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes partition X of the card without wearing out the media.
 
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for the SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in all cases tried so far (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]).


All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', presumably because the usb-storage kernel module does not pass the ATA trim command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>].
==== Low-level formatting of other USB sticks ====
One may try to write zeroes on the whole stick
dd if=/dev/zero of=/dev/sdX bs=8192k
hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.


== installation of crystallographic software ==
== installation of crystallographic software ==
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]
* XDS and related programs (XDS-viewer, xdsstat, xdsgui, generate_XDS.INP): [http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#Linux see XDSwiki]
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Hompage]
* CCP4: download and install from [http://www.ccp4.ac.uk/download/#os=linux CCP4 Homepage]
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]
* PHENIX: download and install from [https://www.phenix-online.org/download/ Phenix-online]
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]
* SHELX: download and install from [http://shelx.uni-ac.gwdg.de/SHELX/download.php SHELX Homepage] or together with [http://www.ccp4.ac.uk/download/#os=linux CCP4]
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]
* [http://webapps.embl-hamburg.de/bundle/download.php?hkl2mp=true&sitcom=false HKL2MAP]
install required (+ additional) software from the Fedora repository:
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2

Latest revision as of 09:40, 9 May 2019

We regularly and successfully use USB sticks for courses where participants bring their own notebooks. The big benefit is that students learn Linux, and realize that they can easily use the hardware they already own. Current notebook hardware is by far fast enough for data processing, structure solution and coot visualization. What we found:

  1. the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do not have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk.
  2. we use 64bit Fedora (FC23 and higher) because its hardware support is very good. We tried to prepare a Ubuntu 18.04 LTS stick that boots on (U)EFI and Bios machines but that - other than the FC28-based stick - did not book on Macs.
  3. For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes powercfg -H off (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should.
  4. The stick can be booted on MacBooks (but typically not on the very newest ones) as well (press the alt key at the boot sound); their hardware works well with Fedora. If the WiFi does not work out-of-the-box on MacBook, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing. After installing, reboot into the new kernel. Essentially this is the same procedure as at [1], except that rpmfusion-nonfree may already be installed. If the keyboard/touchpad does not work on "late 2016" MBP, you need (once) an external USB keyboard; see [2]. Also see [3] and [4].
  5. we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks.
  6. any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync.

To make students familiar with the sticks and how to boot them, one needs 30+ minutes and a few tutors.

Some more details

  1. we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a BIOS-boot partition (1MB; required for booting BIOS system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB or a 64GB stick; in case of 16GB the /data partition does not fit and cannot be used on that stick, but the operating system will still work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field).
  2. it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use
  3. it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up.
  4. (if the installation not already does it for you) you should label the partitions and use the labels for mounting the partitions in /etc/fstab, alternatively use UUID's; don't use /dev/sda1 or the like because depending on the hardware it is booted on, after booting the stick, it may be named /dev/sdb or /dev/sdc or ... !


How to create a bootabled GPT-partitioned USB-stick with Fedora[edit | edit source]

Adjust the BIOS to have UEFI enabled, and LegacyBoot / CSM disabled.

The steps below will then create a USB stick capable to

  • EFI boot on Macs
  • UEFI boot on PCs
  • BIOS boot on newer hardware (legacy boot / CSM enabled in BIOS)
  • BIOS boot on medium aged hardware
  • not boot on really old BIOS hardware which doesn't support booting from GPT (see below)


GPT partition the stick using gdisk[edit | edit source]

The example shows a 32 GB Sandisk Extreme:

Part 1: +1G VFAT            (0700) (before Windows 10 version 1709, VFAT must be the first partition) 
Part 2: +1M BIOSBOOT        (ef02)
Part 3: +250M EFI           (ef00)
Part 4: +13500M /           (0083)
Part 5: +15000M /mnt/data   (0083)

For performance, make sure that the big partitions are aligned to 8192-sector boundaries; gdisk default is 2048-sector boundaries - the default can be adjusted. If you use the above sizes and the + sign for specifying them, this happens to work out automatically.

install Fedora[edit | edit source]

Note: in the following, whenever you see the X in sdX, you must fill in the appropriate drive letter (e.g. a or b or c or d or ...).

sdX3: efi	       /boot/efi
sdX4: ext4             / 
sdX5: ext4             /mnt/data

Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the

dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils fuse-exfat exfat-utils

An experimental apfs-fuse RPM can be installed to (read-only) access the new APFS filesystem on the Mac

dnf install https://forensics.cert.org/cert-forensics-tools-release-27.rpm && dnf install apfs-fuse

after which the Mac's APFS partition can be read-only mounted with e.g. mkdir /mnt/sda2 && /usr/bin/apfs-fuse /dev/sda2 /mnt/sda2

adjust the USB-stick for UEFI/EFI boot[edit | edit source]

install Fedora updates:

dnf -y update    # Fedora's dnf is successor to yum

configure grub2 for (U)EFI systems:

  • disable auto recognition of other installed Operating systems (specific to current computer), and
  • update grub2-efi.cfg
echo 'GRUB_DISABLE_OS_PROBER=”true”' >> /etc/default/grub
grub2-mkconfig -o /etc/grub2-efi.cfg

(last step automatically puts UUID's to recognise the boot device in grub2-efi.cfg ; default after fresh Fedora installation is the device name which might be different on another computer)

Performance tuning (not strictly required):

We use ext4 filesystems and data=writeback,nobarrier in /etc/fstab. To be able to set these options also on the / filesystem, we use tune2fs to set both as default mount options on a Linux machine where the stick (dev/sdX) is just plugged in (i.e. not booted from):

tune2fs -o journal_data_writeback,nobarrier /dev/sdX4
tune2fs -o journal_data_writeback,nobarrier /dev/sdX5

create & label vfat filesystem:

mkfs.vfat /dev/sdX1
dosfslabel /dev/sdX1 VFAT

shutdown or reboot:

exit
systemctl poweroff (or reboot for testing)

Install grub2 for BIOS boot (from chroot environment)[edit | edit source]

Next, add the files to allow non-UEFI (legacy) boot: boot Fedora Live DVD and enter in a terminal:

mount /dev/sdX4 /mnt
mount -o bind /dev /mnt/dev
mount -t proc /proc /mnt/proc
mount -t sysfs /sys /mnt/sys
chroot /mnt

install grub2 for BIOS boot and configure it:

dnf -y install grub2
rm /boot/grub2/grubenv
grub2-install /dev/sdX
grub2-mkconfig -o /etc/grub2.cfg

explanation: on UEFI systems, /boot/grub2/grubenv is a symlink to /boot/efi/EFI/fedora/grubenv created by package grub2-efi. The symlink has to be removed before grub2-install can finish successfully; it automatically creates a new file /boot/grub2/grubenv.

The following is probably not necessary: check /etc/grub2.cfg and if needed, change at all positions: “linuxefi” to “linux16”, and “initrdefi” to “initrd16”

generating images and copies of the stick[edit | edit source]

Save a compressed disk image - we use the parallel gzip program called pigz:

pigz -c < /dev/sdX > usbstick.img.gz

(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB)

write compressed image back to a stick:

unpigz -c usbstick.img.gz > /dev/sdX

(speed: ~ 98MB/s)

For creation/maintenance a bunch of sticks, a multiple port USB-HUB is a very useful tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 SanDisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions); all 12 sticks are recognised by the OS (CentOS7).

Hardware considerations[edit | edit source]

Cheap USB sticks can hold a lot of data, but when it comes to writing small files (which happens when used as the operating system disk), they are either slow from the beginning, or their write performance quickly degrades as soon as free space becomes low for the first time. We have been using CrystalDiskMark (Windows) for characterizing USB media: as a rule of thumb, the "Random write 4k blocks" disciplines should result in at least 1 MB/s, and for sequential writes the numbers should be >50 MB/s. However, CrystalDiskMark does not capture the effect of degrading write performance, which can only be mitigated by TRIMming the media - a feature that few USB sticks support (see below).

The following is about performance and durability of the USB media. If you don't know what TRIM is and how it relates to flash media, read this.

types of USB media[edit | edit source]

The suitability of the media makes a notable difference, in particular when the operating system or CCP4 is upgraded, or when e.g. a new Phenix version is installed.

  • very good: USB sticks that support TRIM. The ones known to us are the Sandisk Extreme USB sticks bought before 2017 (e.g. SDCZ80-032G 32GB for ~22€ or SDCZ80-064G 64GB for ~40€; unfortunately, these are no longer available, at least not at reasonable prices). These sticks are very fast (CrystalDiskMark scores >10MB/s for random 4K writes). The successor are the differently-named current (2018) Sandisk Extreme Pro USB sticks, but only 128 and 256 GB models exist, and they are expensive. Some Mushkin USB sticks are also fast and support TRIM. USB sticks which support TRIM also usually support S.M.A.R.T., which is useful for detecting problems.
  • good: we have verified that a USB3 adapter with (micro)SD card (separate items) is suitable. USB3 adapters are cheap (around 5€ at Amazon, like https://www.amazon.de/EX1-Kartenleser-Micro-Karte-Adapter/dp/B06XX3T219; cheaper at Ebay). microSD cards usually come with a SD adapter, and can be TRIMmed (but only in a SD slot; see below!). microSD cards ideally should be in the Class 1 (A1) application performance class since this is what is defined as requirement for use as extended operating system disk in Android smartphones. 64GB A1 microSD cards cost around 25€. In practice, slightly cheaper non-A1 microSD cards also seem to work well, but may be slower and may require more often TRIMming. The price/performance ratio of the microSD/USB3 combination is very good.
  • acceptable: USB sticks that do not support TRIM but are fast and high quality, e.g. Sandisk Extreme Go or Sandisk Ultra bought 2017 and later. However, their write performance may degrade with time (or rather, with the number of writes to a partition), and not much can be done about that.
  • inacceptable: tiny sticks like Sandisk Ultra Fit become hot, degrade fast and are unreliable. Cheap sticks are usually just too slow.

The drawback of the "good" solution above is: from time to time (e.g. once every week), a knowledgeable person may want to

  1. remove the microSD card from the USB3 adapter
  2. insert it into the SD adapter, and that into a SD slot - it is usually auto-mounted by the operating system
  3. run the fstrim -v /mount/point command (this takes a few seconds)
  4. unmount the card, and revert steps 2. and 1.

This naturally begs the question whether it wouldn't be much smarter to not use an USB adapter for microSD cards, but rather just use the SD adapter in a SD slot right away. That would have advantages - the SD card vanishes in most SD slots which is better than an USB stick sticking out, fstrim works, and possibly the SD controller is faster than the USB3 controller (depending on the notebook model). However, we tried to boot a microSD card in a SD lot and found that it doesn't boot, whereas it does boot in an USB adapter. Thus, setup of a microSD card for booting in SD slot needs to be investigated.

TRIM[edit | edit source]

USB sticks[edit | edit source]

TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which hdparm -I /dev/sdX returns "Data Set Management TRIM supported".

The wiper.sh script (part of the hdparm source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since wiper.sh also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in some cases (tested with this script).

All other methods, like the fstrim command and the discard mount option do not work for USB sticks, because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for blkdiscard.

(micro)SD cards[edit | edit source]

These support TRIM when in a SD slot, but not when inserted in a USB adapter. When in a SD slot, the lsblk -D command reveals that the card can be TRIMmed, and fstrim -v /dev/mmcXXXXX works.

fill empty space of a partition with “zeroes”[edit | edit source]

This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. USB sticks that support “Deterministic read data after TRIM” according to hdparm -I /dev/sdX, and microSD cards (which have zeroes in free space after fstrim in a SD slot) should rather be TRIMmed because this does not wear out the storage cells.

Other media: as root

dd if=/dev/zero of=/mountpoint_of_partition/delete.me bs=10M  # this will stop when filesystem is full
rm /mountpoint_of_partition/delete.me

However, this stresses the flash cells; they sustain only a limited number of write cycles.

Low-level formatting of the media[edit | edit source]

Low-level formatting restores the write performance, and removes any sensitive information (e.g. passwords) - important for workshops.

Low-level formatting of "very good" USB sticks[edit | edit source]

hdparm's (ENHANCED) SECURITY ERASE initializes the whole stick in a few seconds, and restores it to an almost factory-fresh condition, including zeroing the device. This works on all sticks where hdparm -I /dev/sdX reports "supported: enhanced erase" but if used wrongly (e.g. on the wrong device, or with the wrong options, or ...) it may brick your device, or delete valuable data! So use at your own risk:

  • check if hdparm -I /dev/sdX reports "supported: enhanced erase" and "not frozen" (to unfreeze a frozen diks, suspend your system with pm-suspend and wake it up again)
  • set disk password, and erase it (which unsets the password):
hdparm --user-master u --security-set-pass Eins /dev/sdX
hdparm --user-master u --security-erase-enhanced Eins /dev/sdX

If you tried with --security-erase-enhanced, but the disk does not support it, then an error will occur and the disk will still be locked after the failed command, and you will not be able to write anything to it! If this happens, you can unlock it with

hdparm --user-master u --security-unlock Eins /dev/sdX

Low-level formatting of (micro)SD cards[edit | edit source]

In a SD slot, blkdiscard /dev/mmcblk0pX initializes partition X of the card without wearing out the media.

Low-level formatting of other USB sticks[edit | edit source]

One may try to write zeroes on the whole stick

dd if=/dev/zero of=/dev/sdX bs=8192k

hoping that the firmware will understand this as a request for low-level formatting. This seems to work with some sticks, but takes a long time and may stress the flash cells.

installation of crystallographic software[edit | edit source]