1,328
edits
Bootable Linux USB stick: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| (15 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
processing, structure solution and coot visualization. What we found: | processing, structure solution and coot visualization. What we found: | ||
# the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk. | # the sticks should be fast USB3 (very good results with SANdisk Extreme 32GB or bigger; physically small sticks like Kingston DataTraveler look nicer but are slow). The computers do ''not'' have to be recent, nor do they have to have USB3 ports. USB2 ports support up to 30MB/s, but only USB3 sticks deliver this! With a bit of tuning (below) the stick feels as fast as a local harddisk. | ||
# we use Fedora (FC23 and higher) because its hardware support is very good. We | # we use 64bit Fedora (FC23 and higher) because its hardware support is very good. We tried to prepare a Ubuntu 18.04 LTS stick that boots on (U)EFI and Bios machines but that - other than the FC28-based stick - did not book on Macs. | ||
# For Windows clients (press F11 or F12 or sometimes F9 or F10 for the boot menu; if that does not work press F2 or DEL for the BIOS menu and change the boot order), one has to make sure that "fast boot" (or "fast startup") is disabled (or Shift is pressed while shutting Windows down), and sometimes <code>powercfg -H off</code> (as Administrator in a console window) is additionally required; otherwise the USB stick may not boot. Occasionally we find a computer that does not boot from the stick because the BIOS screen can not be reached (due to unknown BIOS password; happens with machines belonging to institutions which administer them centrally) or some such, but 19 out of 20 work as they should. | |||
# | # The stick can be booted on MacBooks (but typically not on the very newest ones) as well (press the <code>alt</code> key at the boot sound); their hardware works well with Fedora. If the WiFi does not work out-of-the-box on MacBook, connect temporarily to the Internet by other means (ethernet cable, WiFi via USB key, tether to your phone via Bluetooth), become root, and install the latest kernel and tools with <code>dnf install -y akmods kernel kernel-devel broadcom-wl --best --allowerasing</code>. After installing, reboot into the new kernel. Essentially this is the same procedure as at [https://medium.com/@wizofe/how-i-installed-fedora-27-on-my-high-sierra-macbook-pro-15-80616909f294], except that rpmfusion-nonfree may already be installed. If the keyboard/touchpad does not work on "late 2016" MBP, you need (once) an external USB keyboard; see [https://gist.github.com/roadrunner2/1289542a748d9a104e7baec6a92f9cd7]. Also see [https://github.com/Dunedan/mbp-2016-linux/blob/master/README.md] and [https://nixaid.com/linux-on-macbookpro/]. | ||
# we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks. | # we just install CCP4 and whatever else we need (XDS, Phenix, Chimera, ..), and then dd or ddrescue (on a machine with USB3 ports) an image of that stick to all other sticks. | ||
# any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync. | # any number of bells and whistles could be added to this, like clients sending their hostnames to a server after booting, and accepting updates by rsync. | ||
| Line 13: | Line 13: | ||
'''Some more details''' | '''Some more details''' | ||
# we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a BIOS-boot partition (1MB; required for booting BIOS system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB stick; the /data partition | # we always create a few-GB FAT32 partition because that makes file exchange with Windows and Macs very simple. The FAT32 partition should be the first partition on the stick; 2GB is enough for us. Then comes a BIOS-boot partition (1MB; required for booting BIOS system from a GPT disk), an efi partition (e.g. 250MB; required for UEFI/EFI boot on PC's/MAC's) and the Linux partition, with 13.9GB, so that the sum of the four partitions is slightly below 16GB. The third partition is then a 16.1GB /data partition. This scheme has the advantage that the image of the stick can just as well be copied (with dd or better ddrescue) to a 16GB or a 64GB stick; in case of 16GB the /data partition does not fit and cannot be used on that stick, but the operating system will still work on the small stick just as well. This requires that the /data partition is not fsck'ed automatically from /etc/fstab (0 in the 6th field). | ||
# it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use | # it is a good idea to give easy root access to the one user you create because certainly some packages will have to be installed or updated when the stick is in use | ||
# it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up. | # it is a good idea to save an image of the stick whenever you made a successful change; otherwise you might need to start from scratch if you mess something up. | ||
| Line 48: | Line 48: | ||
Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the | Not much to say about the installation of programs (CCP4 ?, Phenix ?, XDS ?, ...?). You should also install the | ||
* hfsplus-tools RPM to access Mac | * hfsplus-tools RPM to access the HFS+ filesystem on the Mac; https://superuser.com/questions/961401/mounting-hfs-partition-on-arch-linux/1088110#1088110 explains how | ||
* fuse-exfat RPM to access exFAT SD cards, | * fuse-exfat RPM to access exFAT SD cards/disks, | ||
* binutils RPM to get the "strings" and other useful GNU commands, | * binutils RPM to get the "strings" and other useful GNU commands, | ||
* atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor | * atop, a better top; lbzip2, a parallel bzip2; xxdiff, a diff GUI; nedit, editor | ||
* tcsh RPM to allow installation of CCP4. | * tcsh RPM to allow installation of CCP4. | ||
dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils fuse-exfat exfat-utils | dnf -y install tcsh xterm tk qt-x11 xxdiff atop nedit lbzip2 hfsplus-tools binutils fuse-exfat exfat-utils | ||
An experimental apfs-fuse RPM can be installed to (read-only) access the new APFS filesystem on the Mac | |||
dnf install https://forensics.cert.org/cert-forensics-tools-release-27.rpm && dnf install apfs-fuse | |||
after which the Mac's APFS partition can be read-only mounted with e.g. <code>mkdir /mnt/sda2 && /usr/bin/apfs-fuse /dev/sda2 /mnt/sda2</code> | |||
=== adjust the USB-stick for UEFI/EFI boot === | === adjust the USB-stick for UEFI/EFI boot === | ||
| Line 107: | Line 110: | ||
pigz -c < /dev/sdX > usbstick.img.gz | pigz -c < /dev/sdX > usbstick.img.gz | ||
(time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB) | (time: ~180 secs speed: ~175MB/s, size of image: 1.8 GB) | ||
write compressed image back to a stick: | write compressed image back to a stick: | ||
unpigz -c usbstick.img.gz > /dev/sdX | unpigz -c usbstick.img.gz > /dev/sdX | ||
(speed: ~ 98MB/s) | (speed: ~ 98MB/s) | ||
For creation/ | For creation/maintenance a bunch of sticks, a multiple port USB-HUB is a very useful tool. E.g. the Raid Sonic Icy Box IB-AC6113 accepts 12 SanDisk Extreme sticks at once (port 7 has to stay empty due to spatial restrictions); all 12 sticks are recognised by the OS (CentOS7). | ||
== Hardware considerations == | == Hardware considerations == | ||
| Line 140: | Line 144: | ||
TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". | TRIMming informs the firmware about blocks of the filesystem that do not contain file data. This is available for USB sticks for which <code>hdparm -I /dev/sdX</code> returns "Data Set Management TRIM supported". | ||
The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in | The <code>wiper.sh</code> script (part of the <code>hdparm</code> source distribution) TRIMs ext3/ext4/xfs filesystems, but does not reproducibly seem to work for our preferred SanDisk Extreme 32GB stick if the filesystem is mounted. This is probably due to the fact that this stick has a limitation of max 65536 blocks in one TRIM command (https://sourceforge.net/p/hdparm/bugs/63/). Since <code>wiper.sh</code> also works for unmounted filesystems, and the mapping of unused space is then obtained with a different tool, one should try with an unmounted filesystem as well - this worked for us in some cases (tested with [http://lightrush.ndoytchev.com/random-1/checkiftrimonext4isenabledandworking this script]). | ||
All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>]. | All other methods, like the <code>fstrim</code> command and the <code>discard</code> mount option '''do not work for USB sticks''', because the usb-storage kernel module does not pass the ATA TRIM command through the USB bridge and controller to the device. The same goes for [http://unix.stackexchange.com/questions/97143/utility-to-trim-unallocated-space-on-drive <code>blkdiscard</code>]. | ||
| Line 149: | Line 153: | ||
=== fill empty space of a partition with “zeroes” === | === fill empty space of a partition with “zeroes” === | ||
This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. | This could be done after updating the software or data on the USB media, and before saving and compressing an image of it. | ||
USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should be TRIMmed because this does not wear out the storage cells. | USB sticks that support “Deterministic read data after TRIM” according to <code>hdparm -I /dev/sdX</code>, and microSD cards (which have zeroes in free space after <code>fstrim</code> in a SD slot) should rather be TRIMmed because this does not wear out the storage cells. | ||
Other media: as root | Other media: as root | ||
| Line 168: | Line 172: | ||
==== Low-level formatting of (micro)SD cards ==== | ==== Low-level formatting of (micro)SD cards ==== | ||
In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes the card. | In a SD slot, <code>blkdiscard /dev/mmcblk0pX</code> initializes partition X of the card without wearing out the media. | ||
==== Low-level formatting of other USB sticks ==== | ==== Low-level formatting of other USB sticks ==== | ||