18. Data storage in Linux

We own computers in order to work with data. We acquire it, we process it, and we store it – lots of it. We store it on hard disk drives, on solid state drives, flash memory cards, in the cloud.

Data is moved into and out of storage in discrete chunks, or blocks. As a result, we refer to data storage devices as block devices.

Block devices got some notice last time, when we looked at the /dev directory in the Linux file system. Today we'll take a more detailed look at how to work with them in Linux.

Storage is an area in which every computer is unique. That means there's little point in talking about specific devices and configurations. What I'll do instead is to use the utility programs present on most Linux systems to show how to manage them. If you have a Linux box handy, you can open up a terminal follow along with me. That will let you see how your system is set up.

We'll spend most of our time today looking at local storage: devices physically attached to our machine.

Every computer needs to have some local data storage capacity. Even cloud devices such as tablets and Chromebooks need somewhere to store their operating system, configuration, and some user data. A Linux box generally will have much more space for user data, plus the capability of replacing or expanding it. Let's see how storage works in a Linux system.

There are two general classes of storage in use today:

• Fixed devices connected directly to a data bus (SATA, PCIe) on the mainboard:
  • Mechanical hard disk drives (HDD)
  • Writable memory: solid-state drives (SSD).
• Removable devices connected temporarily through plug-in interfaces, such as USB sticks, SD cards, and DVD/CDROM optical drives.

The general procedures for handling these drives are the same for both classes, the primary difference being the need to dynamically connect and disconnect removable drives while the system is running.

Finding the block devices on a system is easy. In a terminal, type lsblk. This will produce a list of block devices, with some useful information on each one. I'm composing my notes for this topic on Linux Mint laptop. It has a single solid-state drive, and is set up for dual boot with Windows 10. When I run lsblk on it, I get the following output:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   260M  0 part /boot/efi
├─nvme0n1p2 259:2    0    16M  0 part 
├─nvme0n1p3 259:3    0 104.2G  0 part 
├─nvme0n1p4 259:4    0  1000M  0 part 
└─nvme0n1p5 259:5    0 371.5G  0 part /

It shows six devices, one disk and five partitions. For each device it gives information that's useful to me:

• The device name in the system. This is descriptive, based on the type of device.
• What type of device it is: disk or partition.
• Its size.
• Mount point in the Linux file system. Note that the Windows partitions are not mounted.

This works for all block devices that are present – fixed and removable.

Before a device can be used by the system, it must be properly prepared. This is a two-step process:

1. Partitioning: create logical partitions on the drive. This divides the drive into sections, each of which can be configured to work as discrete unit, to be configured for a particular purpose.
2. Formatting: install a supported file system in each partition. It's the file system that organizes data within the partition and provides the means of reading and writing it.

Why is he formatting process necessary, given that the device already is mapped to /dev subdirectory by its driver?

• The device driver that sits behind the /dev directory provides the interface only to the drive itself.
• Creating a partition and formatting it with a file system sets up an additional interface that provides access to the storage medium on the drive.
• How this interface does its job depends on which file system is installed. Currently, the most-used file system for mass storage in Linux is ext4. USB sticks and SD cards commonly default to exfat.
• We'll look at file systems in detail at another time. For now, I'll just note that, with Linux, there are many to choose from – in contrast to Windows and OSX.

In Linux, disk partitions are managed by using a partition editor:

• Text-based parted in a terminal
• Graphical gparted in a GUI.

A quick way to view the partitions currently on your system is to open a terminal and type: sudo parted -l. This will produce a list of partitions, along with their size and installed file system. You also can view them by running gparted. When I do that on my dual-boot laptop, it shows all the partitions, including those for Windows:

Model: KXG6AZNV512G TOSHIBA (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size    File system  Name                          Flags
 1      1049kB  274MB  273MB   fat32        EFI system partition          boot, hidden, esp
 2      274MB   290MB  16.8MB               Microsoft reserved partition  msftres
 3      290MB   112GB  112GB   ntfs         Basic data partition          msftdata
 5      112GB   511GB  399GB   ext4
 4      511GB   512GB  1049MB  ntfs         Basic data partition          hidden, diag

The basic partitioning process is:

1. Choose the device you want to partition.
2. Add a partition, specifying its size.
3. Format the partition with an appropriate file system.
4. Optional: specify a mount point in the Linux file system

Once a partition is created and formatted, it is used by mounting it on a directory in the Linux file system.

Mounting a partition does two things:

1. It maps the partition's interface to a directory in the file system. This provides the read/write access necessary to use it.
2. The interface is matched to the file system in which the partition is formatted: ext4, btrfs, etc. That makes all the features of the file system available for use.

You can get a list of currently-mounted partition by using the mount command. Simply type mount in a terminal, and you'll see which partitions are mounted where.

There are multiple ways to mount a partition:

1. Issue the mount command at the command line, specifying the mount point and file system type. The mount will be effective until either an umount command is issued, or the system is shut down.
2. Add an entry to the /etc/fstab file. This is used at startup to mount partitions on fixed devices.
3. If your host runs systemd, add a systemd unit file. This looks more difficult than it is. I was able set up a file to mount an SSD that I added to my Docker server on my first try.

Linux comes with built-in tools for using data storage on another host on a network. It's a sizeable topic for another day, but there are some things worth mentioning now.

Using network storage is similar in many ways to local storage. In particular, remote devices are mounted to a directory in the Linux file system, making their use transparent to the user.

There are two main methods of working with remote storage in Linux:

• NFS: Network File System. Native to Linux, NFS creates network shares that are equivalent to disk partitions. A network share is mounted to a directory – the same process as internal storage. (See Beginners guide to mount NFS share in Linux with examples) for more information.
• Samba: Linux applications for dealing with SMBs, the Windows method of sharing over a network.

Return to series index