Raspberry Pi Overlay Root Filesystem

I copied this from https://yagrebu.net/unix/rpi-overlay.md so I don't loose it. I have done a lot of network booting and overlay file system work this last year and these are the best directions for overlay file system and don't want to loose them. I will rewrite them to include my own subtle changes at some point, and publish network booting and network booting + overlay file system (which is a total crazy mess with no directions).

Raspberry Pi Overlay Root Filesystem

This document describes a method to protect the root filesystem from writes while still allowing all applications to function as normal while writing to a temporary Overlay filesystem. Figuring this out would have been impossible for me without this excellent post by ejolson on the RPI.org forums.

For my installation I used a RPI 3 and the latest Raspbian Stretch Lite image (2017-11-29). I do not know if these instructions will work without modification on earlier RPI hardware.

Note: When the overlay filesystem is in place your RPI will function as usual, but any data generated after startup is only saved in RAM and will be lost upon reboot.

Let's Begin

First we need to create a initramfs image that contains the overlay module and a boot script to mount our root partition with the overlay. (All this could of course be compiled into the kernel image, but initramfs-tools(8) is an easier way to learn about the early init-process.)

echo overlay >>/etc/initramfs-tools/modules

Place the following boot script in /etc/initramfs-tools/scripts/overlay (download):

# Local filesystem mounting                     -*- shell-script -*-

#
# This script overrides local_mount_root() in /scripts/local
# and mounts root as a read-only filesystem with a temporary (rw)
# overlay filesystem.
#

. /scripts/local

local_mount_root()
{
        local_top
        local_device_setup "${ROOT}" "root file system"
        ROOT="${DEV}"

        # Get the root filesystem type if not set
        if [ -z "${ROOTFSTYPE}" ]; then
                FSTYPE=$(get_fstype "${ROOT}")
        else
                FSTYPE=${ROOTFSTYPE}
        fi

        local_premount

        # CHANGES TO THE ORIGINAL FUNCTION BEGIN HERE
        # N.B. this code still lacks error checking

        modprobe ${FSTYPE}
        checkfs ${ROOT} root "${FSTYPE}"

        # Create directories for root and the overlay
        mkdir /lower /upper

        # Mount read-only root to /lower
        if [ "${FSTYPE}" != "unknown" ]; then
                mount -r -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /lower
        else
                mount -r ${ROOTFLAGS} ${ROOT} /lower
        fi

        modprobe overlay

        # Mount a tmpfs for the overlay in /upper
        mount -t tmpfs tmpfs /upper
        mkdir /upper/data /upper/work

        # Mount the final overlay-root in $rootmnt
        mount -t overlay \
            -olowerdir=/lower,upperdir=/upper/data,workdir=/upper/work \
            overlay ${rootmnt}
}

Create the initramfs image:

update-initramfs -c -k $(uname -r)

It will be placed in /boot/initrd.img<kernel_version> - you can rename it to match your kernel file if you want.

Add the following to /boot/config.txt - where initrd7.img is the initramfs image you created in the previous step:

kernel=kernel7.img
initramfs initrd7.img

And finally, add boot=overlay to the beginning of /boot/cmdline.txt.

This should do it. Reboot and keep your fingers crossed.

Kernel Panic

If the OS doesn't come back up, there is probably a typo somewhere, or you missed something.

You can remove boot=overlay from cmdline.txt to boot as normal.

A serial connection can be really handy when trying to troubleshoot the startup process. USB TTL console cables are dirt cheap, just make sure you get one that outputs 3.3V on the TX line so that you don't fry your RPI. Use screen(1)'s buffer to scroll back and try to figure out where it went wrong.

The documentation is also quite good. Read the man page for initramfs-tools(8) and the default boot scripts themselves:

less /usr/share/initramfs-tools/init
less /usr/share/initramfs-tools/scripts/local

Finishing Touches

Read-Only /boot

The /boot filesystem is still mounted rw. You can protect it as well by adding ro to the boot partitions mount options in /etc/fstab:

PARTUUID=72a9e9a9-01   /boot   vfat   defaults,ro   0   2

overctl

Since I would be updating the filesystem on my RPI regularly, I wanted a secure way to enable and disable the overlay. overctl has the following features:

Usage: overctl [-h|-r|-s|-t|-w]
   -h, --help     This message
   -r, --ro       Set read-only root with overlay fs
   -s, --status   Show current state
   -t, --toggle   Toggle between -r and -w
   -w, --rw       Set read-write root

Place the script in /usr/local/sbin and mark the file as executable. You will also need to create the following files containing the cmdline.txt options that you wish to toggle between:

/boot/cmdline.txt.orig
/boot/cmdline.txt.overlay

motd(5)

I figured I could use a reminder of which state the RPI is in. So I cobbled together this motd(5) script:

#!/bin/sh

str=$(mount | grep ' on / ')

if echo $str | grep -q 'overlay'; then
        printf "\n------ INFO: / MOUNTED WITH OVERLAY ------\n\n"
elif echo $str | grep -q 'rw'; then
        printf "\n++++++ INFO: / MOUNTED READ-WRITE ++++++\n\n"
else
        printf "\n!!!!!! WARNING: / UNKNOWN STATE !!!!!!\n\n"
fi

Place the code above in /etc/update-motd.d/80-overlay and make sure the file is executable.

Get MAC Address for a Pi Cluster

There are a lot of ways to get the MAC address for a Raspberry Pi. For the Super Pi there were a lot of Pi, 1050 to be specific (two of them are dead on arrival and we disabled 24 of them to get the exact number we wanted, 1024), we looked at the DHCP log files. Roy and Mike spent a few days doing this. Think about it, if you spend one minute times 1050, that's 17.5 hours! I'm surprised they didn't go crazy but they seemed to be having fun. For the Mini Super Pi, I need to know the order.

At this point I'm going to digress a little. When building a large Raspberry Pi cluster at some point it's worth network booting. I'd say after 8 the cost of network booting is worth it. Just the time and expense of buying and cloning eight network cards. Network booting a Pi isn't terribly difficult, it's doing it consistently with a lot of them that's the problem. The netboot on a Pi times out after a short period of time, it isn't implemented 100% to spec, I could go on but these are things you can find out with some research. Although it's buried so if people want me to document these issues let me know in the comments. If there are any problems netbooting the Pi can be dead in a big cluster with no way to reboot it except reboot the entire thing. It helps to have a static IP address. Best way of doing this is by using DHCP (yeah, that's confusing) but map the MAC address to an IP address. Oh, also if you have enough Raspberry Pi, the MAC address may not be unique. We have two Pi with the same MAC addresses. Most clusters won't run into this. Also, network booting while not difficult requires the use of overlay file system. The problem comes when combining the two which is rather difficult. I got overlay file system working. Roy merged netboot and overlay and Vijay did this with Oracle Linux. There are a few other options netboot and redirect all writes to separate files and hope you got all them, create an NFS mount for each Pi (yeah, no). The only downside to netboot + overlay file system is you can't run Docker, which means you can't run K8 because Docker uses overlay and you can't use overlay on top of overlay.

OK, back to the reason for this blog post. My new system for obtaining the MAC address is as follows:

1. Install Raspbian on an SD card. I used belenaEtcher on Mac just because it's easy.
2. Copy the following Python script to the Pi and call it pimac.py

import socket

import time

import threading

import datetime

import uuid

import random

# This is needed so we get a network interface

time.sleep(20)

# Global variables

MacAddress = hex(uuid.getnode())

print(MacAddress)

print(type(MacAddress))

if MacAddress.endswith('L'):

    MacAddress = MacAddress[:-1]

    print(MacAddress)

client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP

client.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

client.bind(("", 2222))

server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

server.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

server.settimeout(0.2)

server.bind(("", 3333))

hostname = socket.gethostname()    

IPAddr = socket.gethostbyname(hostname)    

print("Your Computer Name is:" + hostname)    

print("Your Computer IP Address is:" + IPAddr)

def SendMessage(port, message):

    data = message

    endodeddata = data.encode()

    server.sendto(endodeddata, ('<broadcast>', port))

    print("message sent!" + data)

while True:

    SendMessage(3333, MacAddress)

    time.sleep(2)

3. Create a cron job:

sudo crontab -e

4. Add this line to the end of the file:

@reboot python /home/pi/pimac.py &

5. Copy the following Python script to your desktop and run it. I'm running it in Visual Studio Code.

import datetime

import os

import signal

import socket

import threading

import time

import re

import uuid

if os.name == 'nt':

    signal.signal(signal.SIGINT, signal.SIG_DFL)

server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

server.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

hostname = socket.gethostname()

IPAddr = socket.gethostbyname(hostname)

print("Your Computer Name is: {0}\n".format(hostname))

print("Your Computer IP Address is: {0}\n".format(IPAddr))

def waitForPosts():

  print('Ready')

addresses = {}

class Listen(threading.Thread):

    def __init__(self, port):

        threading.Thread.__init__(self)

        self.port = port

        self.UIDCount = 0

    def run(self):

        client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP

        client.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

        client.bind(("", self.port))

        while True:

            data, addr = client.recvfrom(1024)

            mac = data

            if mac not in addresses:

                addresses[mac] = mac

                val = mac.decode()

                print (':'.join(re.findall('..', '%012x' % int(val, 16)))) 

publish_thread = Listen(3333)

publish_thread.start()

waitForPosts()

Note: If you decide you want to remove the dictionary in the code above to allow duplicate MAC addresses to print out, which his a great debugging tool and since UDP is not guaranteed is a perfectly fine way to go, then you might want to remove duplicates. To do this in Visual Studio Code this is one way I found to do it:

1. Control+F
2. Toggle "Replace mode"
3. Toggle "Use Regular Expression" (the icon with the .* symbol)
4. In the search field, type ((^[^\S$]*?(?=\S)(?:.*)+$)[\S\s]*?)^\2$(?:\n)?
5. In the "replace with" field, type $1
6. Click  ("Replace All").

6. So now insert the SD card, plug the Pi into network, and power and wait. The output on your desktop will be a list of your MAC addresses in a known order and it is easy to know if one is dead and which one.

Raspberry Pi Super Computer STL Files Published on Thingiverse

Finally. I'm sorry it took so long. Here it is, the files to print your very own 2U rack mountable Raspberry Pi holder for 21 Raspberry Pi 3B+. Let me know if you have any questions or problems. Just because we made over 50 of these, doesn't mean it's easy.

https://www.thingiverse.com/thing:3958586

Bringing Pi Home

Very Large Raspberry Pi Cluster - Part VI

At this point I've shown how the all the Pi will be held in place to make a 2U rack, and next is the overall structure but that will have to wait until next week. So for this post I wanted to talk a little about the software and what we're going to run on all these Pi. A lot of data will be processed and sent to the cluster. So here's the high level of the software stack:

• Each Pi will network boot from a central server running Oracle Linux
• The boot image will be running Oracle Linux
• It will be running Kubernetes and Docker
• There will be a very large video wall run by a Windows server
• Everything will be written in Java
• Gluon is helping

What exactly it will do will remain a mystery until you come see it in person. So sign up for Code One and come see it!

Very Large Raspberry Pi Cluster - Part V

Last week I said I'd talk more about this part. So here is what the main bracket of the Pi holder looks like and the two parts individually.

The print time for the Pi holder is 98 minutes. It's a lot but it's a lot of plastic too. I print 12 of them at time, at 15% infill, with a .8mm print head, no rafts or supports. I can print one rack in 24 hours. You can do the math from there how many days the printers will be running. I spent a few days optimizing the amount of filament and the print speed with the strength needed. It is over engineers and will not break when being shipped. It is easy to assembly and requires very few parts. The reason for the curves is because it prints faster since the print head doesn't have to slow down when it approaches a corner. I was going to make the front two exes that are square round but it only saved 30 seconds print time and looks better being square when fully assembled.

Here are some views of the models from different angles since these are two separate pieces. Here are two views with the two pieces combined. You will notice a square rod coming out of that slot.

That square rod is a piece of aluminum extrusion. Specifically it is 80/20 series 10 that is .5in x 1in x 18in. I had 16.35mm milled off the end and the ends tapped for 10-32 by a local machine shop (GSFAB in Santa Cruz).

So I print 21 of the Pi Holders and two of them end caps (mirror of one another). Bolt an end cap to one end. Add one plastic threaded insert to the top hole, bolt a Pi on (the bottom holes just sort of clip on), slide it on the 80/20, repeat. After 21 Pis, bolt the last end cap and it looks like this:

So, there's a 2U containing 21 Raspberry Pi in a 19in server rack. I've only got 48 more of these to print and assemble.

Very Large Raspberry Pi Cluster - Part IV

So I've had a couple weeks here without much of a progress update. A lot of work has gone on. So here's a quick update before the holiday. First some photos. The first one is of the original prototype 3D printed out with some Pi in it.

This is a very nice design. The Pi just snap in place, there's room in the back to get to the micro USB power plug. But there are three problems:

1. I can only print half a 2U, so things had to be designed in such a way that once side is put into place, slid over, the other side goes in and is slid over and bolted, the first one slide over bolted, then the two remaining Pi on the ends are put in. And it requires two people.

2. It takes 50 hours to print one 2U holding 21 Pi.

3. It requires access to the rear of the rack. Without those rear bolts it wouldn't survive shipping.

So back to the drawing board. And here is the new design.

It's difficult to see what's going on here but there are 23 individually 3D printed pieces; one for each Pi and one for each end cap. They connect together by sliding on a piece of series 10 80/20 1"x.5"x10" long that has had about 3/4" milled off and the ends tapped. These are at the machine shop now so I will show some photos when I get them back.

One thing that's really nice about 3D printing is the RepRap; or the ability to rapidly prototype and revision. 3D printing is slow, but it's faster enough to do this. Here are some of my tests.

The one in the middle is close to the final design. You can see that two of the bolt hols of the Pi have a post and the last one has a threaded insert for a bolt.

Next post I will go into these two parts in detail. They have been designed for strength, speed of 3D printing and ease of assembly.

Very Large Raspberry Pi Cluster - Part III

I spent most of today fighting with Fusion 360 so I don't have much progress to report besides a serious complaint about Fusion 360. A new updated was released. If I didn't upgrade I couldn't save. If I upgraded I lost my changes. I had no choice but to upgrade, loose my changes and redo them. Then when I loaded up my models there were errors with unresolved imported components. The error states that if I save my model in this state I will loose my model. Personally I find this unacceptable and while I like Fusion 360, I like the disruption to the CAD world, many of the features, the price is reasonable and the fact that it runs on Mac and Windows, I also find it extremely buggy and a toy. I may be switching back to SolidWorks. Hopefully I get all of this resolved in the next few days and can report on the progress.

Update: After restarting Fusion 360 five times, and reloading the model after closing it with the error saying it will be destroyed, I finally got all the linked models and everything to load. So it was just a lot of ignoring errors and closing and reopening things and much ado about nothing, but extremely worrisome that just an update being pushed could cause such concern.

Very Large Raspberry Pi Cluster - Part II

Last week I posted about the very very large Raspberry Pi cluster that I'm building for Code One. Read that post here.

I realized after that post went live that I didn't give any backstory as to what we are actually building. When building something like this it's easy to document the entire process, it just takes 2-3 times as long to build. I figured I'd document the process in a fast and loose to blogs, YouTube and Twitter. So, before we go any further with the status of current progress let's backup.

Last year Stephen Chin and Gerald Venzl came to me with an idea. The conversation went something like this:

Stephen: "Gerald has an idea"
Gerald: "Let's build a HUGE biggest Raspberry Pi cluster for 2019 CodeOne. Like 1000 Raspberry Pis. The developers will love it. We will call it OSCAR!"
Me: "1000 is a lot, but 1024 is a better number"
Gerald, Stephen: "😃"

Back in the lab a few months later Jasper Potts and I did some math about how to pull off a cluster like this and we produced this rendering.

There will be 49 2U racks containing 21 Raspberry Pis (last weeks post was about this).
There will be 22 network switches.
There will be 18 USB power supplies.
There will be one server.
There will be 8 fans. This thing is going to get hot.
There will be 5 six foot server racks.
It will consume 120 amps of power.
It will require a fork lift to move.
It will network boot because can you imagine flashing 1024 SD cards?
In actuality, the server running the entire thing may be more powerful than all those Pis.
All of this barely fits in 5 racks.
This will travel to other Oracle events around the world next year.

WARNING: Actual numbers may vary, but the 1024 is the goal.

Very Large Raspberry Pi Cluster - Part I

I am building a very large Raspberry Pi Cluster and I'm going to document the process here. My goal is to post once a week until it's complete and shows up at Oracle Code One in San Francisco  September 2019.  Thus far an unimaginable amount of work has been put into this project to determine if it's even viable. We think it's viable so I'm going to start talking about it. There may be some mistakes. No, there will be some mistakes. Hopefully I won't make too many of them.

For the first taste, here is a rendering of a 2U rack that will hold 20 Raspberry Pi Model 3B+. I've printed out about 50 versions of this and this packs the most in while still allowing heat dissipation and still being serviceable in a standard server rack.

Disclaimer: The Raspberry Pi model is one I found on the Fusion 360 store thing. The rest is rendered by my own mad modeling skills.

Now that I've modeled this up and printed it I'm going back to the drawing board. It takes 14 hours to print the orange part which holds half of the Pi rack using an Ultimaker S5 with a .08mm nozzle 10% infield and .2 layer height. The Pi snaps in nicely but there are too many issues and this part needs to be nearly perfect.