Introduction

My home desktop has been running Ubuntu 18 since it came out in April 2018. As all even numbered versions, it has LTS (long term support) for 5 years, which ended last year. Yet the reliability of Ubuntu is a double edged sword. It runs so long, without filling the hard drive with garbage, without fragmenting the filesystem, without losing performance, that over the years you customize it so many ways that by the time you need to upgrade, it’s hard to remember everything you’ll need to set up in a new system.

So why upgrade at all? After 5 years there are no more updates. No more security patches, and for most of the applications you can only run old versions. So you need to upgrade to use the latest versions of applications and stay on top of security.

You can upgrade in place to keep everything you’ve configured, with do-release-upgrade. But this only takes you up 1 step to the next LTS version (18 to 20, 20 to 22, etc.). So I’d have to do that 3 times! And it doesn’t guarantee that everything you set up will work – though it usually does.

This time around, 6 years and 3 versions behind, I decided to do a full upgrade from 18 to 24 and wipe the system boot drive. I waited until Ubuntu 24 was released on April 25 and then hit the keyboard.

Install

I actually use XUbuntu, a variant of Ubuntu that uses the XFCE desktop instead of Unity. I prefer the XFCE desktop, as it is more traditional and faster, using less CPU and RAM. Even if the computer is fast enough to support a more heavyweight desktop, why waste the CPU and RAM on that? Even if Unity was as lightweight and fast as XFCE, I would still prefer XFCE.

First I backed up my system using deja-dup. Most of my data files are on separate hard drives, which I would not be erasing. So I only needed to back up my home directory and an “apps” directory that I use for software that is manually installed. I also reviewed my apps and files to ensure that all the software I wanted to install was available on my data drive: DEB, TAR, etc. As my desktop has 4 drives, I wrote down the device and UUID for each so I could recognize them during installation.

Next I downloaded the ISO file, burned it to a DVD, rebooted to the DVD which started XUbuntu. I selected to do a manual install, reformatting the system drive. I selected “minimal installation” because I didn’t want a bazillion crapplets that I’ll never use, but also “include 3rd party and binary software”. This all went smoothly and it also automatically installed the NVidia drivers for my GTX1660.

Next, I removed the DVD and booted, my computer came up in XUbuntu 24. Yay! But now the real work begins…

Configuration

Firefox

Firefox is a mess in Ubuntu 24. First, by default it installs as a Snap, which nobody likes. And, on top of this, they’ve crippled its access to the filesystem. You can remove it from Snap with snap remove firefox and then install with apt install firefox, but the next time you update it will revert back to the Snap install. To get it to stay this way you’ve got to edit some config files to tell the system to always install it from apt.

Next, I found that Firefox could not download files. After scratching my head for a while I realized this is because my home Downloads folder is a link that points to my data disk. And I also discovered that I could not open local documentation HTML files for the Android and Java SDK. Putting 2 and 2 together, I realized that someone decided to cripple Firefox’s access to the filesystem. WTF!?

I suspect this is the “younger generation” of developers, who think the local filesystem is somehow “unsafe”, and really unnecessary anyway since they do everything in the cloud. That is the only reason I can imagine for the crazy idea of crippling such an essential function. Most sites say that installing Firefox from apt instead of Snap solves this. Perhaps it once did, but not anymore in Ubuntu 24. With some Googling I learned they’ve created an AppArmor config that restricts filesystem access for Firefox. My fix was to configure AppArmor to disable the Firefox controls, like this:

sudo ln -s /etc/apparmor.d/usr.bin.firefox /etc/apparmor.d/disable

Altogether, this got Firefox working properly again.

Backups: Deja-Dup and Duplicity

To restore all of my backups I’d have to be root because some of the files to be restored are system, owned by root not by me. No problem, right? Think again. My backups are on a separate disk that I mount to a local folder /dsk3. When I ran deja-dup to restore a backup, it would unmount that drive!

If I ran deja-dup as myself, it would work only because it wasn’t root so its attempt to unmount the drive would fail. This was fine, it still worked, even if only by accident. But with deja-dup it is impossible to restore a backup that you have mounted to a separate disk, if you run it as root.

I could not find a solution to this, so I only restored my home and apps directories mentioned above. That’s all I really needed to back up, so I got lucky.

Thunderbird

I installed from apt and it saw all of my email that was restored from the backup – both my local folders and all of my IMAP email server accounts. Bang!

Python and Virtualenv

In Ubuntu 24 Python works differently. The system Python is now 3.12 instead of 2.7, which is nice. But the procedure for installing virtualenv and virtualenvwrapper that I used for Ubuntu 22, does not work in 24.

First, use pipx instead of pip. That is, pipx install virtualenv.  After that, pipx install virtualenvwrapper. At this point they are installed but the don’t work. Next, I had to change the environment variables and python & wrapper script locations:

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVBIN=$HOME/.local/share/pipx/venvs/virtualenvwrapper/bin
export VIRTUALENVWRAPPER_PYTHON=$VIRTUALENVBIN/python
source $VIRTUALENVBIN/virtualenvwrapper.sh

I created some virtualenvs, listed them, etc. it all works fine – great!

Pulseaudio and Pipewire

Ubuntu 24 audio is working, but it runs pipewire instead of pulseaudio. I had pulseaudio configured to avoid resampling for bit perfect audio. Pipewire is not doing this; it resamples everything to the system default sample rate of 48 kHz. With some reading, I found a simple way to configure this.

The pipewire config files reside here: /usr/share/pipewire

In particular, the primary config file is: /usr/share/pipewire/pipewire.conf

But rather than edit this directly, you can leave it in place there and create another config file that overrides individual settings.

Create this directory if it doesn’t already exist: /etc/pipewire/pipewire.conf.d

And in that directory, create text files (having any name) that override individual settings. For example I created this file: /etc/pipewire/pipewire.conf.d/pipewire-custom.conf

# Daemon config file for PipeWire version "1.0.5" #
#
# Copy and edit this file in /etc/pipewire for system-wide changes
# or in ~/.config/pipewire for local changes.
#
# It is also possible to place a file with an updated section in
# /etc/pipewire/pipewire.conf.d/ for system-wide changes or in
# ~/.config/pipewire/pipewire.conf.d/ for local changes.
#

context.properties = {
    ## Properties for the DSP configuration.
    default.clock.rate = 48000
    default.clock.allowed-rates = [ 44100 48000 88200 96000 176400 192000 ]
}

stream.properties = {
    resample.quality = 10
}

After doing this, when I play audio files using VLC, with an external DAC connected to my sound card’s SPDIF output, the external DAC shows the sample rate changing to the native sample rate of each file that I play. It works!

Also: the DSP library PulseEffects was replaced with EasyEffects, a newer version that works with PipeWire. This new version has a feature I like: it does not resample all audio to the system sample rate like PulseEffects did, but plays audio at its native rate without resampling, changing the system rate to match the audio. Very nice!

Audio Update

Audio stopped working on each bootup. After some troubleshooting I discovered that the kernel module for my Juli@ sound card is not loaded. How to check this?

First, run lspci -k and it shows you all your PCI cards, and for each, which kernel module it uses. This tole me the card is plugged in and the module is snd_ice1724.

Next, check whether that module is loaded, or really check for all sound modules by running: lsmod | grep snd_. This did not list the snd_ice1724 module.

So, load the module manually by typing: sudo modprobe snd_ice1724. After that, repeat the above command to see if it’s loaded. Bingo! It was. And after doing this, audio started working.

Now here’s the mystery: normally you can tell the system to load new kernel modules by creating files in files in /etc/modules.load.d. For example I created this file:

-rw-r--r-- 1 root root 53 Apr 29 18:02 /etc/modules-load.d/snd_julia.conf

The contents are simple:

# kernel module for the Juli@ sound card
snd_ice1724

This should tell Linux to load the module at boot, so I don’t have to load it manually. But my system still doesn’t load this module. Or maybe it’s trying but failing, but I can’t find any errors. And I also looked for it in blacklists, but it is not a blacklisted module. Indeed, if it were the system shouldn’t let me load it manually.

At least I can fix the problem, but I don’t know why it doesn’t load automatically when it detects the sound card, nor manually from the config file, so I’m stuck loading this module manually at the command-line.

Java, Tomcat and PostgreSQL

I installed openJDK versions 8, 11 and 21, using apt, then ran update-alternatives to set the default to JDK 8. Even though it’s old, a lot of software requires it and won’t run on the newer JDKs.

Next I installed Tomcat, version 10 is what comes with Ubuntu 24. That was simple enough.

Then I needed the PostgreSQL JDBC driver so my Tomcat server could talk to databases on other servers in my home network. I found the JAR file online, copied it to /usr/share/java and linked it to /usr/share/tomcat10/lib.

Finally, I copied my audio recordings JSP app to Tomcat. First I made a directory for the app: /var/lib/tomcat10/webapps/audiorec. Then I copied the source files (JSP) to that directory. BANG! It just worked.

Apps and Games

I installed Steam and after configuring it, it saw all of my games on my separate disk drive. And it saw my configs since I restored my home directory. All good? Nope.

Two games I’ve been playing recently are Valheim and Bioshock Infinite. Steam showed how long I had been playing and when I most recently played. And the games are supposed to save your progress in the cloud. But when I started the games, they started from scratch. They had no notion of any prior game play or saves. I lost all my progress in the games.

On the brighter side, the games now run natively in Linux, where previously with Ubuntu 18 they only ran reliably from Proton, the Windows emulator. Same hardware, same drivers, the only difference is Ubuntu 24 versus 18. So I don’t know why, but that is nice…

Printers and Virtualbox

Back in 2019 I bought the TurboPrint driver for my Epson SureColor P400 printer. This is a good thing, since Epson doesn’t have a Linux driver for this printer – even though they do for most of their other models. Yet the TurboPrint driver installed and worked just fine, it doesn’t need the Epson driver. Turboprint is excellent, and essential for anyone who wants to print photo quality from Linux.

I have a Windows 10 VM for software that doesn’t run on Linux. Just a couple of apps, Epson’s CD print and Suunto’s link for my GPS wristwatch. I installed virtualbox simply from apt and it just worked – my VM came up just fine. Of course I did have to edit my system groups to ensure that I am in both plugdev and vboxusers in order for the Windows VM to be able to see USB devices like the printer and the watch.

Addendum: IOMMU

What the heck is IOMMU, you might ask? Indeed, I asked the same. I noticed a bunch of errors in dmesg, like this:

DMAR: DRHD: handling fault status reg 3


With some Googling I found it is related to a CPU chip function related to RAM called IOMMU. I’ve been using Linux since Ubuntu 8 in 2009 on a wide variety of desktops and laptops, and I’ve never seen this before. Maybe this desktop’s CPU is too old to support it – it’s an i7-4770 from around 10 years ago. Still reliable & fast, so why replace it?

The solution was to add a kernel boot parameter iommu=soft. I did this by editing the /etc/default/grub file, then running update-grub. Confirmed fixed.

Note that if I set this parameter to iommu=off, the USB system doesn’t work. No keyboard, no mouse, connect via SSH and lsusb reports no devices.

Conclusion

Of course there was a bunch of other stuff. And I’m not even done yet. But the computer is mostly working again, with most of my data and apps restored. I’ll get the rest of it working over time.

About a month ago emails that we sent were bouncing, being rejected by the destination servers saying one of the IP address where it came from was blacklisted. I host this site and our email through Hostgator, using their SMTP server to send email. So my first guess was that the Hostgator SMTP server was blacklisted. It’s plausible, as it’s shared by many customers, some of whom may be spammers. But the IP address from the email bounce message didn’t match that server. Turns out it was the IP address that Comcast assigned to my home router. So Comcast assigned me a blacklisted IP address! Perhaps Comcast shares that IP with other customers and one of them has been spamming.

However, email sent from my GMail SMTP was not bouncing. Looking at the email headers, GMail’s SMTP does not forward the IP address of the client who sends it; it sends its own. So one possible solution would be to get the Hostgator SMTP to stop forwarding my IP address, but instead use its own, or even use my own IP address for this domain, mclements.net. I contacted Hostgator support and they said they could not configure the SMTP server to do this.

In the meantime, as a temporary workaround I configured my email send to use the GMail SMTP and CC my other email. It’s clumsy but works.

My next option was to change my Comcast IP address. I called support and they confirmed that Comcast uses DHCP, so getting a new IP address should be as simple as turning off my modem & router, then turning them back on. I knew it wasn’t that simple because I had done that and the IP address never changed. The DHCP has a lease with a time duration. You need to disconnect long enough for that to expire before getting a new IP address. I left them off overnight and still got the same IP address.

DHCP servers often (though not always) use the client MAC address as a hash when providing IP addresses. If you can change the MAC address of the router that connects to your cable modem you are likely to get a new IP address. I dug into my router menus and found a config option to do exactly this: you can enter any MAC address you want, or have it copy the one from the PC you are using to connect to the router. When setting MAC addresses manually like this, one must be careful because the MAC address is not just a random number. The first several hex codes are assigned to manufacturer and device type. You should ensure that what you enter is a valid MAC address.

In short, this worked. After changing my router’s MAC, Comcast issued a different IP address that was not blacklisted, and now my email is not bouncing. Since these IP addresses belong to Comcast, I called to let them know the old IP address was blacklisted so they can deal with it.

Summary of steps:

1. Configure your router to use a different MAC address
2. Turn off the cable modem and router
3. Turn on the router
4. Turn on the cable modem and wait…

After step (4) it may take longer than usual to get an internet connection. When the router negotiates with Comcast for a connection, if its MAC address is different and Comcast assigns a new IP address, the process can take longer.

Summary

Android is essentially a Linux distro. It is a mobile device oriented UI / desktop running on a version of Linux. However, it’s not really open source. Every phone manufacturer writes binary drivers to get it to run on their hardware, and they don’t contribute those to the community. And mobile carriers pile their own add-ons on top of this. So by the time the Android phone gets into the hands of the user, it is loaded with proprietary software and bloat-ware.

For example Samsung modifies Android with “TouchWiz” which significantly changes the UI. And Motorola pre-installs the Facebook app – users can “uninstall” it, but it reinstalls itself every time the phone boots. Some people might like these changes, but I think most, like me, would prefer the pure unadulterated open source Android without bloat-ware or crapplets to burn down the battery and hog the storage.

Another issue with Android is that most manufacturers only support a given model of phone for a year or two. After that, no more updates, which means the phone is condemned to planned obsolescence.

In some cases we can avoid these issues, keep our phones for years while running the latest version of open source Android without bloat-ware. If that sounds interesting, read on.

Unlocking or Jailbreaking

The term “unlocked” has 3 different meanings:

• Carrier unlocked: you can use the phone on any carrier’s network, so long as you have the right SIM card, and the phone’s modem & radio supports the right frequencies & protocols (GSM vs. CDMA). Many phones are carrier locked when new. When buying a new phone, when it says “unlocked” it means carrier unlocked.
• Bootloader unlocked: just about all phones are shipped with a locked bootloader. This means you cannot replace the operating system. You can only install factory supported updates using the phone’s settings menu.
• Rooted: the phone allows the user and apps to take on “superuser” or admin privileges. This means the phone can be used as a little computer without any restrictions – direct access to the filesystem etc.

“Jailbreaking” refers to rooting Apple phones while “unlocking” is a more general term.

Carrier: Every carrier is required by law to give mobile phone users a code to carrier-unlock their phone, so long as the phone is fully paid for. Most make this process as difficult as they can in order to discourage users from doing it. But, the process is accessible to non-technical users. Essentially, they send you a long code that you enter into one of the phone’s settings menus.

Bootloader: There is no law that I know of requiring manufacturers to allow users to unlock their phone’s bootloader. Some (like Motorola) officially support this, others (like Samsung) support it only unofficially, and others do not support it at all. In the latter case, hackers often (but not always) figure out how to crack it. When you unlock the phone’s bootloader, the warranty becomes void. Of course, that’s a nothing sandwich if the phone is already out of warranty.

Rooting: This enables you to use the phone for things you can’t do otherwise. For example, direct access to the full filesystem, even system partitions, enables full backups just like a computer, and makes possible amazing customizations. However, some security apps (for example banking and 2FA) detect whether a phone is rooted and refuse to run. So don’t root unless you really need to, and take measures to handle these cases.

Booting to Recovery

Recovery is disk partition on the phone that has a mini-OS that runs as root and enables you do make changes that aren’t possible when it’s booted normally. This includes changing the partition table, wiping the system partition, loading the operating system, etc. The two most common ways to boot to recovery are:

• A button chord: power off the phone, then power it on while holding down other buttons. Exactly which buttons varies by manufacturer. For example, with Motorola press and hold the volume down button while powering on.
• Android tools: Android, being open source, has a full set of developer tools that are freely available to everyone. Install the Android toolkit on your computer (Linux, PC or Mac), connect your phone via USB, and control it from the computer using tools like adb and fastboot.

Every phone comes with a factory supplied recovery, but it is not intended to load custom operating systems. Two of the most popular custom recoveries are TWRP (on Samsung) and boot.img (on Motorola).

Partition Table

The partition table determines how much disk space is allocated for recovery, for the operating system, and left over for user storage. Running a newer version Android or different recovery sometimes requires more space, which means changing the partition table.

Google Apps

Google Apps includes the Google Play App Store, Contacts, Calendar and several other apps that are not part of the Android operating system, but run with special privileges other apps don’t have. These apps cannot be installed from the Play Store but are installed while booted to recovery, just after installing the operating system.

These apps are not strictly required to use the device, but without them you won’t have the Google Play store or other important functions. They are required if you want your device to work like the normal Android that everyone is familiar with.

Compiled ready to install packages of these free open source versions of Google Apps are published in two popular places: OpenGapps and MindTheGapps.

The Process

So, how does one actually do this?

1. Find out exactly what kind of device you have, including the specific model number. For example, not just a Motorola G7 Power but model XT1955-5, because there are several different versions having different processors, radios and modems.
2. Ensure your device is fully functional with the OEM ROM, go to settings and install all updates to ensure it has the most recent version.
3. Ensure to actually use the features from the OEM ROM, to ensure your device is properly registered on the carrier network. For example, on T-Mobile make phone calls & texts over WiFi.
4. Back up any data on your device that you want to keep, because the install will wipe the entire device.
5. Find a ROM that is supported on your device. You can get an officially supported ROM, like those at lineageos. Or you can find an unofficial ROM at places like xdaforums, where individual developers create and support them. An unofficial ROM could be LineageOS or many other versions of Android.
6. Unlock the bootloader on your device. How to do this will vary from one device to another so you’ll have to do some homework.
7. Follow the ROM installation instructions. This typically includes these steps:
   1. Boot to the OEM recovery
   2. Flash a new recovery
   3. Boot to the new recovery
   4. Update the partition table
   5. Install the new ROM
   6. Install Google Apps
8. Boot the device into LineageOS.