Apollo Rebuild Part II

Part 2 ended up being more software based and an SSD installation, surprisingly this took up the entire weekend. Let's go into it!

If you missed the first part of the Apollo rebuild you can check it out here.

This build tackled a very worn SSD, replacing it with a 500GB SSD from Samsung. Since I was replacing the drive I figured I may as well mix up the software and give it some much needed attention.

Previously this build ran Ubuntu Server 14.04 LTS, which has since stopped being supported. It also ran a terribly outdated Linux kernel to support OpenVZ, my first foray into virtualisation a few years ago. I can't remember off the top of my head now, but I think OpenVZ running on top of Ubuntu would only run a Linux kernel of 2.6.x - TWO full versions behind currently development.

While OpenVZ was great, it also provided some admin issues - since an entire OS had to be virtualised I ended up maintaining 13 (at a time) OS versions of Ubuntu, all sharing an old kernel.

Needless to say it got fairly messy, so I decided to dive head first into Docker. I'd spent an afternoon tinkering with it a few years back but it broke my brain, I simply couldn't understand it.

Anyway, enough muttering. Here's part 2 of the build for Apollo.

Seeing what VMs need backing up

      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       101          - stopped   192.168.1.40    dns1
       102          - stopped   192.168.1.41    dns2
       103         76 running   192.168.1.42    smokeping
       104        953 running   192.168.1.50    unifi
       106         47 running   192.168.1.45    ops
       107         57 running   192.168.1.46    deluge
       108         21 running   192.168.1.47    couchpotato
       109          - stopped   192.168.1.48    bitcoin
       110         62 running   192.168.1.51    data
       111          - stopped   192.168.1.52    requests
       112          - stopped   192.168.1.53    mono
       113          - stopped   192.168.1.54    newznab

Most of these had been stopped for various reasons. I'll run through the list very quickly:

dns1 & dns1 - At a time were dnsmasq instances, handling local DNS

smokeping - Ran pings to various destinations to monitor the (now replaced, woo!) ADSL

unifi - A UniFi controller for my Ubiquiti networking gear

ops - A small VM with remote access for remote testing, ran various things like timestamped traceroutes, MTRs, pings etc via cron

deluge - Torrents

couchpotato - For that wanted movies list

bitcoin - A, soon to be returned, very well performing Bitcoin node

data - Held InfluxDB and Grafana

requests - Plex Requests. This was already migrated to Docker but hosted on a NAS

mono - Testing Mono with ASP on Linux. It was a terrible idea. Don't do this. Microsoft and Linux don't play well together, and when they do anything with ASP is a resource hog

newznab - Dev environment for a Newznab indexer

I plan to move all of these, potentially a little more, over to Docker.

/etc/fstab was also backed up with various NFS volumes.
Also, ^^ Markup learning is happening!

UUID=2ce029f6-2465-49d5-94b2-52791ca43a4a       /mnt/storage    ext4    defaults        0       2
#
192.168.1.181:/volume1/movies                    /mnt/movies             nfs     rw,hard,intr,nolock 0 0
192.168.1.181:/volume1/series                    /mnt/series             nfs     rw,hard,intr,nolock 0 0
192.168.1.181:/volume1/dropbox                   /mnt/dropbox            nfs     rw,hard,intr,nolock 0 0
192.168.1.181:/volume1/music                     /mnt/music              nfs     rw,hard,intr,nolock 0 0```
Backing up Plex watched states
The Samsung SSD being installed, this thing is fast!
Some hardware! Universal rails mounted in the cabinet just above the main UPS unit
The SSD in its new home, gracefully thrown in

At this stage I should mention that I'm waiting to order a modular PSU to clean up the cable shit show currently in this chassis as well as proper drive bay mounts for both these drives.

The old faithful Seagate drive is also being replaced with a 2TB WD Black drive.

First boot with the new SSD installed

Also a sneaky view with a desk that I'm currently kitting out - more on this soon!

Booting off a USB drive, Ubuntu installations have definitely become a little more pretty over the years
Install drive config
This is new! Snaps are being pushed pretty hard, and while Ubuntu installs, slick!

Let's make sure everything is there and mounted correctly once we've restored everything:

jeff@apollo:~$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                            7.8G     0  7.8G   0% /dev
tmpfs                           1.6G  1.2M  1.6G   1% /run
/dev/sda2                       457G  5.9G  428G   2% /
tmpfs                           7.8G     0  7.8G   0% /dev/shm
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0                       91M   91M     0 100% /snap/core/6350
/dev/sda1                       511M  6.1M  505M   2% /boot/efi
tmpfs                           1.6G     0  1.6G   0% /run/user/1000
/dev/sdb1                       2.7T  233G  2.4T   9% /mnt/storage
192.168.1.181:/volume1/movies    21T  9.3T   12T  45% /mnt/movies
192.168.1.181:/volume1/series    21T  9.3T   12T  45% /mnt/series
192.168.1.181:/volume1/dropbox   21T  9.3T   12T  45% /mnt/dropbox
192.168.1.181:/volume1/music     21T  9.3T   12T  45% /mnt/music```

Awesome, looks good!

Plex was the first thing to be re-installed. This happened with a .deb since I can't seem to find a reliable Docker image, plus I get Plex Pass downloads so figured I'd install it straight onto the host.

Plex downtime lasted 41 minutes, and was fully restored. Phew! SLA met :D

Then the fun started, time to get Docker up and running with some containers. This was quite a learning curve. Volumes, port mappings, aliasing etc still really baffles me.

I managed to get the basics up and running with a few needed tweaks, some of the documentation on these images confused the shit out of me - so I'll probably write a blog on how I got some of these up and running.

Here's my current container list, all of these are fully functional, albeit with a few permission issues writing to NFS shares.

jeff@apollo:~$ docker ps -a
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
0881a1496a6c        influxdb                  "/entrypoint.sh infl…"   25 hours ago        Up 25 hours         0.0.0.0:8086->8086/tcp   influxdb
86d4fb931609        grafana/grafana           "/run.sh"                35 hours ago        Up 34 hours         0.0.0.0:3000->3000/tcp   grafana
d818dd4f5977        linuxserver/couchpotato   "/init"                  47 hours ago        Up 47 hours                                  couchpotato
cfac2b7f71e6        linuxserver/sonarr        "/init"                  47 hours ago        Up 47 hours                                  sonarr
427772346c72        linuxserver/deluge        "/init"                  2 days ago          Up 47 hours                                  deluge
10ce4a90247a        linuxserver/sabnzbd       "/init"                  2 days ago          Up 24 hours                                  sabnzbd```

That's it for now. Part 3 should include a little more Docker knowledge along with a PSU reinstall and drives mounted properly.

Until next time.