502 Bad Gateway, or how I broke my blog

For the past few years, I've been running this blog on the open source Ghost CMS and it's been great.   Ghost is great for a site like this where the focus is on the content because the focus of Ghost is on creating content (posts), whereas blogging has become a smaller part of the WordPress ecosystem as WordPress has slowly "evolved" from being a CMS to becoming more of an "application" platform.  

So a few weeks ago when I logged into the admin panel I noticed I was missing a critical update in my Ghost install and it was recommended that I upgrade ASAP.  Now I'd been lazy about updates on the site and was still running the 3.x version of Ghost which went EOL in 2022, so I had to do the major upgrade from Ghost 3.x to Ghost 5.x with a stop at 4.x in the middle.  Sounded simple, and ghost makes updates usually a pretty painless thing.  

When the upgrade finished and I tried to restart ghost however I received this error:

/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /node_modules/sqlite3/lib/binding/napi-v6-linux-glibc-x64/node_sqlite3.node)

Great.  The newer version of the nodejs sqllite module needed a newer version of the standard C Application Binary Interface (ABI) than what was on my machine.  This machine was running CentOS 7, which shipped with gcc-4.8.5, and to get the correct ABI modules I'd need to build and upgrade to a newer version of gcc, like 8.

On CentOS (or any Linux system for that matter) you really can't replace the standard libc module with a newer version without breaking almost everything, so you need to build it in a different directory (like /opt) and then tell your newer application where to find the updated libraries.  After downloading gcc-8.5.0, the process to build it was pretty simple:

tar -xvf gcc-8.5.0.tar.gz
mkdir gcc-8.5.0-build
cd gcc-8.5.0-build/
../gcc-8.5.0/configure --enable-languages=c,c++ --disable-multilib --prefix=/opt/gcc-8.5.0
make install

Once the new version was installed, all I needed to do was tell my system where the new libraries were with an environment variable and everything should "just work", and it did.

export LD_LIBRARY_PATH=/opt/gcc-8.5.0/lib/../lib64

So with the updated GCC on my machine, and the new LD_LIBRARY_PATH set I marked the upgrade to Ghost 5 as completed and walked away.  A few weeks later, however, someone asked me why my blog was down.  Embarrassed, I logged in and lo and behold they were right - the ghost process had failed to restart and was down.  

At this point I figured it was just easier to migrate the whole thing to a supported OS rather than continue to attempt to make it work on CentOS 7 so I spun up a new Ubuntu 20.04 LTS VPS at Vultr and migrated the site and its data over.  A few quick changes to my DNS to point to the new server and everything is now up and running correctly on the new host.  Which reminds me I need to write a post on how I mange my DNS with git, but that's for another day.

So what is the lesson here?  Well, first off don't let your software get too far out of date - leaving my blogging platform for over a year without an update just wasn't a smart idea.  And secondly, I needed to find some low cost way to monitor my site - not knowing it was down for weeks on end is not a good look for someone in the IT industry.  So I've added nodeping to my list of personal tools now for monitoring my own infrastructure - a few of my clients use it and I've found it to be a great low cost way to add external monitoring.  

So hopefully this puts an end to the issues with my site and things will be stable now, until the next upgrade.......