April 1, 2020

Server Failure covid-19: day 20 | US: GA | info | act

Well, I’m not sure what happened, but the Google Cloud virtual machine I set up to run a web server went kaput. Everything was going well. Then, a couple of days ago, the server started becoming unresponsive for long periods—like ten minutes. I’m not certain, but it seemed to happen around the time the kernel was updated. These freezes just started happening regularly—three an hour, or so. When it did, web sites would not connect, and I would often get booted off my ssh session. If I didn’t lose the connection totally, the terminal would just become unresponsive. It was like a program was throwing a big equation at the processor that took all of its power for ten minutes. Then, everything would be fine for a little while, until the next freeze. 🤬

I combed through logs, and there was no clue as to what was happening. I did upgrades, to no avail. I removed software I didn’t need, but that didn’t help. I searched Google, but could find nothing. Finally, thinking it might be the kernel, I updated to the latest and completely borked the system. It wouldn’t even boot.

I’ve run servers for years, and I never had anything like this happen. I suppose I could have been hacked, but I really saw no evidence of that, either. I’m really at a loss. It seems I can’t even run a simple web server anymore.

Fortunately, I did not lose anything but time and effort, and at least I (re)learned quite a bit. I pointed the domains back to where they came from, and I’m back up. I guess paying professionals $30 a year to not have to worry about things like this is worth it. The shared server is slow, and not without its problems, but it’s probably the better option. Amateurs should probably not run servers anyway.

Well, I may not be ready to give up. I might try again with a more proven (read: older) version of Linux on Google Cloud. Or, during my insomnia last night, I went Raspberry Pi shopping. Maybe I can run a small web server on my own network. (I’m definitely getting a Raspberry Pi Zero to run Homebridge. In fact, I just ordered a Zero for this very purpose—with this cool case in “Lazer Lime.”) Yes, my Synology NAS is capable of running a web server, but it’s clunky as hell, if my experience with Nextcloud is any indication. I also think servers should be minimal and administered via the command line. Old skool, yo.