VM Ware server set up at my previous employer’s office

I’m writing this from memory so some of the details might be sketchy.

At my last employer in Japan (the place where I took the intro photograph to this site), I was tasked with setting up a basic server to run MySQL. The TL;DR version was that there was a Windows 7 PC that was running MySQL that had data from which reports on Japan’s biggest convenience store chain were run. If this Win 7 PC failed the data would be lost, plus the “server” was underpowered. There was a proposal from a vendor for several thousands of dollars to set up a new server, a new UPS and backup system.

Since we had a second physical server which was not even set up and it was already on the UPS, it seemed foolish to me to purchase and install another one for a PC running MySQL, particularly when it was only being used once a week. I had to set up the disk array, install ESXi, install Linux as one VM, and create a backup routine to our file server which got backed up nightly off site.

Step 1: Set up the disk array

RAID 10 is all the rage these days, but this server wasn’t spec’d for that, so I figured RAID 5 would be the best choice. This server had 4 1.8 TB disks so with RAID 5 we’d have 5.1 TB of storage which was MORE than we’d ever need (the MySQL database was about 10 GB after 7 years, so even 500 GB of total space for this server would be fine).

After fixing a boot issue (one of the engineers tried setting up this server earlier but he didn’t understand what UEFI is and he couldn’t get the server to boot into Linux via a USB drive), I went into the controller to set up RAID. Using the controller, I set up a RAID 5 array:

RAID setup

Great, all set. Now reboot and let’s run ESXi setup. I’ll select the disk volume and get going.

Step 2: install ESXi manager

ESXi installer

Hmmm…no array, it only sees the individual disks. Something doesn’t look right here.  Long story short, after going back into the controller setup, RAID 5 was no longer set up and it only had individual disks again. After running around with this after several passes, I searched for help on the Lenovo forums. First thing was I booted onto a Linux boot disk and check the hardware. I needed which controller was being used, and found this:

I looked up this controller on the hardware compatibility list (HCL) for VMWare, so I’m stumped. Time to turn to the Lenovo forums. This is a reply I got:

(thread is short but can be found here)

But since I checked the hardware compatibility list, it showed this controller as being compatible. I had to reply:

 

Regrettably, I was still going to be sunk:

According to Lenovo support apparently this server’s controller won’t support RAID arrays with ESXi and will require an expensive upgrade.  Ugh.

So skipping straight to the solution:

I had to set up 4 individual disks, with the last one serving as a backup for the virtual machines on the other 3 disks. Since the backup volume ALSO maxed out at 1.8 TB, I made sure the other 3 disks were only 600 GB or less so they could back up to the fourth disk. I also ensured that the MySQL database was backed up to another file server so in case the database became corrupted it was backed up.

So final tally:

3 VMs x 600 GB -> Backed up to 1.8 TB volume (which itself was backed up once every 3 months). The remaining 1.2 TB per volume would be unused at the moment.

1 MySQL database (roughly 10 GB) backed up to a file server once a week (it was only used once a week)

Each of the backups listed above was backed up to our file server, which was backed up nightly offsite.

This solution wasn’t ideal but it would work.

Server down – PANIC!!!

I thought I would look at my website since I view it so seldom, and lo and behold, IT’S DOWN!!!

PANIC!

First I tried restarting the AWS EC2 instance – NO SITE!

PANIC!

Next, I tried to SSH in, but the IP address has changed since i rebooted.

forehead slap

Got the new IP, started SSH, got in. httpd and mysqld aren’t running (Dufuq?)

Tried restarting them  but couldn’t!!!  Have I been hacked?

After I calmed down, I saw that I needed to sudo.

Silly me!

Once I did that, I was able to restart httpd and mysqld

Phew!

But now I can’t see the site in my browser. Clearly this is a Route 53 / DNS issue.

Updated DNS entry with new, correct IP address. Now comes the waiting game, as pinging the domain name reveals the old IP address:

First ping test failed

(My OS is in Japanese. Trust me, the pings were timing out).

After a few minutes, the IP address is correct and the site comes up:

Site's up!

Phew! It’s nice that the right IP address shows up, but I can’t test because it’s not responding to pings:

Ping test 2 success, but no reply

I decided to add a security policy so I can ping just for testing.

Instead of adding it to the main policy, I’m going to do the “Photoshop layer” thing and create a new policy that I can add/remove at will and apply it to the same EC2 instance.

I created a Echo reply rule and allowed it from anywhere (this is for testing).

Custom rules for ping

Okay, it’s set for both incoming and outgoing. Great! Let’s test it:

Ping Still failing

Hmmm. Failure.  Oh,  snap! I need to set the instance itself to allow pings. Duh!

I had to look it up, but I found the iptables command that would work best. I normally wouldn’t set this to respond to pings from everywhere but this is for testing.

iptables to the rescue!

Okay, the security policy is in place, the DNS entry is correct and the iptables/local firewall is set. Let’s test again:

Ping success!!

Success!! I can rest easy now. And since the ping policy is separate from the main security policy, I can turn it off/on at will, or delete it outright.

I love AWS!!

Back after successful migration from AWS RDS to a local database

Okay, now that I’ve made a major change to this website, I’m back on and will keep this site up! I set this site up as part of my Amazon Web Services (AWS) training. A site like this uses a database to store its information, and mine was using Amazon’s wonderful RDS (relational database service) for my database.  RDS has a number of great features; the database is always running, it’s scalable and Amazon does all of the maintenance.

The catch? Cost. Even with the cheapest plan I was paying about $25 a month for the database alone. That’s just too much for a low volume website!

So I came up with a plan: run the database on the same server. It’s not as cool or as flashy, but it will save money which is my ultimate goal. Plus I’ll have to migrate from RDS to a .sql file, how hard could it be, right?

Looking up entries on the web, it turns out there are lots of tutorials on how to do just the opposite: migrate SQL to RDS. This required a lot of digging around. I was hoping for a nice, step-by-step, oral tradition via screenshot guide that would hold my hand through the process. Fortunately, I found something on docs.aws.amazon.com that really fit the bill. Turns out you can run mysqldump on your RDS instance and it will do the job just as if you’re running it on a normal MySQL instance. Viva la cloud-ia!

(Just do you don’t have to rip your hair out like Bender, here is the resource I found.)

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Exporting.NonRDSRepl.html

So I ran mysqldump using my RDS instance ID, database root password and it worked like a charm!

 

Hello Wo– nope, not going to say that!

The phrase that rhymes with “Mellow Whirled” has been BANNED from anything I write involving programming! I’ve read books that show you how to print “He*lo W*rld” on the screen, how to make “He*lo Wo*ld” in all caps, all lower case, reversed order, etc. But I’m sick of these exercises. On this website you’ll only see PRACTICAL uses of programming to solve real problems. So now, the phrase that sounds like “Shmellow Curled” will NOT be used here.