Skip to main content
  1. Posts/

DevOps Diaries: When Terminator terminates your MySQL imports onto an AWS Instance…

··627 words·3 mins·
DevOps Aws

I have begun to use Terminator quite a lot. Terminator’s quite handy when you want to connect to multiple servers on a single terminal thanks to its split pane feature. (And yes I know about tmux & screen - I have screen on my servers, don’t want to get into the headache that is nested screen panes).

Few days back we were preparing for a big migration & deploy and I was tasked upon to prepare a failover just in case the migration goes wrong. I have an automated daily backup from prod and restore to local server, I decided to restore from the MySQL dump on the local server onto the AWS instance via the automated process. Since database restores can take a bit of time, I started the restore and went over to grab some coffee. Once back, I noticed that the import was done and proceeded with a rake db:migrate and realized that rake was doing a migration going all the way back from the start… that did not look proper.

I connected over to the failover instance and realized only about 60% of the restore was done. First thoughts: oh shit!? Why? How? Are my database backups screwed? Hasn’t anyone noticed this? How much did I setback our migration timelines?

Not wanting to waste more time, I decided to restore a RDS instance from Amazon’s snapshots(hey, I guess Amazon would do a better work than me!?), while simultaneously downloading the last dump which my automated process is doing. Some time later, the RDS instance was up and running and the backup download was done. Then I realized that the database dump was about ~400MB gzipped and expanded to about 4.5GB uncompressed. One big ass SQL file - how the heck do I validate its correctness? For funsies I loaded it up on Sublime.. the loading bar was moving slower than a sloth. No dice. Tried with vim.. yeah same result. My *nix instinct kicked in(while simultaneously facepalming myself for not doing this earlier) and I tried the tail -n 100 way. Huh. The dumps are proper. Down to the last table & row. So WTF is causing this to fail!?

I continued to prepare for the migration and half my mind was still on this. Being slightly obsessive about things like these, I decided to restart the import again, this time ditching the automated script and going the “good ol’” manual way, and lo & behold, the same thing happened again, though this time at a slightly different time. However, this time I noticed there was an error:

The error code is 2013: Lost connection to the MySQL server during the query.

error. Hm, why’s that? There was no Internet connectivity issues and all other metrics seem fine. What could be causing this? I restarted the import again, and again this time the import was killed with the same error. However, this time by chance I noticed the trigger for this error: I had resized Terminator pane. Did you go WTF!? Yeah, same here. Still reeling from this, I tried the same: restarted the import, resized the pane and bam! The restore was killed and connection dropped. Now knowing the cause, the started the import(again!) and went all out to ensure I didn’t touch the pane and voila, everything went fine.

I enquired with my colleagues if they had seen something like this, and nope no one did. I still haven’t figured out the reason though. But some takeaways from me:

  • Do your MySQL imports from a separate tab
  • If you still need to use the same tab with a split pane, don’t touch the pane
  • Make sure you check for return codes for your automated stuff(I suppose this is elementary, well shit)
Sathyajith Bhat
Author
Sathyajith Bhat
Author, AWS Container Hero and DevOps Specialist.

Related

Bash bashings..
··131 words·1 min
Tips & How-To's Bash Linux Popd Pushd
Since my new job involves lot more of Linux, shell scripting bash and automation, I’ve been trying to brush up my Linux skills. Ran into this “problem” today where there’s an init script which handles unicorn start/stop/reload and we were debugging some kinks around this. The init script had a chain of cd to the directory and the unicorn invoke script. Was trying to figure out why unicorn wasn’t starting up, till I read a bit more and dropped to using echo to understand what’s happening
Review of Supergiant Games’ Transistor
··837 words·4 mins
Gaming Games Reviews
Transistor is the second game made by Supergiant Games following up on their previous game, Bastion. Bastion didn’t hold my attention long enough so I really can’t say much about it, except at first glance, Bastion seemed to have great artwork and music. Story # Transistor follows the story of Red, a singer in a city called Cloudbank who gets nearly killed. She manages to escape and acquires the Transistor - a sword-like weapon.
GOG Connect – import few of your Steam games into Gog.com library
··276 words·2 mins
Gaming Games Steam
Gog.com(formerly Good Old Games) is part of CD Projekt Group - the publishers of the fantastic Witcher series of games. GOG started off as a firm which (re)distributed classic games without DRM and ensuring they run on modern systems. Off late, they have evolved into digital distribution - and they’re going against the likes of Steam (and to much lesser extent, Origin). Most digital distribution platforms tend to push their own brand, force you to use their clients and lock you down to their own platform - so not having all your eggs in the same basket is a good thing.