DevOps
Of nginx’s mid cut off responses and proxy buffers
· ☕ 3 min read · ✍️ Sathya

Among the services I look after, the biggest and high-profile - is the user facing website. The website is your bog-standard typical frontend(powered by Express/Angular) which fetches data via an API which is powered by the backend(built on Rails). Typical flow is that Express receives the request from the browser, makes a request to the backend which is then served using Rails API via nginx which acts as the reverse proxy.


Statutory warning: Decimals in your logrotate config can be bad for your server’s disk space
· ☕ 2 min read · ✍️ Sathya
Last night as I was about to head to sleep, Sensu started emailing me about disk space warnings on one of the backend servers. That’s strange, I thought. I had set up logrotate with appropriate limits to ensure the log file size is reasonable and rotation happens on a daily basis. Curious, I ssh’d into the server to investigate. Running a df -h indicated as expected the disk space in use was over 70% (which is the trigger for sensu to send a notification) and the log files had grown way over expected size.

Chef Zero, Cookbooks and Data Bags locations
· ☕ 2 min read · ✍️ Sathya

I like chef-zero a lot. There’s so much overlap between chef’s products: chef-zero, chef-solo, chef-apply - yes, each have their own uses, but I digress. Chef-zero has been relatively pain-free for me - except when I tried to get it running - trying to figure out why it wasn’t fetching the cookbooks was so annoying! I went and RTFM couple of times, no dice. Then I decided to not skim read and read the entire thing.


DevOps Diaries: When Terminator terminates your MySQL imports onto an AWS Instance…
· ☕ 3 min read · ✍️ Sathya

I have begun to use Terminator quite a lot. Terminator’s quite handy when you want to connect to multiple servers on a single terminal thanks to its split pane feature. (And yes I know about tmux & screen - I have screen on my servers, don’t want to get into the headache that is nested screen panes).

Few days back we were preparing for a big migration & deploy and I was tasked upon to prepare a failover just in case the migration goes wrong. I have an automated daily backup from prod and restore to local server, I decided to restore from the MySQL dump on the local server onto the AWS instance via the automated process. Since database restores can take a bit of time, I started the restore and went over to grab some coffee. Once back, I noticed that the import was done and proceeded with a rake db:migrate and realized that rake was doing a migration going all the way back from the start… that did not look proper.