Last night as I was about to head to sleep, Sensu started emailing me about disk space warnings on one of the backend servers. That’s strange, I thought. I had set up logrotate with appropriate limits to ensure the log file size is reasonable and rotation happens on a daily basis.
Curious, I ssh’d into the server to investigate. Running a df -h indicated as expected the disk space in use was over 70% (which is the trigger for sensu to send a notification) and the log files had grown way over expected size.