The 600GB system log that killed a file server in Phoenix

Pulled a drive out of a customer's Dell PowerEdge last month and found the system log had ballooned to 600GB because the RAID controller was throwing a warning every 30 seconds for like 4 months. Nobody checked the log because the status light looked green. Now I set up email alerts on every server I touch, especially for the PERC controller, and I tell people to check logs monthly. Has anyone else found a silent log that ate a whole drive?

2 comments

2 Comments

finley_smith1d ago

Yeah but who actually checks logs monthly... nobody does until something breaks. That's the real lesson here.

robin89623h ago

Used to figure a monthly check was good enough until we had a big outage nobody saw coming... that changed my mind real quick.