T-SQL Tuesday #70 - Managing an Enterprise
My first job as an actual SQL Server DBA was for a local hospital. Between two of us we were responsible for about 90 SQL Server instances, including the backend databases for the Electronic Health Record system and the application that monitored the equipment in the ICU units. Adding to what already sounds like a carefree position, this monitoring was done solely through my favorite tool, SSMS. Fortunately the other member of my team was a good teacher because he left after about a year and I was left as the sole DBA.
Like with all things, the more that you know, the better you are at doing your job. With an environment that size, with those vital applications and so little as far as resources, there were no magic tricks to make it easy. There were, however, some things that made it more doable.
I made it a point to work closely with the application owners. I would check in most mornings and we had regular meetings. I encouraged them to let me know when things didn't seem to be working as usual rather than waiting until everything fell apart. This also prevented the grumpy dba vs. those danged users type of environment.
I also created baselines for the most integral databases, both to benchmark important processes and to see the expected behavior on those instances. With that many, there was really no way for me to remember how everything was supposed to look. Baselining helped me to identify whether that spike or that lull was expected or out of the ordinary. I don't remember what I used to baseline back then, but I know that in later jobs I made use of PAL, a free tool available on CodePlex.
Additionally I had scripts that would email me failed jobs, files running out of space, disks that were getting full, high severity level alerts and of course the dreaded 823,824 and 825 errors among other bits of important information. I couldn't anticipate job failures, but definitely didn't want to get surprised with a disk running out of space or bunches of autogrowth events.
I have to admit that I never completely got out of firefighting mode in that first job, but I did get a better handle on it. That has been the largest enterprise that I've managed and the lessons I learned there made me a much better DBA.