Wednesday, June 1, 2011

You need this when? Yesterday? I'll get right on it!

I know we've all heard the expression "Poor planning on your part does not constitute an emergency on mine", but nothing could be further from the truth. Luckily there are some people out there that still understand this concept.

I was tasked with installing a certain piece of software, this directive came practically from the top of the institutional food chain, without revealing too much information, I became suspicious of the software before downloading it, and did my due diligence. There have been reports of ad-ware and spy-ware, as well as censor-ware being included with the official version of the software.

The end-user who had been told to get this installed understood my concerns, and we have started looking for alternatives. It's nice to see people who still "get it"

Programmer <> Software Developer <> "I Write programs"

This is one of my pet peeves, and it keeps showing up in my everyday life as a support guy. There is a massive difference between "good" programs, and "bad" programs.

Let me start with some background about me, I took two and a half years of a computer science degree, I can write programs (I may not be that gifted) but I understand technology, and have run into many of the pit-falls myself.

What do I mean by "Bad" programs? mostly, ones that don't follow good practices, ones that appear to have been written in someone's basement. Don't get me wrong, some of these programs may be very good at what they do, but they are a complete BEAR to support.
2 examples, one bad, one worse.
Bad: A financial budgeting/forecasting suite. We were handed this software, and informed "It will be installed" the purchase had already been completed. So we look into it... It needs a SQL database, good... Wait WHAT... It needs DB_CREATOR rights to the Database server? NOT GOOD, but we can work around it by creating a whole separate instance of SQL, just to run this software, moving on, We have 15 seats, Okay. We can activate it 17 times before we have to call to get more activations (provided we have an active support contract)? wait, there's no way to de-activate an install once it has been activated?

Worse: A specialized contact database for placing students in practicum type locations. The "Program" consists of 2 access databases, DB1 resides on a file-share, and stores the data, DB2 is "installed" on each user's computer and provides the UI to DB1. Aside from my dislike of Access as a database system, there's one MAJOR problem that we ran into, we had to move DB1 as we were retiring the original server it resided on. the UNC path to DB1 had been hard-coded into DB2, necessitating us to contact the "vendor" and having him provide a newly edited DB2 that pointed to the new path. Very very rarely should anything ever be hard-coded.

Enough for today, forgive the rant.

Monday, May 9, 2011

Adobe Install Pain

It is that time again, time for me to update our install of Adobe Reader, and push it out. (Yes, I know, Adobe Reader) We use Administrative Install Points (AIPs) and AD GPOs to install it across the domain. while this works nicely, there is some pain involved in getting the AIP set up correctly.

Pain Aspect #1: Patch Order. I refer you to the Adobe KB When doing AIPs, you can't go 9.4 -> 9.4.1 -> 9.4.2 -> 9.4.3 -> 9.4.4, you have to start at 9.4, and jump to 9.4.2

Pain Aspect #2: Customization. Adobe has released a customization toolkit specifically for those of us that do global installs. but it breaks things, If you have any customization, you have to revert to a vanilla AIP, then upgrade, and then re-apply the customizations

Pain Aspect #3: Regression. This is something that Adobe is well-known for. As an administrator, I have to test, double-test, and then re-test for regression issues. There are known issues with folder redirection that cause perennial heartache. (this is one reason why we aren't moving to Reader X yet)

Long story short, it took me about 3 hours to get the latest installer ready to push out by GPO. I didn't find any issues with the new version, but I'm sure I'll get a call tomorrow from one of the users.

Wednesday, May 4, 2011

Email, the life-blood of an institution

We're just coming back from a somewhat major email outage here. Long story short, a component on the motherboard of our mail server failed, and this brought the mail server down for just over 41 hours, as with any sort of incident, a postmortem exam is in order.

Our response could have been better. I lost count of the number of times I answered the phone saying "Yes email is down, alright, have a nice day." Getting word out to people when a primary channel of communication is down is very difficult. In the end we ended up using voice mail to get the word out, but this was highly sub-optimal.

For those of us not directly involved in getting the server back up and running, life was expected to continue. The lack of email made things very difficult. We are currently at the point in the year when are contacting vendors, getting quotes lines up so that come the start of the new fiscal year, we can pull the trigger without any hesitation. It's kind of difficult to get quotes when you can't get email. (Interestingly enough, there are some places that will only send out quotes electronically via email, but require you to fax a purchase order in, go figure)

Now, given that we use Exchange, it wasn't just email that was down. Calendars were inaccessible unless they happened to have been cached before the crash, likewise with people's contacts. It is kind of scarey how many people rely on a single technology to store practically their whole life.

There were some bright spots to the whole affair though, I can remember a time when the default assumption was "the network is down" rather than "something must be wrong with my computer" this time I'd say I answered roughly equal numbers of "Is the network down?" and "what is wrong with my computer" questions.

All in all, I still think our disaster plan needs some work, but we got through it.

Monday, April 25, 2011

Don't piss off the geeks

Rule #1: Don't piss the geeks off. They control the universe, and can make your life a living hell.

What triggered this brief rant? We have remote control abilities for a reason folks. We don't make a habit of spying on you, face it, you're not that interesting, but we need the tools to do our job.

I had a user (who has been granted local administrative privileges) on his laptop. This user had disabled a portion of our remote control abilities. This is a minor pain, and we have ways to re-activate them. But it just pisses me off.

in short, "Just don't do it"

Tuesday, April 19, 2011

Documen... What? Huh?

Gotta love hybrid systems, sometimes things are done one way, and other times another (for whatever reason) and if you do it the "right" way for a specific group, it works, but another group it won't.

You've got to stay consistent, and you need to keep documentation as to how things are done for individual groups.

I just ran into a case where I shot myself in the foot, because the "right" way wasn't the way I was expecting, and of course, it appears to work for a brief period, and then it fails again.

ah well, my own fault, now lets make it work right.

Monday, April 18, 2011

Generic email accounts. Love to Hate Them

No matter where you work, you are almost certain to run into generic e-mail accounts. There are some places that you are going to have to use them. But what do you do with them?

Some departments are prime candidates for generic email accounts. These are typically departments that deal with outside entities, and ones that need some sort of continuity when staff changes or goes on vacation. Say "Accounts Payable", "Accounts Receivable" and "Human Resources" are good candidates (not to mention "Postmaster", "Abuse", "webmaster", "Hostmaster", and any other pseudo-required addresses)

What to do with them?
Option 1) Just set a person up to pull the messages via POP/IMAP, and let them deal with everything from there. I think this is the worst of the three options I list here. You know darn well that whoever has this account set up isn't going to remember the password to the account, you aren't going to be able to enforce good password security, and if you need to "move" the account to someone else for whatever reason, you'll probably have to reset the password.

Option 2) Don't actually create a mailbox, but rather set up a mail alias. This avoids the problems listed above, but creates its own set. namely that the poor schmuck that gets these messages will have to filter them him- or her-self, and that if something should happen, the next person doesn't have access to historical messages.

Option 3) set up some sort of "Ticket" system that can tie into your email system. Each message that comes in either creates a new ticket, or is attached to an existing ticket. This way anybody with appropriate authorization can get access to any old messages, and you can have multiple people with access to the system each dealing with things, and actions can be logged, I.E. "Invoice Received & imported into Accounting system, waiting to pay pending notification from receiving dept." and if the primary person is on vacation, whoever is filling the position can see what is going on without having to change credentials, forward messages, etc.

This falls under the category of "more work up-front solves problems later on"

Saturday, April 16, 2011

9-5++

Gotta love working in IT. Don't get me wrong, I find it very rewarding, but there comes a time when I want to leave work at work. I've been working at my current position for just under 3 years now, and was at my position prior to that for three years. There is one thing that is somewhat of a constant when working in IT Support. The dreaded Cell Phone.

Theoretically my work hours are from 8am to 5pm, with an hour for lunch. From a practical standpoint, I start getting calls and emails at around 7:30am, and they continue on well past 5. Part of that can be attributed to working in Higher Ed. Classes don't stop at 5, they continue until 10, and some adjunct faculty members aren't on campus during our normal hours. Some of it can't.

I've been on the road to a family function on a Saturday morning, I've taken calls from the CFO while trying to put my 1 year old to bed, I've troubleshot printers while waiting for a ride to a wedding in a different time zone.

I have a company issue iPad, and I have a very hard time ignoring the little "1" on the mail icon, emails will come in all the time, some of them are easier to ignore than others.

My anniversary is come up soon, and my wife and I are planning to get away for a couple days, just the two of us. The kids are going to stay with their grandparents. Do I leave the phone? Or do I take it so that they can reach us of they need to? There's no easy answer.

Ah well, so far it has been a quiet weekend, only 1 email so far, but one that fell under the category of "rather important, and needs to be dealt with sooner rather than later"

On that note, I'm going to wash the dinner dishes, and hope to sleep without hearing the dreaded "you've got mail" chime.

Friday, April 15, 2011

Ampersand Dollar Percent Pound Exclamation Sign

Yes, I'm swearing in Long-Hand today. First thing when I walk through the door I religiously read the latest ISC posts. this one caught my attention MS11-020 (KB2508429) Upgrading from Critical to PATCH NOW.

Things like this really make my day. When you read further it states that the exploit being patched can be executed without authentication! I really don't like the idea of having "my computers" be vulnerable to something like this.

Now don't get me wrong, I have already pushed the patch out to all of my computers via WSUS, what this means though is I am setting an installation deadline to ensure that it is installed as soon as possible. I also now need to go in, and check to make sure that all computers have the patch installed. when I have 500ish computers, and not all of them are physically located on campus all the time (Laptops, Road Warriors, etc.) This can be somewhat difficult.

I don't like forcing deadlines on patch installs, doubly so when the install is going to require a reboot, but there are times when it needs to be done. This is one of them.

Thursday, April 14, 2011

Security Incident - What would you do?

We've received a notification that one of our user's accounts has probably been compromised.

What would you do? Reset the Password, and force the user to preset ID to get a new password? send an email asking the user to change their password?

dangit... we need another policy

Updates, More Updates, and...

This has been a busy week for us. Microsoft released a huge slew of updates, Adobe has an update to Flash that is due out tomorrow.

I run a set of terminal servers, let me tell you, they've been getting a workout of reboots recently. I ran the latest set of MS patches last night, it took about an hour a server to complete them. Once Adobe releases the Flash update, I will have to push it out, and restart them again. Luckily, I can easily drain/stop the servers so that I can bring them down without any disruption of service.

Hint: Keep yourself up to date by following The SANS Internet Storm Center