February 18, 2012

Disaster recovery disasters

Filed under: Adventures in IT — jason @ 2:11 pm

Here you see our server room during a power outage caused by a malfunctioning transformer on a pole down the street.  Kentucky Utilities told us that we lost two phases of the source coming into campus so we were seeing some things work, others not.  In the server room, one large backup power supply lost power, one didn’t.  So, most devices with dual power supplies were still running.

Anticipating the possibility of a total loss of power, we walked through our emergency procedures to cleanly shut down as many systems as we could.  It turned out to be a nice review of our procedures as well as a good exercise for several members of the IT staff to walk through the steps and discuss the scenario.  Did I mention that this outage started at 4:30pm on a Friday?  Power was restored later that night and we had everything back up and running around 10pm.

I can readily summarize the potential disasters from our outage because we had just updated our emergency procedures documentation the week before (more on this later):

  • At best, we’ll get about 15-20 minutes of uptime during a total power outage.  Anything that does not automatically shut itself down when power loss is detected will go down hard, and we have plenty of things that will suffer that fate.
  • We didn’t have a printed copy of our emergency procedures on hand and we wouldn’t have been able to easily see them, anyway.
  • From the dark picture, you can’t see the worst problem of all – we have water sprinkler heads in our server room just like any other part of the building.  Should we ever have a fire in the building that sets off the system, we are going to be in a world of hurt.

At least there are a few points that make me feel a little better (if only a little):

  • We have emergency procedures documented and we know which systems are a priority to shut down cleanly.
  • We have well maintained backups of all systems that are not stored in the same building as the systems themselves.
  • Our administration is aware of our procedures and the precarious nature of our environment.  We hope to make changes in the future as we grow our campus (more on this later, too).
  • We turned a long work day into an excuse to grab pizza and beer on a Friday night.

Now, for the “more on this later” part.  Just one week before, we had updated our emergency documents because I was going to present a summary of our vulnerabilities to my fellow cabinet members.  Our board of trustees had recently formed a technology committee and I had given some folks a tour of the facilities and talked through our issues, which had generated a lot of questions.

I thought it was a good idea to make sure that all of our senior administrators had a grasp on exactly what kind of downtime we’d be looking at after a disaster.  After all, several of us are actively involved in planning the future of our campus.  What better way to solve our problem than to have everyone thinking about how a new building would make a great place for a data center with chemical fire suppression, diesel generator facilities and no bathrooms overhead!

I had just presented this information to the cabinet three days prior on Tuesday.

On top of that, earlier on the very Friday of the outage I had visited some employees of the city’s emergency planning department.  They had offered to give us some emergency alert beacons that we could place in strategic locations on campus, so I was meeting with them to discuss the particulars.  We met in their emergency command post area that you see below.  Ironically, we talked about how this area has no backup power sources whatsoever.  We grumbled about budgets and planning and how everything is acceptable on paper right up until the time it breaks.  If only I had known what was coming later that day!

To round it out, apparently Kentucky Utilities has no way to detect an outage like ours.  They only found out about it once we started calling.

To be sure, we are not alone in our situation.  I’ve seen many places that have no regular backups or that backup to a disk sitting right on top of the system in the same room.  I’ve seen all manner of technology shops with no disaster plans or procedures to speak of.  And on this particular Friday, I got a glimpse of what is going to happen to the city’s emergency planning headquarters when the power goes out.

But we can’t pat ourselves on the back for having properly documented emergency procedures while we continue to have sprinkler heads in our server rooms.  It is much too easy to say “well, we’ve lived with it all this time…” or “at least we’ve got good backups” and feel like we’ve got it covered.  You have to make the problem a priority and address it in some way if you are ever going to fix it.  Maybe that should be our disaster recovery documentation.

• • •

January 1, 2012

Remodeling the IT way

Filed under: Adventures in IT — jason @ 11:12 pm

Our public safety office got a much needed makeover.  We were called in to move their IT equipment and it ended up being a much bigger job than we anticipated.  They have several monitors for DVRs that operate security cameras, a few PCs, radios, phones, an alert box and various other equipment.  We set them up temporarily in a nearby location, then moved them back once the remodel was completed.

The showcase piece is their wall of monitors.  Normally, I should not be allowed around power tools, but I did most of the measuring and drilling for the wall including flamboyant use of a hole saw!

 

 

• • •

May 13, 2011

The little tablet that could

Filed under: Adventures in IT — jason @ 3:49 pm

I am an unabashed BlackBerry junkie.  I’m carrying my third BlackBerry phone these days: the Bold 9700 on T-Mobile.  All I can say is that I love having keys that I can feel and press (I can type really fast on the thing) and I love having UMA (Unlicensed Mobile Access) via TMo that allows me to make and receive calls over wi-fi when I’m somewhere without cellular coverage, such as my basement office at work.  I could go into a whole other diatribe here about why AT&T, with its overly congested network, is stupid for not allowing UMA to get some of that traffic off of their cellular grid and onto broadband, but I digress… and I fear what may come of UMA if and when AT&T absorbs T-Mobile…

When RIM announced the BlackBerry PlayBook, I was very worried.  RIM has quickly fallen from being the leading smartphone maker to lagging in third behind the iPhone and Androids.  As advanced as BlackBerrys were five or six years ago, RIM simply missed the boat when it came to a touch screen interface.  I wondered if they could come anywhere close to the success of the iPad and what a tablet would mean for the future of their phones.  But, I decided that I would have to give one a try and see it for myself.

The PlayBook in its first incarnation is all about the “bridge” to your BlackBerry phone – it does not have a native e-mail client, address book or calendar app – it simply pairs via Bluetooth with your phone and gives you an interface to the data there.  Many a reviewer took big, cheap swipes at the PlayBook for this perceived lack of functionality, but there are a lot of BlackBerry owners out there and clearly RIM targeted them first, so get over it.  Native apps will surely come soon enough to appease even the biggest Apple fanboys.

The first version of the software build that shipped on the thing was rough around the edges.  One of the best things about a BlackBerry (and worst about an iPhone or iPad) is easy file access.  The PlayBook ships with a really nice set of apps from DataViz that are well done mobile versions of Word, Excel and PowerPoint.  Sadly, I could open files from my phone but not edit or save them anywhere.  An update to the PlayBook arrived within a week or so, and most of those file issues were fixed.  The PlayBook does allow wireless access to its file system via a Samba/share interface which is a welcome treat.

When I started out with the PlayBook, I had BBOS 5 on my Bold.  I had several issues in keeping the PlayBook paired with my phone, but eventually upgraded to BBOS 6 which cured that problem.  There is a bridge app that you install on the phone and it certainly looks like more work went into making it go smoothly for BBOS 6.  That may be an issue for older BlackBerry owners that don’t have an upgrade path available from version 5.

The PlayBook’s operating system, QNX, is quite frankly stunning.  Multitasking on the device is very smooth and the use of gestures to switch between running applications is a big plus over pushing the button on the iPad.  Multitasking on the iPad was certainly an afterthought for Apple, and it shows in the way it is implemented.  You can fire up video, chat, bridge and other apps on the PlayBook and switch between them easily without the device ever breaking a sweat.

You can tether the PlayBook to your BlackBerry and work away while on the road and not near any wi-fi, with additional carrier charges, of course.  The bridge apps even include a “bridge browser” that allows you to surf the web via the Bluetooth connection to your BlackBerry without having to tether at all.  Interestingly, AT&T has so far blocked the bridge app for BlackBerry phones from showing up in their App World store, probably because they won’t be able to tell if you are using that bridge browser app and charge you for it.  Morons.

The PlayBook hits the iPad in all of those familiar places where it is tender.  It has support for Java, Flash and has a micro-HDMI port for output to a monitor.  Most of my iPad experience is with the first generation model and its infuriating lack of video output.  Only certain iPad apps will push anything out of the docking port video adapter (which you have to buy separately), and the Safari web browser is not one of them.  In contrast, the PlayBook will mirror everything across HDMI and certain apps like the video player and Slideshow To Go (the PowerPoint app) have a presentation mode that uses the full 1080 lines of output for display while showing you additional controls on the device.  You can even switch between apps to look something up while the presentation mode continues on the external display.

Did I mention the speakers?  Audio coming out of the PlayBook sounds clearer than anything I’ve ever heard on a mobile device of any kind.

Of course, the PlayBook is far behind in the total number of apps available for it, though I might argue that there are a whole lot of useless apps out there for the iPad and Android.  It certainly needs Netflix and Skype clients, but they did crank out a Facebook app pretty quickly.

App development for the PlayBook will be interesting to watch because the native tools for QNX will be C/C++ while most of today’s BlackBerry development is done in Java.  There are a lot of unhappy BlackBerry Java developers out there, not only because it looks like QNX may be the future for new BlackBerry phones, but also because RIM is in a bigger hurry to get an Android app emulator written for the device before they get the native development tools ready.  Right now, your best bets for getting an app on the PlayBook are with Adobe Air or their WebWorks SDK that lets you make apps with HTML and Javascript.

All in all, the PlayBook is a really slick piece of hardware that ships with some very polished and useful business apps.  Multitasking and video output capabilities are second to none, and BlackBerry junkies like me will enjoy the bridge features to access data on their phone in a bigger, prettier interface.  Come on, RIM!  I am really rooting for you on this one.  With the right combination of apps, development tools and most importantly salesmanship, the PlayBook could definitely turn some heads.  And if you learn what you should about touch screens and QNX, you could set yourself up to produce some really nice, new BlackBerry models very soon.

• • •

April 9, 2011

Taking out the trash

Filed under: Adventures in IT — jason @ 6:50 pm

We had a couple of users complain about our fancy new Outlook plugin that provides mail, calendar, contact and task list synchronization goodness with our mail server.  Their complaint?  “Old mail is automatically disappearing from my trash folder!  How am I supposed to keep anything now?”  One even asked if we could change it so that the “Delete” button filed the message in a folder for them instead of actually deleting it.

It is a sad day when you have to add a bullet point to your instructions to remind a person that yes, mail that they choose to delete will, in fact, be deleted.

• • •

November 11, 2010

Drinking the Kool-Aid (er, Apple juice)

Filed under: Adventures in IT — jason @ 10:07 am

I recently purchased my first ever Apple computer for use at home – a vanilla 21.5 inch iMac.  Dallying with iPods and iPads seemed less of a sin, but I finally decided that I need to live with one every day in order to better understand and support them – you know, like a woman. :)   No falling back to the PC, either!  I’m moving all of my stuff to the Mac and riding it ’til it bucks me.

In a fairly short time, I had most of what I need up and running – Cisco VPN client, Microsoft RDP for Mac so that I can connect to stuff at work, moved my iTunes library over, synced the Mail app and iCal with GMail, Blackberry desktop to deal with my phone, Logitech software to program my Harmony remote control, Eclipse for my random development efforts and the Dropbox client (one of the first true cloud apps besides Google stuff that I’ve really found useful).  Oh, and I did indeed install a Flash player – the horror!

I had a couple of small hiccups but nothing major:

  • When the Mail app synced with GMail, it sucked down all of the trash, created a label in GMail called “Deleted Messages” and applied it to all of those trashed messages.  This annoyance resulted in my poor Blackberry receiving over 200 trash e-mails that somehow looked “new” to it.
  • The Address Book will sync with your GMail contacts, but only if you also sync the contacts on an iPod, etc with it.  Strange design choice here, but I was already syncing my iPod contacts with GMail over the air, so I simply changed it to sync with the Mac instead in order to magically populate the Address Book.
  • There was one song in my iTunes library that apparently had too long of a file name for either the Mac file system or the ext2 file system on the D-Link DNS-323 NAS that was the go-between for file transfer from the PC.  iTunes complained about not being able to find that file, but I was easily able to point it to the mp3 with the shortened name.

I’m going to ride without anti-virus software for now just to see how that goes.  As an IT director, it miffs me a little to hear Apple people say that Macs are somehow safer and don’t require anti-virus software.  Macs are not safer because they are in some way better engineered, they are simply not a big enough target for the bad guys to aim at yet.  But as their market share climbs and there are more Macs out there, I’d expect to see some nasty stuff showing up on them.

At work, we’ve had success integrating Macs with Active Directory for authentication, dynamic DNS and file shares.  Domain controlled printers are another story, although that issue seems to be with the CUPS printing system, so Linux users can’t print to a domain printer, either.

Here’s to living how the other half (er, maybe ten percent?) lives.

• • •
Next Page »
Powered by WordPress |•| Wordpress Themes by priss