First thing we do, let’s fire the IT Director

Peter Tesei approves town's backup system: "close enough for government work'

Massive computer failure wipes out 5 terabytes of data at town hall,  and there’s no back up (!)

A trove of electronic records that includes some 11,000 beach card renewal applications was lost over the weekend after a series of hard drive failures sustained by the town of Greenwich’s computer network, according to local officials, who are now scrambling to try to restore the precious data.

“Unfortunately, we have to rebuild the system from scratch and it will take some time,” said Boris Hutorin, the town’s information technology director.

Hutorin estimated that 5 terabytes of data — that’s 5 trillion bytes in layman’s terms — was wiped out as a result of what he characterized as an extremely rare occurrence.

Asked to elaborate on what the town meant by compromised, [Parks Director]  Siciliano said, “Not there, OK.”

All data entered by the parks department from Feb. 27 to date is unrecoverable, according to Siciliano, who said his office is having to pay out overtime to its full-time employees and additional hours to temporary workers to help sift through hard copies of the information that was lost.

The Griffith Harris Memorial Golf Course has resorted to taking reservations for tee times manually as a result of the situation, Siciliano said.

Hutorin characterized the problem as a fluke hardware failure that he said affected multiple storage area networks both Saturday and Sunday in the computer room at Town Hall.

Saturday’s hard drive failure resulted in a minimal loss of data, according to Hurtorin, who said that consecutive crashes on Sunday are to blame for the larger problem.

“The system is designed to work around this kind of failure,” Hutorin said.

While the first hard drive was being replaced on Sunday and the data was being “rebuilt,” a second hard drive failed, Hutorin said.

“It’s almost impossible to have this happen,” Hutorin said.

“Almost impossible”, obviously, isn’t the same thing as ”impossible”.  Do you suppose Hutorin has tenure?

UPDATE: Reader “Georgie” asks whether our Boris is the same Greenwich IT guy who was using town computers (and town time) to surf porn sites. Don’t know, but  the porn site in question was written entirely in Russian. Now, maybe Boris Hutorin has hired other Russians to work for him in the IT department and they’re the ones enjoying pictures of dirty  Russian girls (can we still call them sluts, or is that now off limits?) , or maybe he’s the only commie down there in the basement.  If Peter Tesei knows, he ain’t talking.

23 Comments

Filed under Uncategorized

23 responses to “First thing we do, let’s fire the IT Director

  1. Georgie

    If this doesn’t result in the IT Director getting fired, what the hell will it take? Isn’t this the same guy who was downloading porn on company time?

  2. Libertarian Advocate

    Perhaps Anonymous has struck faire Greenwich? Now that would be weird.

  3. Anonymous

    Chris, I don’t know if this happens to others accessing the GR blog, but it absolutely kills my PC whenever I follow a link to it. Here’s the story direct from the GT: http://www.greenwichtime.com/default/article/Town-investigating-employee-s-erotica-postings-on-650406.php

    Also, this story says that Boris Hutorin is/was being sued for sexual discrimination and names the porn entrepreneur as Alec Verny. http://www.greenwichtime.com/local/article/Sexual-discrimination-complaint-filed-against-676728.php

  4. let’s fire the IT Director

    Let’s fire the IT Director?

    That should only be the beginning: multiple, multiple heads should roll for this one. As a town resident I am so sick of watching my (rather substantial property) taxes go toward paying for the mistakes of these incompetent town hall boobs.

    Just who is accountable here?

    And what does Peter Tesei plan to do about it? The buck for this stops at his desk, and he’d damned well better do something about it!

    If he doesn’t, then his head should roll too.

  5. hmmm

    lftitd,

    people need to get involved…that’s the answer you have been looking for…

    everyone is too busy, i have too many things on my plate, i can’t get it all done, now will somebody please get it done!

    that’s what everyone sounds like…..get in there, volunteer, do something don’t just say something….do it, do it now…

  6. CatoRenasci

    Fire the IT director and his entire staff immediately – change the passwords, disconnect the network so they can’t access anything and have the cops escort them out of the buildings before noon today!

  7. If you fire the IT director and the entire staff then soon absolutely nothing will work. No email, no file storage, no internet access, no departmental applications.

    The type of hardware failure described here is very rare but can happen. The real question is why there were no backups stored off site to reload the data from.

  8. Yes, it’s a universal precaution. If nobody else, the town’s auditors should have spotted the issue and included it in their written audit findings to the town.

    The five terabyte is also highly suspect. It’s way, way high for an operation like town government unless they’re storing something like video surveillance data, which is usually segregated from normal numeric/text data.

  9. If you’re hosting Russian porn sites that would explain it.

  10. FF

    The IT Director, along with the Town Attorney and the Director of Fleet Services are among the few departments at Town Hall that have a straight, direct report to the First Selectman. Peter Tesei can fire him at will, for any reason or no reason

  11. Rick

    If they lost all those beach pass applications does that mean them furreners from Stamford and Port Chester will have unrestricted access to the beaches?

  12. The IT director probably should be fired unless there are mitigating circumstances, like he asked for money to implement data backups and was turned down. The hardware failure was random chance and couldn’t have been foreseen or avoided.

    Peter can watch porn on the interwebs like everyone else.

    • I hear that the guy (the IT fellow) is the least popular man in Town Hall – a real asshole – but no one dares fire him because they’re scared of his threat to take down the system if he goes. True? I don’t work in Town Hall so I don’t know. But if it is true, it’s time for Tesei to discover some cajones.

  13. Anonymous

    a typical 3 or 4 disk RAID system can only withstand 1 disk (hardware) failure and keep running. Plug a new disk in and resync, the problem is when it encounters an error it just stops and the whole data partition disappears. A six disk system (fairly untypical in non-WallStreet environment I guess) can withstand 2 disk failures and keep running.
    I have lost 2-3Tb of data at home due to a similar situation. You must scrub RAID partitions weekly or monthly to proactively scan for errors and of course have offsite backup and live backup servers. Netapp allows for “snap mirroring” to backup servers running in other locations, EMC has similar functionality for its SAN technology.
    Having live backup servers offsite usually doubles the cost of the servers, perhaps that was something the local gov”t wouldn’t spring for…

  14. Fred2

    Seriously, this can happen for a little RAID box with 5 TB, but the whole town relies on _one_ RAID rack?

    Which costs about what these days: $2000? ( 5 100$ 1TB drives and the RAID enclosure?) Ok, let’s say it costs $10K fully installed and running ’cause it’s serious heavy iron.

    Now, as IT guy for a big town like Greenwich it’s not unreasonable to imagine that I’d have SEVERAL of these things mirroring each other so that I’d have a live/hot back up at at least two sites. PLUS offsite back up (in Nebraska or Northern Vermont) on, at a minimum, weekly basis.

    After all, the avoided cost of having your entire freaking DB packing up is worth more than 20-30K down and a couple of grand per year maintenance, no?

    I guess not.

    Geek Speak explanation:
    (RAID = Redundant Array independent disks, a technique where data is copied across multiple disks so that any X number of the hard drives can go bad or be removed without losing data, the box manages copying data to new drives that replace dead ones. RAID type 1 is mirroring ( you write the same data to two drives, Raid type classes go up much higher and the more drives you have the more redundant and safe the data can become because each data chunk is written to multiple drives so more drives can die at time before you get data loss.)

    Seriously these days you can get this for your home system for not that much money and back up all your home machines nightly. so worth it.

  15. Georgie

    The only way I can see this guy keeping his job is if he has documented proof that he tried for many years to get a responsible disaster recovery and that he was denied of such a plan….so then the blame clearly lies with the First Selectman who denied the request.

    When people say that for years they have tried to fire the guy and they can’t…I just don’t get what that means. What, he is some Union Gonzo, that is protected….under what god-for-saken rule? This is what is so wrong with government employees….you can’t even fire someone so clearly incompetent, stupid, or careless.

    • No, as Fudrucker points out, the man is one of the few town employees who can be fired by the first Selectman without cause or reason – Tesei just doesn’t dare do it, for some reason.Perhaps he’ll tell us.