Tuesday, August 31, 2010

McDonnell Wants Independent Probe of Computer Outage


Richmond, Va. --
Gov. Bob McDonnell wants an "independent third party" to investigate the collapse of a Virginia government computer system nearly a week ago that continues to paralyze some agencies.

"I am not pleased that our employees and citizens have experienced this disruption in service," McDonnell said yesterday in ending his public silence on the crisis.

Six of 26 agencies hit by the outage Wednesday still were not fully up and running, despite a promise by state computer chief Samuel A. Nixon Jr. to have their service restored by yesterday at 8 a.m.

This includes the state Department of Motor Vehicles, which may not be able to resume issuing driver's licenses and identity cards at its 74 offices until tomorrow. The agency had expected such services would be available today.

Other agencies still limping: the state Department of Taxation, which could not make tax refunds or process tax payments; and the state Department of Juvenile Justice, unable to handle fully the intake and tracking of young offenders.

The state Department of Environmental Quality had continuing problems with a system it uses to develop computer programs. Voter files are being restored at the State Board of Elections.

McDonnell's directive to the agency that Nixon heads, the Virginia Information Technologies Agency, to open an inquiry potentially puts the Republican governor at odds with Northrop Grumman, which was hired in 2005 to provide communications and computer services under a $2.3 billion contract -- the state's largest-ever privatization program.

Christy Whitman, a Northrop Grumman spokeswoman, declined comment.

McDonnell, who collected more than $75,000 in campaign contributions from Northrop Grumman, had pledged to smooth relations with the embattled company, having lured its headquarters from California to Northern Virginia and having agreed to a new, longer contract that pays it an additional $236 million.

"I have directed an operational and performance review of the situation be conducted so we can determine the proper course of action to best protect the interests of the commonwealth," McDonnell said in a written statement.

His press secretary, Stacey Johnson, said VITA is expected to hire an "independent third party" to explore the cause of the blackout and, perhaps, recommend that the state recover from Northrop Grumman the dollars in lost business and productivity.

The company already is expected to be hit with fines for the latest service interruptions. Nixon said the penalties will be at least $100,000.

Nixon said he had yet to select a firm to conduct the investigation and is choosing for now to focus on restoring service.

Even VITA is having difficulties. Its accounting and billing programs are not working properly.

Contributing to the delays: restoring giant data files, a task that can take 18 hours or more. The episode -- it is the second major outage under VITA/Northrop Grumman since 2007 -- is calling attention to the adequacy of backup systems, for which agencies must pay the company extra.

The new contract with Northrop Grumman requires that the company strengthen and speed up its response to equipment failures and other emergencies.

At VITA/Northrop Grumman headquarters in Chester, more than 400 servers, which support clusters of state computers and their software, failed when two circuit boards crashed mid-afternoon last Wednesday. The boards were manufactured and installed by EMC Corp. of Hopkinton, Mass.

A company spokesman could not be reached for comment.

But VITA said it had been told by EMC that a dual failure was unprecedented. The company shipped backup equipment to Virginia during the weekend, but it has not been installed, Nixon said.

Before the outage, McDonnell had planned to meet with his Cabinet yesterday morning at the VITA-Northrop Grumman offices in Chester. But the session was held in Richmond instead.

Johnson said the meeting was conducted at the governor's office -- the usual venue -- because "the folks at VITA need to get everything back and running."

3 comments:

Anonymous said...

Boards don't "Crash". Software systems "crash", boards "fail", all of which clouds the issue even more. It will be interesting to see what the real cause of the failure is although given the companies and the politics, it's unlikely we will ever know the truth.

Anonymous said...

“...But VITA said it had been told by EMC that a dual failure was unprecedented. The company shipped backup equipment to Virginia during the weekend, but it has not been installed, Nixon said...”

That is a nice fluff comment. Like Board failures are dependent events, and somehow only ONE can fail at a time…!

jmmb said...

And it isn't just DMV that is down. Also Dept. of Taxation and Juvenile services, etc. And now taxpayers will be faced with helping pay for a "third party" to investigate the problems.

In this day and time a red flag should have been raised the first day this took place.

Seems odd..........