New state data complex or $300 million dollar boon doggle

(The following information is taken from the web page of State Representative Reuven Carlyle from the 36th District, Democrat. It was called to my attention by a fan who was once involved with the state’s information services.  I have edited it.  You can get the complete story by going to Carlyle’s web page.)

It’s my understanding from digging a bit that the state data center project, a $300 million endeavor not including operational, hidden, overrun or other unreported costs, is being designed technically by a team of lead architects who have never built a modern data center. Ever.

The agency has chosen not to use outside, independent advisors for the system design but rather use three, internal, long-time state employees who have a combined total of zero experience designing a real data center. Good, decent and genuinely honorable guys I’m sure but it’s their first time. I’ve got a bad feeling about this.

And, it seems from opening the hood of the project, the actual project manager for the entire endeavor–the guy the entire agency will look to manage the full capital and operational budget, schedule, vendors, consultants, construction, etc., etc.–has absolutely zero experience managing a data center project. First time.

As an entrepreneur in the wireless, software and clean technology fields I’ve learned the painful lessons of weak project management and systems design. I take this issue very, very seriously.

In fairness, I’m not looking to take cheap swipes at the state agency as an institutional bureaucracy–or, of course, any individuals– over it’s internal management of the data center project.

I’m not interested in micro-managing but I am interested in the larger systems issue of transparency into big, expensive, hidden decisions.

One of the reasons I’m frustrated by this project is that when you scratch the surface even a tiny bit it’s clear there is a very weak technical and financial case for this entire program. We’ve jumped right to the most expensive, 1990s, government-centric solution possible to a massive technical problem we haven’t even accurately or thoroughly defined.

The agency wants to take a leap forward to provide cloud computing services? Even a non-technical business guy like me can tell they don’t have the technical systems, networks through applications, in place to do that if they tried, and a new building won’t change that until we upgrade a lot more than their agency address.

The biggest technical problem facing the State of Washington has almost nothing to do with the lack of a consolidated data center and everything to do with old-fashioned (1970s, 1980s) databases, middleware and application layer capability to serve real people living real lives.

With this decision to build a data center, we are spending a fortune but not solving the REAL problem. We’re about to simply move our old equipment into a new building and nothing will materially improve for the public, other agencies, the legislature or taxpayers. That’s the real frustration. It’s not just the money, it’s the lack of value.

Need a specific and uncomfortable example? Last month the computer system of Washington State University collapsed. For real. Registration, financial aid, housing allocations, course schedules…all down on the very weekend when students arrived.

It’s working now with duct tape. The university and DIS are working together to try and get a viable, long-term solution up and running. But it broke because it reached the end of its life cycle (yes, it’s a 1970s computer system) and, interestingly, WSU begged the Legislature in 2009 for money because they were worried their system was about to collapse. And seven months later it did just that. That’s the real problem and it exists in virtually every area of state government.

Some other random examples when you scratch the surface:

DIS says they need this data center for emergency redundancy. Yet the agency itself has conducted disaster recovery exercises that have shown they are currently barely able to keep Internet access up and running when they do a fail over test to their Spokane backup facility. That’s to say nothing of actual critical data such as foster kids, Medicaid, prison population information and more. So if the big one hits, you can pretty much guarantee that the state’s data is inaccessible.

It’s my understanding that the agency made a decision to purchase a large number of new Cisco routers. I get that. No one ever loses their job for buying Cisco, and a case can be made that they are solid systems. But, since you are buying a total and complete commodity (sort of like concrete or wood for a building…) and they can be very, very expensive, why not do a Request for Proposal (RFP) and get a good price?

Not only did the agency potentially fail to issue a RFP to bargain a better deal with the most expensive vendor in the category, it seems they allowed the vendor to design the entire configuration.

It is well known and common knowledge by everyone in the technology field that Cisco routers are routinely 2-3 times the cost of their competitors and at least 25% higher in annual maintenance fees. 

The Legislature is the Board of Directors for the State of Washington. It’s time that we started to see our role in oversight, audits, performance measurements and outcomes more seriously in the technology arena.