Sacramento Techies – we will replace your system for 1/10th the cost or I will eat your mainframe

According to the Sacramento Bee, there are some big legacy software problems in the State of California! One of their legacy beasts, the payroll system in this case, is so unwieldy that it can no longer respond to necessary, legislated change in a timely manner.

“Democratic state Controller John Chiang said Monday it would take at least six months to reconfigure the state’s payroll system to issue blanket checks at the federal minimum wage of $6.55 per hour, though Schwarzenegger insists such a change should occur this month.”

No wonder there is a gap between what the Governator expects and what the people in the trenches are telling John Chiang. How could it possibly take “at least six months” to make such a trivial change?

Well it may seem crazy, but the estimate doesn’t surprise me. I’ve seen that kind of crazy many times:

• Adding a check-box to a DMV system written in PL/1: 6 months and $100,000.
• Changing the sales tax rate from 6% to 5%: 9 months and $150,000.

Par for the course when your computer system is written in a “Vietnam-era computer language.” Love that turn of phrase! Shows how bad legacy systems can really get. But it’s not all COBOL’s fault. It’s extremely difficult to prevent computer systems in general from getting worse over time, in the face of constant change. It’s a kind of progressive fossilization that turns nearly any system into a rats nest over 20 or 25 years. We have already had requests to look at systems written in Visual Basic and even Java!

So why are these systems not “upgraded” before they get too hard to change?

“California has tried to modernize its payroll system throughout the past decade, dating back to former Controller Kathleen Connell. It has faced numerous delays as state legislators have avoided investing the $177 million it now will cost.”

One Hundred And Seventy Seven Million Dollars!!! That’s like the annual GDP of a small country. That’s like buying a bright-red Lamborghini for almost two thousand bureaucrats!

"I eat mainframes for breakfast"

This estimate is ridiculous. The article says that this system has “tens of thousands” of lines of COBOL code, making it very small by modernization standards. A few million lines would be more typical. And Gartner suggests that replacing a legacy system can be done for maybe twenty dollars a line. So if the payroll system in question has say 100,000 lines, and Gartner is right, it would be a two million dollar project. So the estimate in this article is inflated by… by… I can’t even compute it in my head… 8850%.

I suppose I can’t blame the incumbent technology people for making estimates like that. We have lived through an era that has “proven” that modernizing a legacy system is an incredibly risky venture. So just like they taught us in Computer Science school, project estimation always involves tripling the initial (science-based) estimate and then adding one to the unit of measure, i.e. 2 weeks becomes 6 months, 4 months becomes 12 years, etc. Better to make the project financially impossible than incur the risk of failure.

So I challenge the technology people in the California government. If we can’t re-architect that system for one-tenth of the current estimate, I will eat your mainframe.

- Tom Metzger, Director of Sales Support @ MAKE

Re: Making meals from your mainframe leftovers

A great post on leveraging your existing legacy assets and infrastructure: Making meals from your mainframe leftovers over at Smart (Enough) Systems with a link to a great article on the California DMV modernization as an example, which is a reference project in the legacy modernization industry.

Modernizing a system piece by piece is a solid, pragmatic, way of extending your existing system, and a method to partition your project risk. I would add though to that when you take in account the back integration challenges which are introduced; the reliance on legacy domain experts and overall complexity actually increases. (see Modernize your Cat? Sure.)

We just modernized all of the DMV systems of Nova Scotia (the 7th biggest economy of Canada ) in about 3 years, and they can now dump their mainframe and its maintenance cost, at a savings of over $280,000 per month. Sometimes biting the bullet and increasing the overall scope, can make things easier. Changing modern systems is a hell of a lot cheaper than bolting on new, or replacing existing, functionality on 20 years of cruft.

- Mik Lernout

Celebrating A Modernization Deployment

Over the last couple of months we, with the help of our great partners Barrington Consulting and Service Nova Scotia, have deployed a new system for both the Vital Statistics and the Department of Motor Vehicles of the province of Nova Scotia, into production. Hundreds of people all over the province of Nova Scotia lined up to get copies of their birth certificate and register their new truck. And it worked. Some of the regulars told us it was one of the smoothest go-lives they had ever experienced in the province.

So: We went in, aligned and analyzed their existing, expensive, mainframe application and replaced 2,393,000 lines of Natuaral/ADABAS code with 483K lines of Java and 195K lines of XHTML, and then successfully deployed the sucker.

You could also rephrase that as: we went in, took out a barely beating heart, and put in a brand new one; patient got up, thanked us and walked away. The enterprise software equivalent to jumping over a car.

One could call that an exceptional achievement, maybe even heroic, but we like to consider ourselves to be pretty exceptional and heroic to start off with. For a lot of us, me included, this project was a huge confirmation of our tools, our methods and our people. We know what we are doing, and we are about to do it again and again.

We have already started several other engagements, and have improved our processes and methods to yet another level. While some companies have to introduce continuous improvement, we at MAKE could not imagine not trying to do it better, faster, cheaper and, most importantly, safer, every single time. So what does a team do when they beat the industry average and insurmountable odds: they go out for a couple of pitchers of beer. And then go back at it, because we still have a lot of systems to go.

- Mik Lernout

Is “Business Centricity” a buzzword yet?

No, at least not yet. Google returned about 430 instances, many relating to a GE product. What I’m getting at is the shift that information technology vendors need to make in order to serve the mission of their customers, whatever that mission looks like. Gone are the days when monolithic IT approaches were brought into unsuspecting organizations under the guise of “re-engineering”, a trend that began when I was a BCIT neophyte studying Operations Management in the early 1980s. We laughingly called ourselves the “Op-Man Axemen”, not realizing that the Re-engineering movement would actually put a lot of people out of work in favor of automation solutions.

These automation solutions now form the fabric of what is called “legacy” systems today. Characterized by stovepipe and (often) client/server architecture, arcane and/or obsolete code (except COBOL still soldiers on), and green screen terminals, legacy systems represent a crumbling foundation upon which many medium and large-scale enterprise and government operations are built. It is no wonder that the term “legacy modernization” is becoming an increasingly-heard siren call.

The market has, of course, responded with a flood of out-of-the-box “solutions” for these old software systems, mostly in the flavor of technology fixes that leaves the code base as-is but connected to the web, or converted into a modern language, resulting essentially in a GIGO* situation. Tools-only approaches still ignore the business, and are therefore the antithesis of “Business Centricity”.

At a provincial government Department of Motor Vehicles (DMV) modernization project we recently completed, the story was much more than dollars saved (~$280,000 per month), screens reduced (4,100 down to 650), reduction of code base (2.4m LOC of Natural to 250K LOC of Java) and elimination of batches. These are just the IT and brute cost metrics. The real story lies in the significant streamlining of business operations that result in lower costs and improvements in productivity, customer service, security and business intelligence, just to name a few.

Massive hierarchical menus were reduced to attractive web-based UIs with drop-down menus. The entire system now is entity-based, around the driver, so any piece of information entered, such as a license plate, ticket number, etc. automatically locates the driver and exposes all of the relevant information about that individual. Data is now shared with the (also modernized) Vital Statistics system so birth certificates, marriage/divorce certificates, etc. can all be accessed directly from the driver. Taxes can be calculated at the time of a transaction, such as a vehicle purchase, rather than shuffled off to a tax group for later deciphering. You get the picture. These are benefits above and beyond the computer system operating costs, and they provide tangible ROI for many years to come.

We have found that the actual modernization of the application comes at the end of the process. It must be front-ended by careful consultation with business stakeholders, IT stakeholders and C-level executives. In order to serve the business needs for agility and competitive advantage, legacy replacement strategies need to grow from a business-centric approach.

Perhaps someday we’ll see Business-centric Modernization, or BCM. Just what we need. Another technology industry abbreviation to kick around.

- Chris Dollard

* Garbage In, Garbage Out