Sacramento Techies – we will replace your system for 1/10th the cost or I will eat your mainframe

According to the Sacramento Bee, there are some big legacy software problems in the State of California! One of their legacy beasts, the payroll system in this case, is so unwieldy that it can no longer respond to necessary, legislated change in a timely manner.

“Democratic state Controller John Chiang said Monday it would take at least six months to reconfigure the state’s payroll system to issue blanket checks at the federal minimum wage of $6.55 per hour, though Schwarzenegger insists such a change should occur this month.”

No wonder there is a gap between what the Governator expects and what the people in the trenches are telling John Chiang. How could it possibly take “at least six months” to make such a trivial change?

Well it may seem crazy, but the estimate doesn’t surprise me. I’ve seen that kind of crazy many times:

• Adding a check-box to a DMV system written in PL/1: 6 months and $100,000.
• Changing the sales tax rate from 6% to 5%: 9 months and $150,000.

Par for the course when your computer system is written in a “Vietnam-era computer language.” Love that turn of phrase! Shows how bad legacy systems can really get. But it’s not all COBOL’s fault. It’s extremely difficult to prevent computer systems in general from getting worse over time, in the face of constant change. It’s a kind of progressive fossilization that turns nearly any system into a rats nest over 20 or 25 years. We have already had requests to look at systems written in Visual Basic and even Java!

So why are these systems not “upgraded” before they get too hard to change?

“California has tried to modernize its payroll system throughout the past decade, dating back to former Controller Kathleen Connell. It has faced numerous delays as state legislators have avoided investing the $177 million it now will cost.”

One Hundred And Seventy Seven Million Dollars!!! That’s like the annual GDP of a small country. That’s like buying a bright-red Lamborghini for almost two thousand bureaucrats!

"I eat mainframes for breakfast"

This estimate is ridiculous. The article says that this system has “tens of thousands” of lines of COBOL code, making it very small by modernization standards. A few million lines would be more typical. And Gartner suggests that replacing a legacy system can be done for maybe twenty dollars a line. So if the payroll system in question has say 100,000 lines, and Gartner is right, it would be a two million dollar project. So the estimate in this article is inflated by… by… I can’t even compute it in my head… 8850%.

I suppose I can’t blame the incumbent technology people for making estimates like that. We have lived through an era that has “proven” that modernizing a legacy system is an incredibly risky venture. So just like they taught us in Computer Science school, project estimation always involves tripling the initial (science-based) estimate and then adding one to the unit of measure, i.e. 2 weeks becomes 6 months, 4 months becomes 12 years, etc. Better to make the project financially impossible than incur the risk of failure.

So I challenge the technology people in the California government. If we can’t re-architect that system for one-tenth of the current estimate, I will eat your mainframe.

- Tom Metzger, Director of Sales Support @ MAKE

COBOL Architecture Astronauts: Mountain MOVErs?

This is another entry in my ‘Legacy development paradigms are the greatest source of legacy problems’  series.

Architecture Astronauts are developers who create a complicated solution to a simple problem in order solve a bunch of other potential, and probably imaginary, future problems. Every day developers all over the world fight their inner Architecture Astronaut, and every day, thousands of them lose that fight.

Over the last couple of years I have seen a lot of legacy systems built in COBOL, NATURAL, RPG and PL/1.* Expecting overly designed hyper-abstract “frameworks”, I am always surprised at the incredible pragmatism and restraint shown by the developers of those systems. At worst you’ll find a dynamic call, where the name of the program being called is passed in as a variable, or a common error routine. 

Legacy mainframe code usually is incredibly verbose, not necessarily because the language requires you to write a lot, but because legacy developers did not reuse business logic or at the very least they try to avoid doing it. Or: 20 years ago code reuse was the stuff Architecture Astronauts came up with while mainstream mainframe developers avoided it.

Programmer A: So, I need to validate this postal code. Didn’t you already write code that does that?

Programmer B: Oh yeah, here: copy & paste the bugger!

Programmer A: Eh: can’t we move that in a common function?

Programmer B: ARCHITECTURE ASTRONAUT! ARCHITECTURE ASTRONAUT!

As usual I am exaggerating, but there is a core of truth there: reuse was not harder to implement than in modern languages, but developers just did not do it on a regular basis. I’d like to say that that was caused by unsophisticated specification and design phases and a less mature software engineering practice, but that is not true: most, not all, reuse in modernized development is done by developers, while implementing, after the designs were created.

Developers communicate business needs in code, and computer restrictions into conversations with users. But the whole practice of software development itself is also guided by a higher level of fads: assumptions about what can, should and may be done in code and how it should be implemented. The more I think about it, the more important these ‘paradigms’ seem to be compared to the individual choices of each developer or team. The quality and business alignment of your system is heavily influenced by the paradigm that was used to develop it.

Next in this series: ‘SOA: where Architecture Astronauts mate’.

- Mik Lernout

* Note that I always refer to legacy languages in uppercase but I do not line break after 80 characters. Inconsistent, I know.