I’m kicking myself for not realizing this earlier.   I’m starting to build an architecture around a new business domain - Pharmaceutical Clinical Trials.  I’ve learned quite a bit about the domain over the last few weeks, but I just realized that if I had read the Wikipedia pages on Clinical Trials, I would have had a great head start.

From a 1987 article:

Much of present-day software-acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software-acquisition problems spring from that fallacy. Hence, they cannot be fixed without fundamental revision–revision that provides for iterative development and specification of prototypes and products.

Incremental development–grow, don’t build, software. I still remember the jolt I felt in 1958 when I first heard a friend talk about building a program, as opposed to writing one. In a flash he broadened my whole view of the software process. The metaphor shift was powerful, and accurate. Today we understand how like other building processes the construction of software is, and we freely use other elements of the metaphor, such as specifications, assembly of components, and scaffolding.

The building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be specified accurately in advance, and too complex to be built faultlessly, then we must take a radically different approach.

Let us turn nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and selfrenewing. The secret is that it is grown, not built.

So it must be with our software-systems. Some years ago Harlan Mills proposed that any software system should be grown by incremental development. [10] That is, the system should first be made to run, even if it does nothing useful except call the proper set of dummy subprograms. Then, bit by bit, it should be fleshed out, with the subprograms in turn being developed–into actions or calls to empty stubs in the level below.

I have seen most dramatic results since I began urging this technique on the project builders in my Software Engineering Laboratory class. Nothing in the past decade has so radically changed my own practice, or its effectiveness. The approach necessitates top-down design, for it is a top-down growing of the software. It allows easy backtracking. It lends itself to early prototypes. Each added function and new provision for more complex data or circumstances grows organically out of what is already there.

The morale effects are startling. Enthusiasm jumps when there is a running system, even a simple one. Efforts redouble when the first picture from a new graphics software system appears on the screen, even if it is only a rectangle. One always has, at every stage in the process, a working system. I find that teams can grow much more complex entities in four months than they can build.

(Hat tip: Jeff Sutherland)

Heh,

After my last post, I discovered the official Web Services With Rails website, and decided to see if I could figure out why the Google w/WSDL example from their “booklet” didn’t work.

But instead, I got this:

NoMethodError in CodeController#googlewsdltest

undefined method `create_rpc_driver' for #

RAILS_ROOT: E:/webfiles/rorbe.com/public/../config/..

(<snip>)

So it wasn’t just me.  Apparently something significant has changed in Rails web services, and the documentation has yet to catch up.

It’s taken some time, but I’ve finally found something significantly unsatisfactory in RoR.  Debugging support.

Essentially, you have to add the breakpoint method to your code wherever you need a breakpoint.  That’s odd, but reasonable.  Then, in a seperate command shell, start up the breakpointer.  (This can all be found here)

A little confusing, but so far, so good.

The irb that is created from this is a ruby shell, not a debugging shell.  This means:

• you can’t step
• you can’t drill down

All you can really do is inspect, tweak values, add code and either abort or exit, allowing the program to continue.

Not a very satisfactory debugging environment.   And it is apparently the “official” debugging environment for Rails.

Unhelpful Advice

Worst part is the comment to this post:

Proper test coverage should reduce a need to debug your application. The question is: how good is your test suite?

You know, that’s what I really was looking for.  Someone telling me that if I was better at writing unit tests, I wouldn’t need to  use the debugger.  Of course, this commenter has no idea what kind of problem the writer is having.
That was probably the most annoying thing in this whole scenario.  Ruby programmers arrogant?  nah….

Resolution 

Anyways, using IRB, I was eventually able to figure out that, as far as I can tell, my  RoR (ruby 1.8.2, rails 1.1) installation is inconsistent with the documentation, from a number of angles.  Specifically:

• the SOAP::WSDLDriverFactory has a create_driver method only.  No create_rpc_driver method, like the documentation says.
• the Web Services With Rails document from O’Reilly is the one that told me to use create_rpc_driver, so it too, is apparently incorrect.

(Caveat: it is perfectly likely that somehow my installation is incomplete or out of date, but I don’t know how to check, beyond simply looking at my versions of Ruby and Rails.  I suppose there may be an older version of some libraries on this machine, but as far as I know, that is not the case)

Myspace is trying to figure out what to do about adults preying on minors on Myspace.  EKR has some comments on the overall ineffectiveness of the approach.

My thoughts:

• One of the things I’ve learned in my time is that a fairly low barrier still seems to keep a lot of people “well behaved”.   There’s no logical reason for it, but people still respect short fences.
• I am reminded of Orson Scott Card’s masterpiece - Ender’s Game, and the “hobby” developed by his brother and sister.  Do we want more control?
• I’ve played quite a bit of Dungeons and Dragons Online over the last few months, and I’ve played with a number of kids (Based on their voices in voice chat), and a number of people who were probably kids based on their typing.  Even if kids pretend to be adults, there are still “tells” that indicate they are kids.  Perhaps a “I think this account is a kid” button reputation system might be in order
• As difficult as it may be to pull off, the “right” solution is to have a good relationship with your kids so you know what is going on in their lives.

I found a rather amusing set of 80s music videos this morning.  This one by Dead or Alive, in particular, summarizes the entire genre of New Wave music videos.

A long time ago, I read Fred Brooks’ “The Mythical Man Month” and “No Silver Bullet“, and I agreed with it.

But now, many years later, I think he was wrong.  I think there are lots of silver bullets that make software easier to build.

The problem is that what we need to build keeps getting more complex, providing diminishing returns on the advances in technology.

I realize this is a controversial statement to make, so let me provide some examples:

Pong

How hard is it to make the game Pong for Windows?  I think most people would agree that it is close to trivial - there are libraries to manage launching the game, libraries to track sprites and mouse movement, and libraries to manage text on the screen.

But 25 years ago, it wasn’t trivial.  Everything had to be written from scratch, often in assembler.  That’s why Pong was so entertaining - nothing had ever been built like it before.

The core concept of Dr. Brooks’ argument is that there are no order-of-magnitude changes in software development.  But I would argue that over the course of a generation, Pong is now an order-of-magnitude easier to build than it used to be.    No one change made this so - but a host of smaller, incremental changes led us to the point where a teenager can write a Pong game in a weekend.

Spreadsheet

The spreadsheet was once one of the most sophisticated applications ever produced, and it changed the world.  Back when it was first developed, Lotus 123 and Quattro and Excel battled for dominance providing basic math, and a suite of simple functions and graphing tools.

Nowadays, people write spreadsheet software in JavaScript to run in a browser.

Again, something that was once very hard, and very sophisticated is now, relatively speaking, trivial.  (This is not to imply that producing a JavaScript spreadsheet is trivial, but it is trivial relative to the production of the first spreadsheet program).

And why is that? Because the tools are better.  The libraries are richer and the graphics and communications pipelines are easier to use, more accessible and easier to debug.  Here’s a thought experiment - using nothing but software from 1990, produce an online spreadsheet that people can use with their personal computers.   You’ll need to provide:

• Network socket code
• Server socket code
• Dynamic download/upload code
• Zero-click install
• etc
• etc
• etc

The Enduring Legacy

The reason that the assertion that “There are no silver bullets” still seems to be true is twofold:

1. People expect their applications to do more than they used to.
2. Business logic isn’t any easier

Increasing Expectations

Over time, Pong isn’t any fun anymore, and a basic spreadsheet doesn’t help you with your more advanced problems.  You need new help, more sophisticated games, with 3d capabilities, and you need macros and programming built into your spreadsheet, and change tracking, and audit trails and so forth.  As the scope of the application expands, it hits the edges of the known, and forces the programmer into the unknown.  At the edge, in the unknown, there are no silver bullets.  And so the general problem of software remains hard, even as the specific elements grow easier over time.

Business Logic

The second problem is that business logic doesn’t get any easier.  If anything, it gets harder to capture all the rules that govern more sophisticated applications.   Business logic is a human phenomenom, and it is not well suited for automation.  Certainly there are tools that claim to improve the collection of business logic, and engines that claim to let people write business logic in human terms.   But in my experience, these are, at best, modest improvements.  The bulk of the sophistication in capturing and coding business logic remains.

In Conclusion

• I think software development would be easier if we could build apps on older, well-proven technology instead of newer, less supported systems
• But as software problems become “standardized” and well-supported they are no longer valuable
• Because people expect their software to do valuable things, they will always insist that at least some of the work be done at (or near) the cutting edge
• Business logic is much harder to standardize and takes up an increasing amount of the complexity in modern software

Questions

• Do you disagree?  Can you show me examples of older systems that are just as hard to build now as they were when they were first developed?
• If we could convince users to accept somewhat older technology, do you think it would make software development easier and faster?
• Are there societal changes that could make business logic easier?

Welcome to the June 3rd edition of the Carnival of Agilists - the bimonthly roster of the best posts from the Agile Blogosphere. Sorry about the delay, we had some crossed wires.
Individuals and interactions over processes and tools

• Simplicity ain’t easy - a long, thoughtful post by Brad Appleton. Lots of interesting points and a great linkography of interesting websites and documentation at the end.
• Scott Sehlhorst writes about the same topic, but in the context of testing - Test Smarter, not Harder.
• Mishkin Berteig takes a page from Malcolm Gladwell, and talks about how people problems are easier to understand and accept than technical problems. Can you reframe your problems so they’re about people, not technology?
• Jon Kern gave a talk at Dave Nicollete’s company, and it didn’t go as well as Dave had hoped. He shares his thoughts on Jon’s approach, what might have gone wrong, and what to do with a semi-hostile audience.

Working software over comprehensive documentation

• Glen Alleman says that software development methodologies are a zero sum game. I vehemently (but respectfully) disagree. What do you think?

Customer collaboration over contract negotiation

• In the spirit of agile development, Tom Looy asks his son - How long is a piece of string?
  
• Skip Angel talks about 5 types of bad estimators and what to do about them.

Responding to change over following a plan

• Dave Churchville says that agile methodologies are far more disciplined at the business level than BDUF models. This connected well with me - agile development is focused on the discipline of getting the customers what they need.
• A similar post from Ed Gibbs - Early Releases provide early return-on-investment.

Case Studies and Analysis

• Jeff Sutherland and some friends have produced an interesting case study of a large-scale Scrum/XP project using partially outsourced teams.

Books

• Clarke Ching is working on an “agile novel”, and of course, he is releasing his work early and often. You can read it, and provide feedback here.

Previous Editions
All previous editions of the Carnival are referenced at the Agile Alliance website.

Join in the Fun!
Have something that you think is worth sharing? Don’t be shy! We love new ideas and insights. Send us a link to your post at agilists.carnival@gmail.com.

The Carnival is published on the first and third Thursday of each month. If you would like to participate, please send us a link to your post at agilists.carnival@gmail.com. Or, if you prefer, use this handy dandy carnival submission form.