Thinksum

Is 4.0 the new 3.0?

In the 1990s, key products often reached maturity in their 3.0 release.  Windows 3.0 and Notes 3.0 were two of the most successful software products of the early 90s.

In 2007, especially at smaller companies, we have to move fast to win in the marketplace and our release cycles are compressed.  Software libraries, open source, platforms and tools are all more advanced, so development teams don't have to code everything themselves and we can build and test more quickly.

At TNT this week we are rolling out our 4.0 release.  Customers began installing last night although our public launch will be in a few weeks.  We're very excited that this release represents a new level of maturity in our product and our capabilities to protect critical servers and information, while simultaneously we continue to lead innovation in internal network security.  It's a 4.0 release, but we feel young and vibrant, just like 3.0 used to feel.

Congratulations to the team and thanks to all who helped!

AJAX Elements

We're implementing some new product interfaces AJAX-style.   Our needs are modest in that our interface is directed at a very limited number of people, not a large consumer base.  We were starting with a Java back-end that already had most capabilities exposed with a variety of methods and web services.

We evaluated a variety of approaches to designing he new interface and how it would connect to our back-end.  Some AJAX frameworks (like GWT) are Java-based and attempt to connect on the server side.  We evaluated a number of frameworks and libraries that cut across different capabilities and approaches, mainly GWT, Echo2, Wicket, Tapestry, Lazslo, Dojo and  Prototype (some other such as ZK, Zimbra/Kabuki and MochiKit we looked at a little but didn't make our first cut for various reasons).  The key factors that we evaluated were:  ease of development, ease of integration with our Java back-end, performance and scalability, richness of capability, and level of community support.

In the end we decided to go with an approach where we have both a client framework and a server framework.  We decided to use Dojo and Wicket.

Finally, we wanted some charting and graphs that would look great and provide some interactivity.  We have limited Flash skills in-house, so we basically decided to look at pre-built widgets or libraries we could use.  We took a look at Dojo charting but decided the look and capabilities were not mature enough yet.  We ended up evaluating two very nice commercial Flash chart libraries FusionCharts and AnyChart, both of which proved solid and which had very reasonable licensing terms.  I won't say which we chose because I think both should be evaluated if you are in need of similar.

Development now is fairly far along, and we feel the Dojo-Wicket combo is working well for our purposes.

Most smart people in the world ...

... don't work in your company.

There's a quote like this I once heard, I wish I could remember who said it.  (if you remember, let me know)

The point is that unless everyone in the world works for you, there are a lot of people outside the small group you work with who are really smart.  Maybe smarter than you.  And if you can take advantage of their knowledge and skills, you benefit.

In many parts of business, it is common practice to seek outside advisors to help.  But in engineering groups, and in start-ups, I think it's often unappreciated how cost effective and beneficial it can be to get some of those smart outside people involved.

For example, let's say you're building a new database application that needs to be highly scalable and perform very well.  Lot's of people have done this, maybe even you yourself.  And there's lots of books and online information about best practices.  There are great design tools.  So you might figure you have the staff, knowledge and resources to do it yourself.

But, assuming you really want to succeed, how much benefit would you get if you brought in another really smart person, an outsider, to review your design for a few days or a week before you start to implement?  In my experience, you'll get a lot ... some of it might be design improvements, some of it might be good advice for pitfalls to avoid in the future, and some of it might just be that smart person's experience and thought process rubbing off on members of your permanent team.

But if you're in a startup you might think "well I can't afford it."  I'd say think again.  A day or two of consulting time doesn't cost much at all, and if you can't even afford to pay that you can probably find someone willing to do it in off-hours for a few good meals and maybe a nice thank-you gift.

I'm in a 40 person start-up.  We're venture backed, but we try to be very expense aware.  Our small team is stocked with really smart people from Georgia Tech, Carnegie Mellon and other places.   But as VP Engineering I look for every chance I can to engage really smart people from outside to help us along, review our designs, or generally look over our shoulders and give us advice.  Whether it's database tuning, security product integration, cryptography or something else, it's a fact that most of the smart people in the world work somewhere else.

The 50% Code Limit

Even in a start-up where you want high productivity and you want to deliver new product releases yesterday, I believe your developers can code too fast.  If programmers are writing new code more than 50% of their time, I'd contend that quality will be compromised and you'll pay in the end either with delayed releases or sub-par quality (or both).

In other words, if you don't invest 50% of a developer's time purely towards quality *during* code development, you will need to invest more than that later (by my experience I'd estimate 2X or 3X investment when you count the time from QA and Support).

Of course, it's not always so simple to get programmers to slow down and spend more time on quality.  But I think the 50% code limit -- no programmer will spend more than 1/2 their time writing code, and all other time will be spent towards quality -- is a good way to discuss and plan for excellent quality.  This requires you to build project schedules accordingly -- if a feature will take 5 days to develop (write the new code) it should be allotted 10 days on the schedule, and it should be very clear to each programmer that the extra days are for finding and eliminating all bugs, for working with QA to write clear test plans, for adding automated tests to the build, etc.  You don't move on to the next feature until the quality work is done for the last feature.  And the extra days are not padding in case the coding takes longer than expected ... if the feature takes longer than expected, then the days allotted for quality need to be increased too (so every 1 day slip in coding time results in a 2 day schedule slip).

You also need to be careful because many programmers will want to move on to their next task before the quality work is done.  For instance, a programmer may spend 5 days writing code, spend 2 days testing, and then believe that there's no more they can do for quality so they don't need the extra 3 days allotted in the schedule.  But have automated build tests been added?  Have tests been documented?  Have a number of realistic customer scenarios been tried?  Chances are the 2X allotment of time will not be enough to do everything that could be done for testing and quality, so it's necessary to be very skeptical if someone says they don't need all that time.

I'd be very interested to hear if people have experience with this or similar approaches.

Does QA Matter?

Marc Hedlund on the O'Reilly Radar has a write-up of software development practices he keeps running into at Web 2.0 companies.

Most of them make good sense.  Except one.  He says he's seeing a lot of start-ups with no QA team.  Even worse, it sounds like a bunch of start-ups have convinced themselves that this budget-saving measure is actually the right way to go.  The rationale seems to be that developers will be more careful if they know they're doing their own QA, and since the web-based software can be delivered to users transparently then users can find the bugs.  Why fix anything that wasn't found by a user?

Ahem.  Since I work in a security software company, let me state the most obvious:  if you have security problems in your software, you're better off catching them before your users.  And data corruption too ... imagine if Typepad introduced a bug that when I was trying to write a post I could accidentally delete my entire blog?  Or doing something to my blog could damage someone else's blog?  I sure hope Typepad has a QA team (and I bet they do).

And what if the user finding the bug is a product reviewer, or a key decision-maker at a potential customer?  Again, it pays to find and eliminate key problems before users find them.

Does QA matter?  Of course it does.  While some start-ups may decide to live without a QA team for awhile (something I wouldn't recommend having been through a few start-ups myself) that shouldn't be taken as some breakthrough in software development driven by Web 2.0.  This isn't new.  Lots of software start-ups have been under-investing in QA for a long time, and often with a similar sour grapes rationale -- "we can't really afford good QA (either in money or time) but we realized that we don't really need it."

The idea that developers and users can provide sufficient QA is dangerous at best.  Expecting that developers will be more careful without a QA "safety net" is a fallacy -- good developers are careful with or without QA, but I've never seen a developer who could find all their own bugs (and the ones they miss are often the worst).  And fobbing QA off on users is a great way to frustrate users and leave yourself open to competitors who have better quality.

I've written about this before, but I believe the main issue is that software QA (and product quality) remains under-appreciated, so in companies large and small it often suffers from under-investment.  The costs of poor quality are many:  failed product trials, low customer loyalty, increased customer support.  The best companies recognize that quality is one area where it almost always pays to invest.

Software is a Team Sport

It's not news that I'm a big sports fan.  In my office I have a Red Sox baseball and some memorabilia on the wall including the picture of Jackie Robinson stealing home.  And I have another prized possession, a signed copy of John Wooden's Pyramid of Success.

For those who don't know, John Wooden was the UCLA men's basketball coach from the 1950s through the early 1970s (he was also an All-America basketball star at Purdue).  In the 1960s and 1970s he led UCLA to a seldom seen run of success, winning 10 national championships in 12 years.  Because my dad went to UCLA and was a season ticket holder, I had the privilege of attending many games during those years and watching Coach Wooden closely.  I even went to his basketball camp one year when I was about twelve.

The things I remember most about Coach Wooden is that he stressed attention to detail, that he advocated worrying about yourself and not your opponent, and that he treated each person on his team exactly the same way, whether it was Lew Alcindor (Kareem Abdul-Jabbar), Bill Walton or Pete Trgovich (if you know who he is you're a good college basketball fan).  Success came from each person doing their job within the team structure.

We live in an age when individuals are elevated to silly proportions, and the computer software industry is no exception.  From Dan Bricklin to Andy Hertzfeld to Tim Berners-Lee to Brin/Page (all people whose achievements I highly respect) many people seem to believe that the major work is done by stars.  And many executives espouse a performance rating system that focuses on identifying and rewarding the top individuals.

But software is a team sport.  I repeat -- software in almost all cases is produced by a team.  Success comes when each individual does their part within the structure and toward the goals of the team.  We remember Bricklin and Hertzfeld and Berners-Lee and Brin/Page because things they worked on were successful, maybe even great -- and those things were successful because many other people i.e. the team worked with them side-by-side.  As Coach Wooden said, "the main ingredient of stardom is the rest of the team."

If you agree with this thesis, then you begin to see that it has profound implications.  Hiring for instance -- it's not always important to hire the most talented person, but a person who can mesh with the team and complement the existing players and skills.  Or a person who has versatility and can possibly fill many roles as needed (the Bill Belichick philosophy).  And when it comes to performance reviews and rewards, it can be disastrous to single out individuals (positively or negatively) in ways that create internal competition as opposed to keeping everyone focused as a team on your real competitors.  I'm a firm believer that the team should be rewarded and reprimanded fairly and equally -- we succeed or fail together, and we benefit or don't as a result.  Of course certain people may be paid more or less based on experience and skills, and certain people may need to be called out when they are not holding up their end of the bargain, but that's just fair.  What isn't fair is giving your top 25% of performers a bonus and not giving anything to anyone else simply because that's your business philosophy, and nothing is more guaranteed to destroy team chemistry.

Maybe this seems self-evident, but of all the software companies I know large and small, very few seem to me to have really embraced a team view.  A strength of many startups is their technical team, which is why some of the most amazing progress comes from startups, but once management gets involved the team view is often lost and the team breaks down.  Most companies still seem to take what I'd call the star system approach, trying to find and keep stars without enough attention to the team dynamic.  But the star system usually doesn't lead to success, and the limited success it may achieve usually doesn't last for long.

The happiest programmers I know are ones who work in a strong team environment, where they've developed a sense of cameraderie and a successful blending of skills that has put them on the road to product success.  As in sports, building a winning team isn't easy, but the way to win is to build a team.

Software Managers Start Your Blogs

I really wish there were more blogs about nuts and bolts issues in software engineering.  Lots of folks are writing about exciting new product areas, cool things that they're interested in, predictions for the future and the like, but not so much about the basics of what we're learning and experiencing in day-to-day software development.  If we shared more experiences about what's working and what's not for project planning, product design, architecture design, release management, quality processes, recruiting, retention, etc. then we'd all benefit.  The ability of blogs and the read-write web to facilitate an open, cross-connected forum of ideas and people is a perfect vehicle for faster learning and feedback. 

I'm learning a lot from blogs about all kinds of things going on in the world, but I'm learning diddly about software engineering. I'm not talking about long essays on the inner-workings of Ruby on Rails or the benefits of Python vs. Java, all of which totally have their place and are useful, but I mean more Tom DeMarco Peopleware kind of stuff right from the actual front lines (why the heck is Tom DeMarco not blogging, anyway? although that's a different issue).   Joel Sposky does some of it, but other than that ... ??

Are these blogs out there and I'm missing them? (please let me know, really)

I think maybe a lot of folks think that blogs have to be about something exciting and different, or with a strong, unique point of view.  But we're not all Dave Winer.  To think it all has to be catchy and commercial is vastly underestimating the value and true usefulness of this medium.

Anyway, I'll try to put my money where my mouth is and get to work on actually contributing more to the conversation myself.

What Would Jackie Robinson Do?

Although he played before my time, when I was young I was steeped in the lore of Jackie Robinson, to the point that I regard him as a personal hero and would call him the greatest (U.S.) athlete of the 20th century.  I grew up in southern California and my father ran track at UCLA, where Jackie Robinson preceded him as an unparalleled 4-sport star in track, football, basketball and baseball.  I also grew up a Dodger fan, and the LA Dodgers in the 70s were still very connected to the Brooklyn Dodgers of the 40s and 50s, the team that was led by Jackie Robinson.  Jackie of course broke the color barrier with the Dodgers, which is what most people rightly remember him for.  But when I think of Jackie Robinson, I think of him stealing home.

I keep a large picture of him stealing home in the 1955 World Series in my office, and here's why:

Most key choices we make in software development involve some balance of risk and reward.  If you try something new or different or unexpected, you're taking a risk and you've determined (presumably) that the reward is worth it.  If the risk is not worth the reward, then you don't do it.

Products that make plenty of money and have lots of customers (like Microsoft Windows) don't need to take risks.  Anything new needs to be carefully chosen and carefully done, and they can afford that approach.  They're sort of like the New York Yankees -- long-term winners who can afford the best of everything so they don't have to go out on a limb.

But when you're not the proven leader or you're in a new arena there comes a time when risk is appropriate and making the unexpected move might just win the day.

In Game 1 of the 1955 World Series against the Yankees, after having lost 4 World Series to the Yankees in the past 7 years and having never won a World Series in Brooklyn, Jackie Robinson was standing on third base with his team trailing 6-4.  Jackie was already in the later stage of his career, and though he was famous for stealing home he was maybe half a step slower than he used to be.  The decision to steal was always left up to Jackie, so there was no signal from the dugout or the third base coach.

Standing there on arguably the biggest stage in American sports (at the time), Jackie made his decision, his own risk-reward calculation, and he broke for home.  The tag by catcher Yogi Berra was a virtual tie, and although Berra argued vehemntly Jackie was safe.

The Dodgers still lost the game, but they went on to win their first World Series -- the only World Series they would win in Brooklyn and the only World Series that Jackie Robinson would ever win.  And people still remember Jackie stealing home as the emblematic moment, the signal that somehow the upstart Dodgers were finally going to find a way to beat the Yankees.  Jackie Robinson was the on-field leader of that team, and his competitive fire and willingness to take risks were the elements that made him such a great ballplayer and helped make those Brooklyn Dodgers memorable.  Jackie Robinson rose above the game and was truly heroic for the way he carried himself in the face of enormous racial pressure, but if you search for images of him you'll find that the lasting action image is Jackie stealing home.  It is one of the rarest, most audacious of baseball plays and something rarely seen anymore (and now only usually as part of a double steal).

In software as in many types of business, you can't always play it safe and succeed.  So when I'm faced with a key decision, I look at the picture on my wall.  What would Jackie Robinson do?

Year 2K5

After Year 2K you would think that everyone who had a timebomb in their program would make every effort to get it out and let the world know.  Last week at Epiphany, and also at other companies like Cisco and BEA, folks had to scramble because on July 13th Sun notified the world that a widely used Java library had a component that would become non-functional two weeks later, on July 27th.  The Java Cryptography Extension (JCE) package is used by many J2EE applications to encrypt passwords or other sensitive data (and as of version 1.4 it is actually formally part of J2EE).  This error meant that after July 27th, older applications not recently upgraded would fail.

I'm not trying to bash Sun here.  The version with the timebomb had been replaced by a newer version which eliminated the timebomb problem a few years back.  But clearly, we at Epiphany (where we work hard to monitor these things) and others like Cisco and BEA were somewhat blindsided by this, since we all had to release emergency upgrade patches and notices to customers in the last week.  Luckily, at least at Epiphany, we seem to have effectively addressed the issue at customer sites.

The morale of the story is this:  any software that you write should not have any timebombs, period (free trial versions of software aside).   If the software is widely distributed (and when you're building it that's usually your hope) it will probably be in use much longer than you think.  If you do have a timebomb in some software you've written and distributed, you owe it to your users to do much more than normal to alert them as early as possible.

Labor vs. Creator

About a week ago on NPR I heard a report on overtime pay for software company employees, spurred by the class action lawsuit a group of employees brought against Electronic Arts.  This detailed report from one employee's significant other describes the policies and practices that led to the lawsuit (mandatory 12 hour days, mandatory six and then seven day weeks, no overtime pay).

Under legal pressure, EA has agreed to institute some overtime pay.  But on the NPR report, an unrepentant company spokesperson Jeff Brown indicated that the company does not take a positive view of complaining employees and the expectation of overtime pay, saying that EA is surprised they want to be treated as labor, instead of creators.

Overlooking the silliness of this statement -- since the employees are both labor and creators, and if anyone is treating them as labor it's EA management -- it also raises another interesting question.  What is the best way to get maximum productivity from a team over an extended period of time?

Here are the simple keys to productivity I've found over the years:

• Hire good people who enjoy the job
• Give them all the tools they need
• Don't make anything mandatory
• Provide individual financial rewards for exceeding expectations (in productivity, quality, creativity or teamwork)
• Let the team share in the results (bonuses tied to product revenues, stock options and the like)

My theory is that motivated, happy teams will decide when it's appropriate to have extended hours and will work them accordingly, with much more allowance and tolerance for individual's personal schedules and family lives.

The most mystifying thing to me about the EA situation is that management would have ever thought the best way to get a new product out the door sooner would be to micro-manage people's work schedules (and thereby their lives) and to devise schedules that required extended overtime.  I wish we had a lab where we could run two identical projects in parallel:  one managed the EA way, and one with no mandatory work schedule but with the same deadline.  My guess is that the second team would deliver a much better product, probably sooner too, and definitely the team would be much happier at the end.