Judicium Dei – Trial by Coding (Part 2)

(This is definitely a long post, but this part is where things blow up, so I’ve got a bit to say.)

Calm has returned at this point, now the last week of September. I can relax in my chair at work, enjoy the tasks at hand without any desire for the day to end, and I feel comfortable wasting time. There are no more issues popping up that have to be finished by the end of the day or the client can’t function when they start again their next day.

Last week was also an easy week by any measure, but I still felt anxious for the work day to end, which got me that much closer to the weekend. Even though I knew in my head that the calm had settled in for sure by this time, as there hadn’t been any pressing work for a couple weeks, and I’d shifted back over to focusing on our newest “alpha” version of the application, I still felt impatient with an eagerness for closure of that work day. Stress builds up and leaves a residue that takes time to clear away, and there really aren’t any quick fixes nor fun camps that help speed up the process.

The client is now quite happy and working away in their processes with the new and improved application much as they were happy leaving the business requirements meeting in London a few months back. At that time, we didn’t have a new and improved application, but we had some tentative plans and solid designs for what we’d end up with. My trial by coding was still roughly 6 weeks away, and all seemed good.

My company and I had decided before and during the meeting that we’d try taking a slight risk (or what we had assumed was only a slight risk) in the next development cycle with this client. We’d been planning and doing preliminary design work for our next alpha version of the application. Part of this was prototype work with a new framework, Java Server Face (JSF)*, that we thought would be nice and helpful in developing the next version. We planned on, or hoped to, develop part of our new version during this cycle based on some of the client’s requirements (being a small dev shop, it’s nice when your internal dev plans match up with client changes because it’s always good getting paid for your work rather than having to develop out of pocket during an off cycle).

*We’re a Java dev house, and we primarily use JBoss and an Oracle database (although we’ve used other app servers and databases), and one of part of our application was using hibernate (and we’re expanding this into the rest of the application). However, our pages were mainly a mess of JSP code tied sloppily into HTML and javascript. All the years of development on this version of the application, with many developers of varying ability working on it, we’d turned the whole things into a spaghetti monster of code, which was a complete pain in the ass to maintain with little flexibility for improvement, and basically nearing the end of the line (well beyond the time for a totally clean rebuild in my opinion).

In the weeks prior to the London meeting, I had been writing technical specs for this new part of the application as well as preliminary UI designs. We’d planned on using a team of Indian workers to develop this part of the application who we’ve had working for us for a while now on two other applications for this client. They had been mirroring our prototype work with JSF to get up to speed on the technology, and we felt confident enough that they could handle the task. So, the two weeks following the London meeting consisted of me filling in gaps in the technical spec and explaining pieces of it to the India team while doing some enhancements to the UI design that had come out of the meeting.

At that time, another manager had been directly managing the India team, and I was working more through him than with the team on the daily basis. This was really his team (he’s an Indian American – not native American – who hired the team), and I had other projects to manage and a new hire to supervise, so this was the natural setup. Two weeks after the meeting, 6 weeks left in the development plans, and everything appeared well enough.

The first of two major problems came about five weeks into development. Based on the initial version of the development timeline, we plotted that we’d being done with development at the end of August, which was a four weeks away. Two weeks before the end of August, user testing would start on parts of the application that were being completed at that stage, so we could speed up User Acceptance Testing (UAT), which would start at the beginning of the next month. This gets the clients familiar with the changes up to that point, and because we’re doing iterative development with an incomplete requirements document, this is a required step that gives them a chance to make comments, request further alterations, or make clarifications on the initial requirements having seen a live version (we were constantly clarifying and redefining up till then, but this would be their first time seeing a live version that they could test).

With a week left before user testing begins, I took over the daily management of the India team because their normal manager was swamped with other projects, so I stepped in to take off some of the workload. Problem was, the “progress” the team had been making wasn’t what it appeared to be. The last two weeks, they’d shown us preliminary, incomplete versions of their part of the application. It looked OK, but it wasn’t really where I wanted it to be when I took over (they hadn’t really been following my UI design), and when I had a chance to see where they really were, I nearly shit my pants.

They basically had nothing, and it was smoke and mirrors in my opinion (and lots of promises that the code was farther along behind the scenes). Granted, they still had three more weeks, but I wasn’t sure they could finish even with the remaining three weeks. I know, I reviewed and QA’ed the code when I took them on. The original assumption that we could get this part of the application done using a new framework as part of our alpha version was dead wrong.

The technology was apparently too much still for them to deal with even after months of prototyping, and they also weren’t yet capable of taking on this kind of development work where they’re required to be more creative as the technical doc I wrote wasn’t an exact step-by-step plan of do-this-here and do-that-there. I’m disappointed that neither the team, nor their manager spoke up about the lack of progress or major challenges they were really having. There’s no reason we couldn’t have provide more support at any earlier point, and after having gone through coding hell month myself because of this, I’m extra annoyed at the situation.

Still, as much as I’d like to completely blame someone else, easy as that always is, I know part of this was my fault for not foreseeing this situation. There were signs that it might be too much, but their manager was eager to take on the project, and I and others were eager to have them take it on. The risk seemed acceptable, but risks always look a little less acceptable when shit hits the fan, which it didn’t complete do until the next problem popped up that same week.

At probably the worst time possible another major problem reared its head. This problem was that one of the three main persons on the client side (two are the actual business side clients, and one is the technical client side manager) had misinterpreted user testing to mean full UAT of a fully complete application. Here, we’re five weeks into development and a week away from initial user testing, and someone thinks the whole project is actually supposed to be complete for them to sign off on.

We think we’ve still got three weeks of development left at this point, so we’re obvious not ready. Sadly, this business side person heard this from us, and complained about slipping deadlines and such to her (very important, high up) boss, who’s never been included in anything from planning to requirements. Things bat around with important persons who know nothing beyond what this one mistaken person thinks, and we’re pretty much screwed.

Now, we have to scramble to explain why we are where we are, and what we can deliver by next week, and why that isn’t what they’re expecting. At this point, their butts clench up, and we’re at full speed trying to manage the recovery. To our joy, their technical manager, who’s the highest up of the three primary clients that we work with (he gives all final approvals and makes sure both them and us follow any required processes), agrees with our interpretation of the timeline. However, the mistaken business person seems to have convinced the other business person that we need to deliver by the end of the week or we’ve missed the deliverable deadline, and all of a sudden nearly everyone is expecting a finished product now.

Looking at the big picture, we’re 3/4th complete with the other major pieces of development nearly finished, if not exactly ready for UAT, but good enough. Still, the last piece of the puzzle is complete shit and no where near finished. We explain that this interpretation of the timeline was never the intention, but we can still deliver 3 of the 4 major parts at by the end of the week, and a large part of the 4th by Monday, and the remaining that by following weekend (at this point, we actually thought the 4th part was salvageable, and didn’t know how completely fucked it was).

We calmed them down some, but their butts were still tightly clenched, and from our point of view, they still wanted an accelerated timeline. Although, to some of them it was a week late (and so you know, if you work at one of those – all too many – places that’s late by definition when it’s a modern day tech project, we’re never late). We had some support from the client side that our position was correct, but not enough to return the situation to a state of normalcy. So, we ramp up and get everyone involved for speedy development planning to work through the weekend to satisfy the clients.

Come Thursday, I decide that we can’t use the India team for this project anymore, and I fired them (well, actually, I just removed them from the project because they’re still working for us on their normal projects – but I like thinking of it as firing them). I assemble a task force of three people, one of them being me, as our most experienced developers to finish the work needed. We started to crunch on Friday, and weren’t sure what we could do with the crap from the India team. It became painfully obvious that it would take more time working with their code than writing it all from scratch.

That weekend, starting Saturday morning, we huddled with the two head honchos of our company and all 5 of us agreed that this was the only option. However, even with completely new code, we still had to meet the clients expectations for Monday and the other remaining deliverables each following day as we had planned out for them in our recovery mode. In order to do this, we ditched the new framework and went with what we knew best. What came out at the end was actually quite an attractive piece of software (the users really enjoy it and find it much smoother), but the unseen code was a complete and utter hack. And, while we sorta technically delivered what they wanted by the end of the following week, we still really had three more weeks of hard long hours of work. More on that next time.

“In defiance of the rising sun” – Judicium Dei – Trial by Coding (Part 1)

The last grain of sand fell, and with it I too looked forward to resting, albeit in defiance of the rising sun. While I had welcomed the new rays of light spreading across the empty parking lot the day before, I now cringed at the sight as though I was hearing some painfully true accusation of wrongdoing from a good friend. And yet, thinking good progress had been made, I knew nothing of the trail by coding that I would experience even after that 24 hour warm up session.

That 24 hour session, sun rise to sun rise, was actually day 2 of the coding ordeal that would drag on for 4 weeks, but the story begins a couple months earlier. I had flown to London with the President of my company to gather business requirements for the 3rd and latest iteration of deploying and upgrading our software for our biggest client (a Fortune 100 company). Our flagship product, which they used along with a couple others we’d written for them, is customized compensation software covering the focal review process (Human Resources – budgets, raises, bonuses, stock options mixed with international currencies, taxes, and other headaches).

We were beginning our 3rd year working together, and while we’d had good experiences with them before, we were in an awkward position this time as they weren’t able to begin this process the month before as we would have preferred (we stay in contact year round, even though we only go through this initial process once a year). They were going through possible acquisition dealings with a few other large companies, and didn’t have the time to deal with this next iteration of updates because they didn’t really know what they wanted/needed thanks to their constantly changing environment. Additionally, because our software was uniquely the most encompassing of any software they had on certain financial aspects of their company, they were trying to use it in a way that it hadn’t been intended, which was as a primary reporting and strategic decision tool in this merger & acquisition process.

Now, we usually have a general idea when clients want to start their focal review process before we start talks. This obviously helps us determine when we should start the next round, so we can be sure that we start a reasonable amount of time ahead of that based on any changes they request (and there are almost always changes). The client’s official start date (our go live date for the new updates) isn’t usually determined until after we’ve already begun, and even then, it’s usually flexible within reason, but we have some room to play with given their needs. However, this time, with this client, not only could we not start when we would have preferred, but they also set the go live date before we started this process as well as set a number of near non-negotiable change requirements and product enhancements.

We were more than happy to work with them, and you’re usually in a great place if your clients are knocking at your door with a big problem that only you can solve, so we were a little anxious, but confidence we could do whatever was needed. The President, my London counterpart, and myself did our best to trim back their demands to their core, truly necessary requirements. Surprisingly, this didn’t include much change to the primary product, but mostly consisted of changes to peripheral applications that supported the main one, but are included as a part of the whole package.

This was because they were as much, if not more interested in the reporting and strategic decision making functions as they related to the on going M&A process as compared to their interest in improving the standard focal review process. But don’t be fooled, they still required a fundamental alteration to the primary application. So, one change instead of many, but a big one that they couldn’t do without as it tied together the whole package, and gave them some newly required flexibility along with the additional functionality.

From this week long meeting, we came out with a timeline for development and User Acceptable Testing (deploying the application for go live at the end of the UAT). Because of the many large changes and the short time we had to define all of them, we had to finish without completely understanding the requirements (both us and them had a lack of understanding). This is OK when everyone is on board (clients definitely included), and if everyone understands that you’ll be going through an iterative process that unfolds as you make progress, meaning that you redefine or update the requirements as things come up.

I don’t usually have a problem with this because at any scale of development (small or large), I believe that iterative development is the only right way to work. Projects, like life, change dramatically as you’re going through them (in reality the much acclaimed – when it was “invented” at least – Software Development Life Cycle (SDLC), aka the Waterfall Process, is actually crap, although good to know). Many times because you can’t really understand the situation until you’re in the middle of it with your hands dirty. It’s usually only after the 3rd or 4th time doing something that you can start generalizing, but even then, you’ll inevitably encounter lots of new, important, and unexpected challenges if you’re always pushing for better.

However, this iterative process and the communication and relationships required for success turned out to be a problem for us later on down the road. Not because of inexperience on either side as this is how we work and have worked with clients – this one included. They’ve experienced this kind of development process, and they prefer it having worked with us like this, which is probably half the reason they waited so long to have this business requirements meeting as they were more than comfortable with our ability to adapt and adjust to critical changes.

Making iterative development work, though, requires that both sides remain friendly and understanding throughout the whole process or it falls apart. Not that things became nasty, but myself and others felt a little betrayed at one point by the near reversal of attitudes on what had been agreed to, and how we had agreed to accomplish the large number of developing requirements (I’m sure I’ll bring this up in more detail in the following posts).

Before I get to that, though, everything looked good. Everyone was confident, although a little nervous perhaps, yet happy with the definitions and understanding we’d come out with. And really, there’s only one big mistake that could have been avoided at this point. That one big mistake, or risk we’d thought reasonable at the time, is what caused my trial by coding.

When best of the best is nothing special

I’m still working my way out of the beating at work. The process isn’t complete, but should be closing soon, yet challenges keep rearing up. Hope to make a post soon on this ordeal, probably a few posts for all that’s gone on. Until then, here’s that piece on Navy Seals and my friend who just joined their ranks.

For months, you have been put to the test. Grueling physical and mental strain has intentionally been unleashed on you, and you’re a willing participant through it all. You’re in a constant fight against your environment as it slams against your almost broken body in a constant wave assault. Your ultimate goal is making it another day, another meal, another minute, and their ultimate goal uncovering your limits.

At the end of this process, you’re at the top of your game. As you walk around campus, others make sure to divert their eyes and show complete respect for you, what you’re doing, and what you’ll become. You know you’re the best, and so does everyone else. Except, that’s not really true, and you know it and those you’ll be joining know it.

When becoming a Navy Seal, you must be a rock in both body and mind. Obviously, you can’t do what’s required of a Seal if you aren’t strong and fit. However, the real challenge is the mental challenge. I really admire the BUDs classmen even if they don’t make it through to the end. It seems like a constant fight against oneself and one’s own limits more than anything else.

When the only thing you have to do is say you’re done and ring a bell, there isn’t much keeping you from quitting when you’re already on the edge and possibly beyond what you thought the edge was. But to make that even easier, your instructors do everything they can to make quitting easy such as bringing that damn bell down to the shore while you get a beating from the waves, so you don’t even have to walk far to quit and ring it. Of course, this also gives you the perfect view of everyone else who’s finally pulling out, pushed to their limits once and for all.

You really have to be doing this for yourself. As much camaraderie as there is, and as much as you’re doing it for your country, or anything else, it’s really only those who do it for themselves that really have a chance. Not only do you have to go through a beating to get to the finish line, but once there you get little recognition to go along with the more daunting challenges to come. The Seal does not do what he does for recognition (it’s in their motto), and that’s definitely built into the training (most of their missions will be classified anyway, so get used to it).

Even to the last hurrah at graduation, they make sure you don’t forget to be the rock they made you into by welcoming you to mediocrity. Even as you yell to the next class of BUDs to hit the sand, you’re reminded where you too stand now that you’re joining a team of people who have already done what you’ve done, and they’ve even used their training in live action. You’re the freshmen again, for all you’ve gone through, and they make sure you remember that. Stories about 4 Seal in Iraq being surrounded in a Humvee killing over 200 insurgents before being overcome starts to make sense after understanding what’s produced by a program such this.

However, what’s really surprising is how jovial, friendly, and approachable these characters are. They’re programmed to be on the edge, and it blows my mind how they can come from training into the town of Coronado surrounded by rich, chic, older women in pink outfits and trendy guys with cool shades wearing Tommy Bahama, and not snap (or snap someone else’s head) from leaving such an intense environment and joining such a me-me, attention grabbing, easy street world. It was like a punch in the face to me, and I’d only been around for a couple days.

Yet, it makes sense in the same. How else could one go through such a rough training program, and not come out the other end all twisted and screwed up without having a great sense of humor and accepting the things that go on around you. As tightly wound, ready to act, as these guys are, they’re completely in control, and totally cool. Yeah, if they wanted to stare you down, you’d crap in your pants, but they’re just normal 20 something guys when they’re not doing their thing. Sure, they can shoot the eye out of a bird 200 yards away, and throw a knife across the room drunk and still hit a dime, but, they also want to appreciate life like the rest of us, and have fun with friends, kick back, play some Wii, and drink a few drinks.

We each have our own tests to overcome, our own limits to face. The ordeals faced by Seal trainees, and Seal graduates, is both an inspiration and an example for us all. “Be someone special,” and “The only easy day was yesterday” exemplify the Seal experience. It’s a good reminder that what we’re facing ain’t that bad, and if someone else can pull through so much more, then surely we can come out the better from our own challenges. Here’s to class 261, I know you’ll protect our country with the best of them, and make our enemies crap in their pants (which is the least they can hope for if they face a team of Seals).