(This is definitely a long post, but this part is where things blow up, so I’ve got a bit to say.)

Calm has returned at this point, now the last week of September. I can relax in my chair at work, enjoy the tasks at hand without any desire for the day to end, and I feel comfortable wasting time. There are no more issues popping up that have to be finished by the end of the day or the client can’t function when they start again their next day.

Last week was also an easy week by any measure, but I still felt anxious for the work day to end, which got me that much closer to the weekend. Even though I knew in my head that the calm had settled in for sure by this time, as there hadn’t been any pressing work for a couple weeks, and I’d shifted back over to focusing on our newest “alpha” version of the application, I still felt impatient with an eagerness for closure of that work day. Stress builds up and leaves a residue that takes time to clear away, and there really aren’t any quick fixes nor fun camps that help speed up the process.

The client is now quite happy and working away in their processes with the new and improved application much as they were happy leaving the business requirements meeting in London a few months back. At that time, we didn’t have a new and improved application, but we had some tentative plans and solid designs for what we’d end up with. My trial by coding was still roughly 6 weeks away, and all seemed good.

My company and I had decided before and during the meeting that we’d try taking a slight risk (or what we had assumed was only a slight risk) in the next development cycle with this client. We’d been planning and doing preliminary design work for our next alpha version of the application. Part of this was prototype work with a new framework, Java Server Face (JSF)*, that we thought would be nice and helpful in developing the next version. We planned on, or hoped to, develop part of our new version during this cycle based on some of the client’s requirements (being a small dev shop, it’s nice when your internal dev plans match up with client changes because it’s always good getting paid for your work rather than having to develop out of pocket during an off cycle).

*We’re a Java dev house, and we primarily use JBoss and an Oracle database (although we’ve used other app servers and databases), and one of part of our application was using hibernate (and we’re expanding this into the rest of the application). However, our pages were mainly a mess of JSP code tied sloppily into HTML and javascript. All the years of development on this version of the application, with many developers of varying ability working on it, we’d turned the whole things into a spaghetti monster of code, which was a complete pain in the ass to maintain with little flexibility for improvement, and basically nearing the end of the line (well beyond the time for a totally clean rebuild in my opinion).

In the weeks prior to the London meeting, I had been writing technical specs for this new part of the application as well as preliminary UI designs. We’d planned on using a team of Indian workers to develop this part of the application who we’ve had working for us for a while now on two other applications for this client. They had been mirroring our prototype work with JSF to get up to speed on the technology, and we felt confident enough that they could handle the task. So, the two weeks following the London meeting consisted of me filling in gaps in the technical spec and explaining pieces of it to the India team while doing some enhancements to the UI design that had come out of the meeting.

At that time, another manager had been directly managing the India team, and I was working more through him than with the team on the daily basis. This was really his team (he’s an Indian American – not native American – who hired the team), and I had other projects to manage and a new hire to supervise, so this was the natural setup. Two weeks after the meeting, 6 weeks left in the development plans, and everything appeared well enough.

The first of two major problems came about five weeks into development. Based on the initial version of the development timeline, we plotted that we’d being done with development at the end of August, which was a four weeks away. Two weeks before the end of August, user testing would start on parts of the application that were being completed at that stage, so we could speed up User Acceptance Testing (UAT), which would start at the beginning of the next month. This gets the clients familiar with the changes up to that point, and because we’re doing iterative development with an incomplete requirements document, this is a required step that gives them a chance to make comments, request further alterations, or make clarifications on the initial requirements having seen a live version (we were constantly clarifying and redefining up till then, but this would be their first time seeing a live version that they could test).

With a week left before user testing begins, I took over the daily management of the India team because their normal manager was swamped with other projects, so I stepped in to take off some of the workload. Problem was, the “progress” the team had been making wasn’t what it appeared to be. The last two weeks, they’d shown us preliminary, incomplete versions of their part of the application. It looked OK, but it wasn’t really where I wanted it to be when I took over (they hadn’t really been following my UI design), and when I had a chance to see where they really were, I nearly shit my pants.

They basically had nothing, and it was smoke and mirrors in my opinion (and lots of promises that the code was farther along behind the scenes). Granted, they still had three more weeks, but I wasn’t sure they could finish even with the remaining three weeks. I know, I reviewed and QA’ed the code when I took them on. The original assumption that we could get this part of the application done using a new framework as part of our alpha version was dead wrong.

The technology was apparently too much still for them to deal with even after months of prototyping, and they also weren’t yet capable of taking on this kind of development work where they’re required to be more creative as the technical doc I wrote wasn’t an exact step-by-step plan of do-this-here and do-that-there. I’m disappointed that neither the team, nor their manager spoke up about the lack of progress or major challenges they were really having. There’s no reason we couldn’t have provide more support at any earlier point, and after having gone through coding hell month myself because of this, I’m extra annoyed at the situation.

Still, as much as I’d like to completely blame someone else, easy as that always is, I know part of this was my fault for not foreseeing this situation. There were signs that it might be too much, but their manager was eager to take on the project, and I and others were eager to have them take it on. The risk seemed acceptable, but risks always look a little less acceptable when shit hits the fan, which it didn’t complete do until the next problem popped up that same week.

At probably the worst time possible another major problem reared its head. This problem was that one of the three main persons on the client side (two are the actual business side clients, and one is the technical client side manager) had misinterpreted user testing to mean full UAT of a fully complete application. Here, we’re five weeks into development and a week away from initial user testing, and someone thinks the whole project is actually supposed to be complete for them to sign off on.

We think we’ve still got three weeks of development left at this point, so we’re obvious not ready. Sadly, this business side person heard this from us, and complained about slipping deadlines and such to her (very important, high up) boss, who’s never been included in anything from planning to requirements. Things bat around with important persons who know nothing beyond what this one mistaken person thinks, and we’re pretty much screwed.

Now, we have to scramble to explain why we are where we are, and what we can deliver by next week, and why that isn’t what they’re expecting. At this point, their butts clench up, and we’re at full speed trying to manage the recovery. To our joy, their technical manager, who’s the highest up of the three primary clients that we work with (he gives all final approvals and makes sure both them and us follow any required processes), agrees with our interpretation of the timeline. However, the mistaken business person seems to have convinced the other business person that we need to deliver by the end of the week or we’ve missed the deliverable deadline, and all of a sudden nearly everyone is expecting a finished product now.

Looking at the big picture, we’re 3/4th complete with the other major pieces of development nearly finished, if not exactly ready for UAT, but good enough. Still, the last piece of the puzzle is complete shit and no where near finished. We explain that this interpretation of the timeline was never the intention, but we can still deliver 3 of the 4 major parts at by the end of the week, and a large part of the 4th by Monday, and the remaining that by following weekend (at this point, we actually thought the 4th part was salvageable, and didn’t know how completely fucked it was).

We calmed them down some, but their butts were still tightly clenched, and from our point of view, they still wanted an accelerated timeline. Although, to some of them it was a week late (and so you know, if you work at one of those – all too many – places that’s late by definition when it’s a modern day tech project, we’re never late). We had some support from the client side that our position was correct, but not enough to return the situation to a state of normalcy. So, we ramp up and get everyone involved for speedy development planning to work through the weekend to satisfy the clients.

Come Thursday, I decide that we can’t use the India team for this project anymore, and I fired them (well, actually, I just removed them from the project because they’re still working for us on their normal projects – but I like thinking of it as firing them). I assemble a task force of three people, one of them being me, as our most experienced developers to finish the work needed. We started to crunch on Friday, and weren’t sure what we could do with the crap from the India team. It became painfully obvious that it would take more time working with their code than writing it all from scratch.

That weekend, starting Saturday morning, we huddled with the two head honchos of our company and all 5 of us agreed that this was the only option. However, even with completely new code, we still had to meet the clients expectations for Monday and the other remaining deliverables each following day as we had planned out for them in our recovery mode. In order to do this, we ditched the new framework and went with what we knew best. What came out at the end was actually quite an attractive piece of software (the users really enjoy it and find it much smoother), but the unseen code was a complete and utter hack. And, while we sorta technically delivered what they wanted by the end of the following week, we still really had three more weeks of hard long hours of work. More on that next time.