The big divide at work (for me at least) is between the hacker coding style of development and object oriented engineering. I’m in the OO camp, and my two superiors are in the hacker style camp. I’ve been working on the hard sell of trying to change how we develop as a team, and frustratingly enough, I don’t seem to be making much progress. Good thing I’m stubborn.

What I call hacker style is a method of development where getting things to work and out the door is the primary goal. Time spent thinking and planning development is limited to small scale, low level, single function planning. Nothing is really orchestrated, and the “system” is more of a patchwork of processes that when run in the right order, perform the overall tasks as though it were actually a whole. The focus is on speed, and being able to do new things.

Now, I don’t have any problem with that in general as there’s a trade off when moving to a more thoughtful, and deliberate OO style of development, which can have a real positive or negative impact on business. And for startup companies, hacker style is the preferred and probably optimal style of development. Business demands require speed and getting things out the door, so you can finally have a proof of concept that generates real revenue. Spending extra time to choreograph the whole system is money lost that’s hard to replace when you’re already on a shoe string budget.

Admittedly, I work for a startup, but it’s a startup with real revenue (million of dollars of revenue a year for multiple years with continued growth), and I’d says it’s in a transition between the “we’re still a startup” and the “we’ve got revenue streams to worry about” stage. The engineering team to date has done a great job doing what needed to be done. That’s something to be proud of, but now the code base is old and messy.

If there’s anything to be said about my last job, it’s that I’m experience in old and messy code (I’d include my time at Citigroup too, but the code was just old, not as much messy). And I’m definitely experienced in leading a development team to a more appropriate style of development. I personally prefer a more deliberate OO style of development, but my last job needed more hackers. Still, OO can be implemented with great successful if you have the right experience in earlier stages of the game. With this my new company, which is a little bigger, they’re about at the end of the hacker stage as I see it, but they’re holding on tight.

With a team of good developers, how do you transition them from being hackers to engineers, and what is the benefit of an OO approach to engineering as well as the business? As your code base grows, its complexity grows. Maintenance, scalability, adaptability, once corner stones of the hacker style, now become elusive and out of reach. Even with well build systems, things get old and messy. From my experience, most systems only last 4-5 years before needing a good overhaul, or the business risks losing it’s ability to change and respond to new situations. Developer time gets lost to fixing bugs and time gets eaten up by the many workarounds that inevitably happen when implementing additional functions and features.

The ultimate goal of OO design is building an intelligent code base. Intelligence is not a measure of behavior like most people think. Intelligent is an understanding that allows one to predict future events. A code base can “predict” future events by understanding what is likely to change in the future, and having that part designed in a way that makes changing it easy. You don’t make everything easy to change, but that which is likely to change. You need to manage the complexity of your code or it’ll blow up in your face.

People who aren’t familiar with OO design, or aren’t experienced with good OO design (it’s easy to screw up if you don’t know what you’re doing), they may see OO as more complex and slower to implement. When transitioning between development styles, in the short term that may well be the case, which is why startups generally don’t do OO design unless they’ve got some special previous knowledge in the matter. But as complexity grows, patchwork systems lead to more bugs and a more complex interaction that will drag down a development team, and in the extreme, can cripple a business’ ability to react.

One of the main pieces missing when doing hacker style development is a big picture view of the system and the various interactions. As the system was developed in pieces, the patchwork of functions were developed with limited consideration of other existing functions, and usually no consideration for upcoming functions. Without a big picture view, you can’t pull yourself out of that patchwork. You need a guide to get yourself to a better position.

There’s an additional drawback to the patchwork (utility class) design. Because it focuses more around processes and less around object relationships, it’s impossible to discuss the parts of the system. Engineers will be forced to learn from the code instead of other engineers. Additionally, as the code base is process focused, you’re code will give as much if not more weight to dealing with edge cases. This is part of the reason why engineers have a tough time discussing such existing code because you can’t generalize and talk about the overall interactions, but instead are caught up mulling over special cases.

The benefits of OO engineering range from greater ability to react as system complexity grows to better team communication as you move away from process acted upon multiple complex objects to relationships between a common set of objects. Getting a team to handle this change properly requires a map, but also vision and a mission.

Businesses have two functions. Marketing (PR, advertising, sales), and invention (engineering). The marketing side of the business will most certainly have a set of ideals such as VMOS (there are many varieties of this), vision (the dream), mission (what you’re trying to do in the next couple years), objectives (focus or important metrics for accomplishing the mission), strategy (lines of action for meeting your objects). The highest level long view down to actionable strategies. Engineering needs a similar plan that describes how they’ll implement the business’ VMOS. The engineering VMOS will reflect the business’ VMOS, but translated for engineers and around the act of inventing instead of marketing.

Somewhere in the objectives and strategy level, you can setup a diagram of your system. Not how it is today, but how it should be tomorrow. This’ll be constantly changing, but it gives engineering a way to communication, and a guide for how to develop. Where do you spend the extra time planning? What is really an at risk area? What functions are important to keep on the radar?

OO engineering can easily break down without a plan, but also without a leader. Just as the graphic design team will have a head designer who will pull together the design and keep it looking like one common design, engineering needs a head engineer to pull together overall development as though it was actually done by one person. Since any problem can be solved in 20 different ways, you need someone to short cut the discussions, and make a decision even if it doesn’t look like the best choice to everyone. Just because you’re given more room to plan and think deliberately, doesn’t mean you can waste valuable time not getting code out the door (I’m not advocating a waterfall approach mind you).

This gives the engineers the tools to both communicate and predict. With the high level ideals defined, you can develop a plan for transitioning. How do you meet business demands and still maintain and improve the code base? It’s laid out in the engineering VMOS. Without this, all you can do is react to the nearest term demands and situations, which becomes tougher and tougher the more patchwork and complex the code. The engineers are the protectors of the code, and they must always be mindful of that while they get pressure for more from the business, so you can’t let the marketing side make all the plans and call all of the shots.

Wrapping this up, both styles work with the variety of agile development approaches. Both styles have value for a business at different stages in it’s life. Both can be good or bad if you go too far to an extreme. Balance is key as is so much in life. In the case with my current job, moving to OO engineering removes much current and future complexity.

Advertisements

I’m still here, even though I haven’t posted in a while. I’ve been working at a new company for the last few months that’s a small startup in the casual game publishing and development market (they make and sell video games online). I’m part of the website team that helps build the community and product publishing features (and any other feature or function of the website). It’s been a lot of fun so far, and it’s nice working for such a young company that’s also very driven and aligned well with its business needs.

I’ve been making a little controversy with some of my ideas as I’m still trying to build trust and gain acceptance of my positions on the team. That means I’m full of ideas and stories that I could write about, so we’ll see if I find the time and motivation to put some of it down on my blog. Until then, life is pretty good even after quitting my previous job during a market crash and now into the ensuing economic recession ahead of us.

I created a demo video of the application I helped build for my last company, which I no longer work for. They have a live version that can be accessed online, but since it’s a sophisticated tool that requires some prior knowledge, it’s easier for me to give people a general overview of the look and feel by using this kind of video.

I don’t show all of the application, just a couple main pages. At this time, some pages are old versions that aren’t of the latest internal version from when I left, and some pages are updated with new stuff that I had nothing to do with, but the video shows pages that haven’t really changed since I left. Since I can’t vouch for the quality or state of the application now that I’m gone, this video gives some general sense of the product.

Quality is pretty crappy being a Youtube video. I have a .mov file, but I can’t upload that here, so if anyone is interested in a higher quality, then let me know (~30MB file).

Overall, I’m happy with the product. I feel like I did a hell of a lot of work on it doing everything from UI Design, client-side coding to server-side and business logic coding to some database work as well. There’s plenty of points that could be improved, things I’d do differently if I did them again now, but such is the way with small companies and tight deadlines. The demo had been tested against IE5.5-7, FF2 (Mac and Windows), and Safari (Mac). I believe Opera worked fine as well. FF3 has a couple minor visual glitches, but probably not noticeable by someone not familiar with the product. FF3 came out after the initial design, and client work (and my boss’ priorities) kept me from fixing the issues before I left (UI Design was the least important thing to pretty much everyone, but me). C’est la vie.

I recently posted some thoughts about P&L and my own version of P&L, productivity and learning. I’ve been asked to shed some more light on the subject of productivity, so I’m delving into the topics of capability and capacity. In my previous piece, I argue that productivity is made up of a few components, and capability and capacity are two of the tangible elements at play (other intangible elements would include level of challenge of assignments and happiness with work). I also note quality as a key component in the piece, but more as a component of learning, and less as a direct component, although still influential, with productivity.

Estimating capability and capacity judges readiness and ability to adapt. This allows you to highlight potential problem areas. The method won’t give you a number. It defines the boundaries of your team.

With this understanding, you can then see how your team’s characteristics match your overall functional priorities, and from that point, you can develop plans for getting or keeping your team aligned with your overall goals. In terms of measuring results, you want to measure based on primary goals, and not based on lower level components such as capability or capacity, or you greatly risk managing by misdirection (for instance, increasing productivity may have in impact on revenue, but could instruct people to push out crap instead of better satisfying the real goal). If you do develop a means of measuring sub-components of your team (for the purpose of reviews, and ultimately, effecting compensation), then it should always be understood in relation to the overall impact on the overall goals.

Understanding Capability

Capability is another way of saying, what does your team know? In understanding your team, you’ll want to map out the Domain of Knowledge that your interested in. You first break it down into a set of business or functional categories. Such as software engineers obviously need to know about writing code, but also potentially about the subject matter of the business itself like advertising, or farming, or insurance. You may think of 3 or 4 different areas of interest even if the primary area, engineering, is the main area of concern as understanding finance or marketing or design may play a smaller, but still consequential role in the team makeup.

Once you have broad categories listed, you then want to break each area down into 3 or 4 levels of understanding like beginner, intermediate, and expert. Software engineers under the engineering category would look something like: (Beginner) basic coding skill, maybe some specifics like JSP, HTML, CSS, Javascript, (Intermediate) Object oriented programming, data structures, security, database, (Expert) design patterns, high performance applications, specific app server experience like with JBoss, specific experience with a particular tool of interest. You can get more or less specific, but do the same for each of the other areas.

Then see how your team stacks up against that breakdown showing you obvious holes. Both specific expertise and broad knowledge individuals are useful. Typically people with a broader knowledge base will be less of an expert in any one area, but could be just what you need. You have to understand the daily and weekly demands, and make sure that you team can meet those needs, but also the growing needs of an evolving business. People who are experts in a small area provide a lot of value immediately, but lose a lot of value if you go away from that area giving you potential incentive to not chance when you might need to.

Nothing mind blowing there, but if you don’t take the time to think about what you need and what you have, and how that aligns with your overall goals, it’s impossible to fix a problem when it arrives. And, if you’re lucky, you might be able to cut off some problem areas before they become a problem, or at least plan for a known gap.

Understanding Capacity

Capacity can be variable based on your own ability, not experienced or skilled enough, but also extremely open to external pressure such as being set up with inappropriately short deadlines or simply too much work no matter how good you are. For a normal week, though, no work should require overtime*. If the normal operating procedure includes overtime like normally working 12 hours a day or working 6 days a week, then there’s probably something wrong with the system, and it should be investigated, and plans should be developed to fix that problem. Otherwise, the cost of abnormal pressure will become too much, and you’re out of room to be agile or adapt to changing situations, nor will you likely have time to innovate.

*I understand that some people will be exempt salaried and therefore not receive overtime, but I’m using overtime to mean anyone working more than 8 hours a day regardless of whether they explicitly get paid for it or not.

Grade A

You’re able to handle a week’s worth of work, 40 hours, in less than a week’s time, something like 30 hours or less. You have time to optimize what you do, and research better ways of doing what you do, which would bring you up to 40 hours of work a week. If there’s a sudden increase in critical, time restricted work, then you probably won’t have to do any overtime to absorb the increase. This should be the standard for any given work week.

Grade B

You’re able to handle a week’s worth of work, 40 hours, in a week’s time, 40 hours or less time. If you have a sudden increase in critical, time restricted work, you can still make a weekly deliverable with some minimal overtime, less than a total of 48 hours. Essentially, you can handle the work you’re given, and don’t have to push yourself to get it done in time. You can stay ahead of the curve.

Grade C

You have to push to keep up, or barely keeping up with current workloads. Getting head of the curve is a problem. Probably have to work overtime more regularly to keep up with everything. You’re probably in over your head either because there’s just too much to do, or you’re not experienced/skilled enough to do the required work in the required time. Anything over 48 hours instantly puts you in this spot if not verging on an F.

Grade F

There is no D. You’re either barely keeping up, or you’re behind and failing. This doesn’t mean you’ve missed deadlines, but it means you’re probably in risk of missing deadlines, if you have already.

Productivity

Bringing these two components together, you can get a sense of how productive your team can be. Improving productivity would then generally be matter of realigning your team’s knowledge around the domain of knowledge, or fixing capacity problems that are causing even your best employees to have productivity pains.

I don’t like thinking of number of widgets produced as productivity, but instead as a combination of factors including capability and capacity. You can increase the number of widget your team produces without really improving the status of your team. And as a member of the team yourself, there are some good selfish reasons to keep your team running smoothly.

By mapping out the domain of knowledge, and having a standard metric for measuring capacity, you can approach someone with the power to fix the problem, if that not be yourself. Increasing hours is always a very short term solution that has major long term impacts on the intangible attributes of productivity if not kept to the short term. Rapid increases in productivity general come in the form of re-engineered processes, and not re-engineered people. People take time to improve either through on the job experience, or in the class education. If you’re not fundamentally addressing capability or capacity, then you’re not really dealing with productivity.

I’ve been doing a lot of re-education and new education since quitting my job. I’m into continual education, so I’m always learning even when on the job, but that part of me got a little squelched at the end before I quit, and there’s nothing like unbroken education versus on the job or interrupted by the job education. So beyond the general stuff like rehearsing what I know (or knew at one time), and investigating things I wished I’d have more time for, I usually stumble on to new concepts, and a Hierarchical Temporal Memory (HTM) network is a new and ultimately fascinating one to me.

HTM is a method for developing intelligent machines based on how our own brain works, or at least how our neocortex works. It’s a brilliant idea thought up by Jeff Hawkins, creator of Palm and Handspring. I was watching a presentation he gave at TED covering this very briefly, and since that time I’ve delved head first into this whole thing. As he puts it, there’s nothing ultimately new about this concept as it takes a lot of piece from other similar AI techniques, but he feels that his direction, which is ultimately guided by the biology of the human brain, is the best direction to go (both to better understand our brain by developing and testing theories, but also as a means for driving technical innovations).

His belief is that he’s got the right pieces put together. Just as when he took a bunch of parts off the shelf to create the first Palm (which was as much about what he didn’t take as what he put into it, which makes me think of the iPod beating the Zune), he’s doing the same with his theory of the brain and HTM for intelligent computers. And to me, he’s got it right (admittedly I know next to nothing).

There are comparisons to other types of networks like HTM such as the more common Bayesian network. I remember first hearing about Bayesian networks, and doing some research, I didn’t exactly get it. Sure it sounded powerful, but it didn’t strike me. Unlike that and other similar networks, though, I get HTMs at first sight. It makes sense, and I can see the potential that this kind of computing can have both in implementation, accuracy, scalability, and so on. Or at least, I have a concept as I’m still learning this stuff, but my gut says something right here where in other cases it said something was off.

HTMs are at the first level about pattern recognition. You get accuracy in pattern recognition through training, which for HTM, it takes a slightly different approach from other networks as it assumes the network learns all causes from a null starting point using data that changes both spatially and temporally much like holding an apple in your hand and moving it around. This is where the network learns the various sequences and causes of the world it’s trying to understand through a learning stage. Additionally, while the No Free Lunch principle still stands for any learning algorithm, HTM has an advantage for solving common real world problems because it assumes the world is constructed through a hierarchy of patterns (which I believe is true as described by Jeff), and hence a hierarchy learning system best matches and most efficiently solves the world it’s trying to understand.

That’s just the first stage, though, as the power of the tool comes in it ability to predict. Well, the power is also in its means of learning as there’s an added level of efficiency over other networks which make this method theoretically more scalable than any other of its type. As I was convinced, Jeff argues the intelligence isn’t about behavior, which is what AI seems to be concerned with, but with an ability to predict the world around oneself. Through that ability to recognize causes in a world and predict outcomes, you can intelligently react to them where a behavior can be a pre-programmed response as computers are made today, or behaviors can be reactionary and truly intelligent through an understanding of the relationships in our world like humans do today.

Now, HTM networks won’t be making human like robots anytime soon. Maybe in the future, but the real interesting thought is making machines that can evaluate causes in the world like we do, but with enhanced sense that we don’t readily have available to us. Machines that can solve problems we have a hard time solving that is. The first commercially useful HTM networks will be deployed for solving smaller problems in our world. Problems like collision prediction in vehicles, or ISP network management problems, or CAD optimization for manufacturers.

I’ve been playing with the test setup of the tools offered by Numenta, and the various test HTM networks supplied, and I’m looking forward to creating my own HTM network. It is cool doing image recognition on a laptop, though, without much effort from what they give you to start with, so we’ll see how hard this actually is to use practically in my own means.