Gary Allison's Leadership Blog

Have you heard? Blogging is dead. It officially died November 30, 2022. RIP.

Not that it wasn’t on its last leg before then, but its officially now dead. If you missed the obit, feel free to adopt this as the official notice.

The consumption of video and the reduction of the attention span of the average person to the length of a tweet had already been a harbinger of the demise of blogs. But ChatGPT has officially killed it.

Why would anyone continue to blog when everything you write will be aggregated into a neural network to be regurgitated later without attribution in a chatbot?

That’s a shame, but as I’ve been putting all my energy into my book, its fine.

I’ll leave up most of my posts from the past, because, well, they’re most likely already in the neural network of every AI model anyway.

It has been a minute since I’ve devoted the time this blog deserves. Does anyone even read blogs in the TikBookTwitGram world? Does anyone have an attention span longer than 30 seconds? At the end of the day, perhaps it doesn’t matter. Perhaps this is more cathartic than having any illusion of being widely read.

For the last 2 years, I’ve been threatening to write a book. The introduction and first chapter are even mostly complete. I’m not sure exactly why I haven’t made more progress on this, but by starting to share more here, hopefully inspiration for dedicating the time to the book will come.

Motivation for the book comes from seeing so many of the challenges repeated over and over again in building engineering excellence. In my new advisory role seeing even more companies, the truth of common challenges is even more enforced. The book is about much more than that, but here lets focus on a few of the big ones.

In the next few series of posts, I’ll be tackling what I call the impedance mismatch between Engineering teams and the rest of the company. While Sales, Finance, Marketing and even HR all run based on quarterly commitments, measurements, and goals, most often Engineering teams using Agile processes find themselves at odds with other organizations. When those other departments have to commit to quota numbers, marketing qualified leads, accounts receivable collections, etc, for the current quarter and against an annual plan, they don’t understand why Engineering can’t commit and deliver to a quarterly roadmap.

Engineers at an individual level who buy into Agile on the contrary can find themselves at odds with a roadmap planning process – “We’re Agile” can be seen by those outside of Engineering as an excuse to not make commitments. 

The truth is we have learned over the last 50 years of software development that waterfall processes fail again and again. We have learned that it is impossible to know everything up front in a world of rapid change. We have also learned that in technology, speed wins every time. Early to market innovators, those that serve their customers quickly, teams that are responsive to market changes all win.

So, the art in solving this impedance mismatch is to keep the power of Agile, but also be able to stay in step with all the other teams in the company and fit into their quarterly processes. What are some proven ways to do this? We will dive into these in the next few blog posts. 

The short answer is: it requires Product and Engineering to work together seamlessly as one team. They can be in fact different organizations, but they have to have complete alignment and common goals / visions of success.

I’ve built these processes successfully in many companies over my career but have only been able to build the seamless relationship with Product a few times. Often the leaders of these organizations have other agendas than truly partnering with Engineering to build “proud but achievable” roadmaps. Too often, they have dysfunctional behaviors rooted in self promotion or preservation to put the energy into building a sustainable process for product planning.

If you have a great product team or at least a product team that wants to be great, I feel confident you can use these ideas to build a planning process that will build trust with the rest of the company. That trust has to have its root in trust built between Product and Engineering teams. Without that trust foundation, roadmaps will be a continual struggle and frequently disappoint.

Experience with the AWS Solutions Architect Associate exam:

Experience with the AWS Certified Developer Associate exam

Know your limits for both Certification Exams

{This is the final post in a 3 part series intended to tell the story of how we have been able to achieve an epic rearchitecture of our core platform. Special thanks to all of those who have helped in review and editing the original white paper.}

Rearchitecting the Team

In addition to rearchitecting the service to scale, we also had to rearchitect our team. As we set out on this journey to rebuild our solution into a scalable, cloud based service oriented architecture, we had to reconsider the very way our teams are put together.  We reimagined our team structure to include all the ingredients the team needs to go fast.  This meant a big investment in devops – engineers that focus on new architectures, deployment, monitoring, scalability, and performance in the cloud.  

A critical part of this was a cultural transformation where the service is completely owned by the team, from understanding the requirements, to code, to automated test, to deployment, to 24×7 operation.  This means building out a complete monitoring and alerting infrastructure and that the on-call duty rotated through all members of the team.  The result is the team becomes 100% aligned around the success of the service and there is no wall to throw anything over – the commitment and ownership stays with the team.

For this team architecture to succeed, the critical element is to ensure the team has all the skills and team players needed to succeed. This means platform services to support the team, strong product and program managers, talented QA automation engineers that can build on a common automation platform, gifted technical writers, and of course highly talented developers. These teams are built to learn fast, build fast, and deploy fast, completely independent of other teams.

Supporting the service-oriented teams, a key element is our Platform Infrastructure team we created to provide a common set of cloud services to support all our teams. Platform Infrastructure is responsible for the virtual private cloud (VPC) supporting the new services running in amazon web services. This team handles the overall concerns of security, network, service discovery, and other common services within the VPC. They also set up a set of best practices, such as ensuring all cloud instances are tagged with the name of the team that started them.

To ensure the best practices are followed, the platform infrastructure team created Beavers (a play on words describing a engineer at Bazaarvoice, a BVer). An idea borrowed from Netflix’s chaos monkeys, these are automated processes that run and examine our cloud environment in real time to ensure the best practices are followed. For example, the Conformity Beaver runs regularly and checks to make sure all instances and buckets are tagged with team names. If it finds one that is not, it infers the owner and emails team aliases of the problem.  If not corrected, Conformity Beaver can terminate the instance.  This is just one example of the many Beavers we have created to help maintain consistency in a world where we have turned teams lose to move as quickly as possible.

An additional key common capability created by the Platform Infrastructure team is our Badger monitoring services. Badger enables teams to easily plug in a common healthcheck monitoring capability and can automatically discover nodes as they are started in the cloud. This service enables teams to easily implement these healthcheck that is captured in a common place and escalated through a notification system in the event of a service degradation. 

The Proof is in the Pudding

The Black Friday and Holiday shopping season of 2015 was one of the smoothest ever in the history of Bazaarvoice while serving record traffic. From Black Friday to Cyber Monday, we saw over 300 million visitors.  At peak on Black Friday, we were seeing over 97,000 requests per second as we served up over 2.6 billion review impressions, a 20% increase over the year before. Â There have been years of hard work and innovation that preceded this success and it is a testimony to what our new architecture is capable of delivering.

Keys to success

A few ingredients we’ve found to be important to successfully pull off a large scale rearchitecture such as described here:

• Brilliant people. There is no replacement for brilliant engineers who are fearless in adopting new technologies and tackling what some will say can’t be done.
• Strong leaders – and the right leaders at the right time. Often the leaders that sell the vision and get an undertaking like this going will need to be supplemented with those that can finish strong.
• Perseverance and Determination – building a new platform using new technologies is going to be a much bigger challenge than you can estimate, requiring new skills, new approaches, and lots of mistakes. You must be completely determined and focused on the end game.
• Tie back to business benefit – keep business informed of the benefits and ensuring that those benefits can be delivered continuously rather than a big bang. It will be a large investment and it is important that the business see some level of return as quickly as possible.
• Make space for innovation – create room for engineers to learn and grow. We support this through organizing hackathons and time for growth projects that benefit the individual, team, and company.

Reachitecture is a Journey

One piece of advice: don’t be too critical of yourself along the way; celebrate each step of the reachitecture journey. As software engineers, we are driven to see things “complete”, wrapped up nice and neat, finished with a pretty bow. When replacing an existing system of significant complexity, this ideal is a trap because in reality you will never be complete. It has taken us over 3 years of hard work to reach this point, and there are more things we are in the process of moving to newer architectures. Once we complete the things in front of us now, there will be more steps to take since we live in an ever evolving landscape. It is important to remember that we can never truly be complete as there will be new technologies, new architectures that deliver more capabilities to your customers, faster, and at a lower cost. Its a journey.

Perhaps that is the reason many companies can never seem to get started. They understandably want to know “When will it be done?” “What is it going to cost?”, and the unpopular answers are of course, never and more than you could imagine.  The solution to this puzzle is to identify and articulate the business value to be delivered as a step in the larger design of a software platform transformation. Trouble is of course, you may only realistically be able to design the first few steps of your platform rearchitecture, leaving a lot of technical uncertainty ahead. Get comfortable with it and embrace it as a journey. Engineer solid solutions in a service oriented way with clear interfaces and your customers will be happy never knowing they were switched to the next generation of your service.

{This is the second in a 3 part series of posts intended to tell an impressive story of how we have been able to achieve an epic rearchitecture of our core platform. Special thanks to all of those who have helped in review and editing the original white paper.}

The Journey Begins

One of the first things we decided to tackle was to start moving analytics and reporting off the existing platform so that we could deliver new insights to our clients showing how reviews are used by shoppers in their purchase decisions.  This choice also enabled us to decouple the architecture and spin up parallel teams to speed delivery. To deliver these capabilities, we adopted big data architectures based on Hadoop and HBase to be able to assimilate hundreds of millions of web visits into analytics that would paint the full shopper journey picture for our clients.  By running map reduce over the large set of review traffic and purchase data, we are able to give our clients insight into these shopper behaviors and help our clients better understand the return on investment they receive from consumer generated content.  As we built out this big data architecture, we also saw the opportunity to offload reporting from the review display engine.  Now, all our new reporting and insight efforts are built off this data and we are actively working to move existing reporting functionality to this big data architecture.

On the front end, flexibility and mobile was a huge driver in our rearchitecture.  Our original template-driven, server-side rendering can provide flexibility, but that ultimate flexibility is only required in a small number of use cases.  For the vast majority, a client-side rendering via javascript with behavior that can be configured through a simple UI would yield a better mobile-enabled shopping experience that’s easier for clients to control.  We made the call early on not to try to force migration of clients from one front end technology to another.  For one thing, it’s not practical for a first version of a product to be 100% feature function capable to the predecessor.  For another, there was just simply no reason to make clients choose.  Instead, as clients redesigned their sites and as new clients were onboard, they opt’ed in to the new front end technology.

We attracted some of the top javascript talent in the country to this ambitious undertaking. There are some very interesting details of the architecture we built that have been described on our developer blog and that are available as open source projects on in our bazaarvoice github organization. Look for the post describing our Scoutfile architecture in March of 2015. The BV team is committed to giving back to the Open Source community and we hope this innovation helps you in your rearchitecure journey.

On the backend, we took inspiration from both Google and Netflix. It was clear that we needed to build an elastic, scalable, reliable, cloud-based data store and query layer. We needed to reorganize our engineering team into autonomous service oriented teams that could move faster. We needed to hire and build new skills in new technologies.  We needed to be able to roll this out as transparently as possible to our clients while serving live shopping traffic so no one knows its happening at all.  Needless to say, we had our work cut out for us.

For the foundation of our new architecture, we chose Cassandra, an Open Source NoSQL data solution based on influence of ideas from Google and their BigTable architecture.  Cassandra had been battle hardened at Netflix and was a great solution for a cloud resilient, reliable storage engine. On this foundation we built a service we call Emo, originally intended for sentiment analysis.  As we made progress towards delivery, we began to understand the full potential of Cassandra and its NoSQL based architecture as our primary display storage.

With Emo, we have solved the potential data consistency issues of Cassandra and guarantee ACID database operations. We can also seamlessly replicate and coordinate a consistent view of all the rating and review data across AWS availability zones worldwide, providing a scalable and resilient way to serve billions of shoppers.  We can also be selective in the data that replicates for example from the European Union (EU) so that we can provide assurances of privacy for EU based clients. In addition to this consistency capability, Emo provides a databus that allows any Bazaarvoice service to listen for the kinds of changes the service particularly needs, perfect for a new service oriented architecture. For example, a service can listen for the event of a review passing moderation which would mean that it should now be visible to shoppers.

While Emo/Cassandra gave us many advantages, its NoSQL query capability is limited to what Cassandra’s key-value paradigm. We learned from our experience with Solr that having a flexible, scalable query layer on top of the master datastore resulted in significant performance advantages for calculating on-demand results of what to display during a shopper visit. This query layer naturally had to provide the distributed advantages to match Emo/Cassandra. We chose ElasticSearch for our architecture and implemented a flexible rules engine we call Polloi to abstract the indexing and aggregation complexities away from engineers on teams that would use this service.  Polloi hooks up to the Emo databus and provides near real time visibility to changes flowing into Cassandra.

The rest of the monolithic code base was reimplemented into services as part of our service oriented architecture. Since your code is a direct reflection of the team, as we took on this challenge we formed autonomous teams that owned everything full cycle from initial conception to operation in production. We built the teams with all the skills needed for success: product owners, developers, QA engineers, UX designers (for front end), DevOps engineers, and tech writers. We built services that managed the product catalog, UI Configuration, syndication edges, content moderation, review feeds, and many more.  We have many of these rearchitected services now in production and serving live traffic. Some examples include services that perform the real time calculation of what Brands are syndicating consumer generated content to which Retailers, services that process client product catalog feeds for 100s of millions of products, new API services, and much more.

To make all of the above more interesting, we also created this service-oriented architecture to leverage the full power of Amazon’s AWS cloud. It was clear we had the uncommon opportunity to build the platform from the ground up to run in the cloud with monitoring, elastic resiliency, and security capabilities that were unavailable in previous data center environments.  With AWS, we can take advantage of new hardware platforms with a push of a button, create multi datacenter failover capabilities, and use new capabilities like elastic MapReduce to deliver big data analytics to our clients.  We build auto-scaling groups that allow our services to automatically add compute capacity as client traffic demands grow. We can do all of this with a highly skilled team that focuses on delivering customer value instead of hardware procurement, configuration, deployment, and maintenance.   

So now after two plus years of hard work, we have a modern, scalable service-oriented solution that can mirror exactly the original monolithic service. But more importantly, we have a production hardened new platform that we will scale horizontally for the next 10 years of growth.  We can now deliver new services much more quickly leveraging the platform investment that we have made and deliver customer value at scale faster than ever before.

So how did we actually move 300 million shoppers without them even knowing?

Divide and Conquer

As Engineers, we often like nice clean solutions that don’t carry along what we like to call technical debt.  Technical debt literally is stuff that we have to go back to fix/rewrite later or that requires significant ongoing maintenance effort.  In a perfect world, we fire up the the new platform and move all the traffic over.  If you find that perfect world, please send an Uber for me. Add to this the scale of traffic we serve at Bazaarvoice, and it’s obvious it would take time to harden the new system.

The secret to how we pulled this off lies in the architecture choices to break apart the challenge into two parts: frontend and backend.  While we reengineered the front-end into the the new javascript solution, there were still thousands of customers using the template-based front end.  So, we took the original server side rendering code and turned it into a service talking to our new Polloi service.  This enabled us to handle request from client sites exactly like the Classic original system.

Also, we created a service improved upon the original API but was compatible from a specific version forward.  We chose to not try to be compatible for all version for all time, as all APIs go through evolution and deprecation.  We naturally chose the version that was compatible with the new Javascript front end.  With these choices made, we could independently decide when and how to move clients to the new backend architecture irrespective of the front-end service they were using.

A simplified view of this architecture looks like this:

Simplified view of New Architecture

With the above in place, we can switch a Javascript client to use the new version of the API through just changing the endpoint of the API key.  For a template-based client, we can change the endpoint to the new referring service through a configuration in our CDN Akamai.

Testing for compatibility is a lot of work, though not particularly difficult. API compatibility is pretty straight forward, which testing whether a template page renders correctly is a little more involved especially since those pages can be highly customized.  We found the most effective way to accomplish the later since it was a one time event was with manual inspection to be sure that the pages rendered exactly the same on our QA clusters as they did in the production classic system.

Success we found early on was based on moving cohorts of customers together to the new system. At first we would move a few at a time, making absolutely sure the pages rendered correctly, monitoring system performance, and looking for any anomalies.  If we saw a problem, we could move them back quickly through reversing the change in Akamai. At first much of this was also manual, so in parallel, we had to build up tooling to handle the switching of customers, which even included working with Akamai to enhance their API so we could automate changes in the CDN.

From moving a few clients at a time, we progressed to moving over 10s of clients at a time. Through a tremendous engineering effort, in parallel we improved the scalability of our ElasticSearch clusters and other systems which allowed us to move 100s of clients at a time, then 500 clients at time. As of this writing, we’ve moved over 5,000 sites and 100% of our display traffic is now being served from our new architecture.  
More than just serving the same traffic as before, we have been able to move over display traffic for new services like our Curations product that takes in and processes millions of tweets, Instagram posts, and other social media feeds.  That our new architecture could handle without change this additional, large-scale use case is a testimony to innovative engineering and determination by our team over the last 2+ years. Our largest future opportunities are enabled because we’ve successfully been able to realize this architectural transformation.

{more to come in part 3}

{This is the first in a 3 part series of posts intended to tell an impressive story of how we have been able to achieve an epic rearchitecture of our core platform. Special thanks to all of those who have helped in review and editing the original white paper.}

At Bazaarvoice, we’ve pulled off an incredible feat, one that is such an enormous task that I’ve seen other companies hesitate to take on.  We’ve learned a lot along the way and I wanted to share some of these experiences and lessons in hopes they may benefit others facing similar decisions.

The Beginning

Our original Product Ratings and Review service served us well for many years, though eventually encountered severe scalability challenges. Several aspects we wanted to change: a monolithic Java code base, fragile custom deployment, and server-side rendering. Creative use of tenant partitioning, data sharding and horizontal read scaling of our MySQL/Solr based architecture allowed us to scale well beyond our initial expectations. We’ve documented how we have accomplished this scaling on our developer blog in several past posts if you’d like to understand more.  Still, time marches on and our clients have grown significantly in number and content over the years.  New use cases have come along since the original design: emphasis on the mobile user and responsive design, accessibility, the emphasis on a growing network of consumer generated content flowing between brands and retailers, and the taking on of new social content that can come in floods from Twitter, Instagram, Facebook, etc.

As you can imagine, since the product ratings and reviews in our system are displayed on thousands of retailer and brand websites around the world, the read traffic from review display far outweighs the write traffic from new reviews being created.  So, the addition of clusters of Solr servers that are highly optimized for fast queries was a great scalability addition to our solution.

A highly simplified diagram of our classic architecture:

Simplified View of Classic Architecture

However, in addition to fast review display when a consumer visited a product page, another challenge started emerging out of our growing network of clients.  This network is comprised of Brands like Adidas and Samsung who collect reviews on their websites from consumers who purchased the product and then want to “syndicate” those reviews to a set of retailer ecommerce sites where shoppers can benefit from them. Aside from the challenges of product matching which are very interesting, under the MySQL architecture this could mean the reviews could be copied over and over throughout this network.  This approach worked for several years, but it was clear we needed a plan for the future.

As we grew, so did the challenge of an expanding volume of data in the master databases to serve across an expanding network of clients.  This, together with the need to deliver more front-end web capability to our customers, drove us to what I hope you will find is a fascinating story of rearchitecture.

It has been an absolutely crazy year since my last post. Work and family have both had me consumed, and while that hasn’t fundamentally changed, there is a story that must be told and I’ve carved out the time to write a rather long narrative that I’ll break up into three parts for our Bazaarvoice developers blog and will cross post here.

The story is one of an epic replatform that has reached an critical milestone completion.  It is a story of innovation and determination by brilliant people who I’ve had the honor to lead through a good portion of the effort. It is also a story of leadership and organizational change.  Truly, it deserves its own book.  Hmmm.

This is a brief teaser of what is to come.  I am so proud of the team at BV and what we have accomplished.  Can’t wait to share it with you.

Monday, an article was posted on Forbes titled “Why Software Doesn’t Follow Moore’s Law”. This topic has been on my mind some lately as we’ve been working to rollout a new platform with greater scalability, lower maintenance costs, and significantly enhanced capabilities. For engineers that have worked only on the new platform, it is easy to disparage the original platform. Many of them were were not even professional engineers when the original platform was written and they lose sight of how software architectures have changed over the years from the traditional three tier architecture to the lambda architectures of today.

It’s interesting to break this down a bit more. Are engineers smarter today than they were 8-10 years ago? And were those of 10 years ago smarter than those of 20 years past? In my humble estimation no, and one could argue todays graduates know appreciably less about how computers actually work but lets not digress.

What has changed is the way software is constructed. One can argue that modern languages have a major role to play in this, but one can argue also they are an outcome of the changes in architecture. But why has the architectural change occurred?

I believe a significant part of the change in software architecture is attributable to Moore’s law, somewhat in opposition to Mr Maccaba’s article in Forbes this week. Certainly engineers 10 and perhaps 20 years ago understood the advantages of isolation and service oriented architectures. Why didn’t they build systems in this way?

A major reason what that the systems and networks in place really couldn’t support it. Without gigabit networks, fast processors, and super cheap compute, the emphasis was on highly optimized code paths. REST Call? You would have been laughed out of town. Now the opposite is true.

Thus, Moores law has enabled much less efficient architectures to be feasible. Mr Maccaba also asserts this, but missed the point. These new architectures are much more scalable, can be built with fewer dependencies, can be changed independently since they are loosely coupled, and allow previously unobtainable compute goals to be broken down into independent parts and delivered. This is true whether we are talking about large scale web application infrastructure or big data analytics.

By constructing software architectures that take advantage of Moore’s law, we are solving problems that could never be solved before and constructing software systems of higher quality that we can actually deliver in much faster times to market. While certainly not doubling every 18 months, the time to market of new highly scalable solutions is measured in months today instead of years.

At the end of the day, I feel the point Mr Maccaba has missed here is that Moore’s law doesn’t apply to humans. Thus, we leverage Moore’s law to make humans more effective in software engineering through architectures that do deliver significant increases in scalability, quality, and capability.

The last few months I’ve started a new an exciting role at Bazaarvoice, and it has been a whirlwind of learning about the people, technology, and history that have made the company great. What an incredible place – its great to be a part of taking the company to the next level. I’ve learned so much in the last few months, and the more I learn I feel the more there is to learn. What a thrill to be surrounded by innovative, brilliant people in every part of the business.  In travels from San Fran to New York, it has been energizing to meet great leaders and passionate BVers in every part of the org. We are truly a strong and innovative company.

I’ve had the privilege to be part of BV IO 2014, our Conference and Hackathon this week. What’s super impressive is how the team can coalesce and organize something of this scale with no need for top down guidance. People step up and take on ownership like it was a startup. The panel of speakers assembled for day 1 included notables such as Dr. Bob Metcalfe, who unlike Al Gore, actually did help invent the internet. The day was packed with quality speakers and energy.

Today, the hackathon got into full swing and over 20 teams formed independently, taking on customer ideas and coming up with many of their own to further social commerce. You could taste the energy in the room. The XBOX tournament is about to start across the hall.

Throughout this brief time, I’ve been reminded of some important things. There’s a clear correlation of the need for Leadership in a team with their technical skill and the greater the level of skill arguably the more strong leadership is needed. It is more important to listen and ask questions than to speak. When you act, act swiftly and decisively. People will only listen to you after you have listened to them. And, I really suck at XBOX.

Coaching & Mentoring for Dummies

I’ve often thought about writing a book about leading software development teams to share some things I’ve learned through the years.  Perhaps I still will, but in a way this blog serve the same purpose.  It seems the market for this particular topic may not be all that large and  to appeal to larger audience would probably water down the content so, I’ll likely just stick to this blog as time permits.

Recently I had the pleasure to work with Marty Brounstein and have been reading his book Coaching and Mentoring For Dummies.  At first you may be put off by the title, but I’m really glad I dove in. In fact, it makes me glad I haven’t tried to write a book.  It’s that good. Many times, I found myself say yes, yes, that’s exactly right.

What I really like about this book is that it takes many of the things you may innately do and codifies them through a set of simple disciplined tools.  Now following this to the letter will not make you a great leader, but there is so much in here that is fundamental to building a great team. Read it. Do it.

The scope of this book is too broad to cover it here in this brief post. Its a cornucopia of leadership 101 as much as coaching. Not surprising since these are so intertwined. Right off the bat in chapter 2, the topic of building commitment and the strategies there I have found to be spot on through my career.  When I see managers manage from positional influence, it makes me nauseated and I instantly lose respect for them. Leadership is exerting personal influence and can only be accomplished through hard-won respect. I blogged about this about 8 years ago and still feel strongly about it today.

If you are a book skimmer, don’t miss Part III, The Fine Art of Mentoring. Some of my most admired mentors and greatest leaders I’ve worked with have perfected the techniques here. I’m still a work in progress, but try to follow these practices, because they really do empower others and help you be a more effective leader. Just one little gem here I’ll call out:

In many situations, such as when mistakes are made or performance efforts don’t go as well as hoped, opportunities exist for employees to learn lessons and correct their course. Using questions to explore what happened and what can be done to make matters work better provides experiences for learning and growth.

Bingo Marty. This approach can turn a very difficult situation where people are already feeling defeated by failure into one of a hopeful feeling for an better future.  Blame is easy, improvement is not.

Next week I embark on a new professional adventure which I am very excited about. I am challenging myself to be a great coach even when the pressure is high and things don’t go as planned. I hope you will find Marty’s work insightful and helpful.

As a leader of a Engineering Team, you may have had your fair share of experience with “Free kittens”. Free kittens are one off products, partner integrations, new platform support, or customizations that really only apply to one customer or are focused in a market that is outside your wheelhouse. These will be presented frequently to you as a fuzzy small requirement in a SOW, contract, or just as likely an email you’re copied on – fuzzy, cute, and often not described in much detail.

Of course, free kittens need some one to take car of them, buy them food, expensive trips to the vet when they are sick, and no matter how cute, theres a lot of crap to clean up. Same is true for the kittens that show up at your office door / inbox, usually delivered by Sales. (To be fair though, they can come from Product Management, Support, and directly from Customers themselves.)

As leaders, it is not usually effective or productive to say, “no thanks, your kitten is kinda ugly”. We are in the business to build and deliver software and services that delight customers which requires us to be adaptable and aggressive in meeting their needs. What is often more productive is to ask questions and educate the kitten custodian about what it will take to live with this kitten for the next 5-10 years (software kittens sometimes, but not always, have a shorter lifespan that real kittens).  In so doing, hopefully a business decision can emerge.

I’ve had more kittens than I can count show up at my door over the years. The latest one that came in very recently was run a one-off instance of our SaaS solution in one of the newer cloud hosted IaaS platform that I won’t name here. Of course we can do it (all things are possible with software) is the answer. However, there is this small matter that no one in this company has ever worked with that particular IaaS cloud provider.

I often find people miss the point on how to provide a service – its not a one time effort. Clearly training is needed for everyone who would touch this new platform: Dev, QA, Support, Services, etc. You need need a smooth repeatable process to dev, build, QA, deploy, secure, monitor, troubleshoot, update, backup, and recover from disasters. Turns out those cute little free kittens are somewhat pricey! Is it worth it for this one customer, or can we amortize this effort over a whole set of customers that we would move to this new platform? Can this large investment strategic? Good questions…

Of course, Sales people are not comp’ed on taking care of the kitten, just on its adoption. Sometimes this is called “happy ears” – they just want to hear yes, whether from you or the prospect. The sales person or product manager won’t have to feed it or clean the litter box, so it doesn’t hurt to inform them of the smells that can occasionally accompany these cute little guys. With the right business conversation that considers both the true value and cost of the kitten, you can drive a decision that works for your SaaS platform.

I had a ping recently to remind me that people actually read my blog and its been a while since I posted.  That does happen when you take a new gig. I’ve been working on several ideas in my Evernote and so it is definitely time!

I recently had one of my new team members ask, so what does it take to be a VP.  I was instantly reminded of the famous “So You Want to Be a Computer Programmer” post.  With a little less sarcasm, here’s my take on what you need to be a successful executive in this business:

• Ability to see big picture – beyond a specific instance of a problem to understand the cultural, process, and personal reasons of the situation.  For example, if a team is missing sprint commitments it’s easy to see, but uncovering the core issues and what is driving them can be tougher.

• Ability to set, communicate an effective vision, strategy, and goals – you must be able to explain the why.  Great teams care as much about the why as the what.

• Deep understanding of each role in the team – preferably having personally performed a number of those roles – developer, QA Engineer, program manager, UX designer, product manager, and so on….

• Familiarity and curiosity of new technologies, technical market shifts, etc. For example exploring new AWS services, Docker, node.js and other new movements, even if there is no direct applicability to a current project so that you can ask the deeper questions on future projects and foster innovation in your team.

• Fundamental understanding of the business, accounting, and budgeting. So much of your capability to execute comes straight from creative uses of the budget and making the most of what you have.  You don’t need an MBA, but a minor in finance or accounting is very helpful, and having a CFO that will teach you as much as possible about corporate finance is a Godsend.

• Deep understanding of people, what motivates them, drives them and how to make them successful.  This is critical to building a mentoring / coaching culture where you can pass these skills down and scale the team. Very helpful here is an understanding of behavioral styles (ala Meyers Briggs / DISC, etc) and how to communicate and inspire each type of person.

• Must be able to quickly assess the strengths and weakness across the team and particularly with the leadership so that you play to their strengths and give them support in the areas where they are weaker.

• Be ready to make and be accountable for difficult decisions. You are going to have to make tough calls on people, priorities, technologies, strategies and more.  At times there will be no great choices, only painful ones and at the end of the day the worst thing you can do is make no choice.  (Nowadays they call it fail fast.  There are lots of cute terms for truths that have been around a long time.)

• And perhaps number 1 – the ability to sell innovation and change to your team, to executives and to customers. People are generally change averse and whether it is driving a major new product architecture or whether it is helping someone digest the risk they perceive in a new product, the executive must be able to sell innovation and change before they are able to drive effective realization of that change through goals and objectives.

What a team!

Through tough times and celebrations, we’ve built our team. Each person hand selected to grow our ranks, and each bringing a special gift and a lot of old fashioned hard work to make the whole better. The team we built together is a thing of Quality in the truest sense of the word.

A few Friday’s ago we said our farewells. I was overwhelmed with gratitude at the send off – from the motorcycle cake, to the fun contests, to all the heartfelt words. I’m so glad to have been a part of building something so special for the last 5+ years, a team that continually reinvented itself. The best thing about the send of was that it felt like a new beginning rather than an ending.

It was an honor to serve as your leader; the experiences and success we shared together will always shape our future. As we look ahead, I wish all the best for each of you. Your mission is to continue to grow as leaders and forever shape the tech landscape of Austin. Know that the Quality of what we built will not be forgotten, nor will the champagne shower!

Its that time of year again – time for performance appraisals. Sometime used for great good but mostly dreaded as a necessary evil. It should be a great opportunity for a review of how we did towards the goals set during the year. But, it often turns into something else.

Ever been forced ranked or had to have your performance reviews fit into a bell curve distribution so that most people are “Meets Expectations”? Since when is it ok to meet expectations? Is that the culture we as leaders should promoted and foster? When we hire people, do we expect them to surprise us with innovation, creative solutions, outside the box thinking, or to meet our expectations? If we hire the former, then why would we believe that the majority of performance evaluations should be “meets expectations”?

Just read this post from today and wanted to share as well as pile on: Learn about your managers through the people they manage
by Richard Rosenblatt. On understanding what is really going on in your organization, Richard recommends

So, how do you get a more multi-dimensional perspective on how your direct reports are doing? It’s easy. Spend time with their peers and direct reports.

In practical terms, I recommend using one-on-ones with all your direct reports and augment this with skip level on-on-ones over lunch.  I also have found that round table lunches with non-manager thought leaders in the team are invaluable in gaining feedback and also suggestions for improving the team.

If you feel you don’t have enough time for one-on-ones with your team, I encourage you to read Ben Horowitz’ blog from last August on the topic.

I’ve worked for CEO’s that used skip level one-on-ones and hallway conversations to both keep their finger on pulse on the business but also to help understand what was really working in the organization.  Personally, I never felt threatened by this because of the utmost trust I have in my teams but also because we were talking frequently in our one-on-ones.

This emphasis on transparency builds trust and trust is an essential ingredient for winning teams.