Why Engineering Teams Fail

Most engineering projects fail. And it’s true. 

What in the fuck is going on? 

The traditional quote is that, it’s 99% of engineering projects fail. And 90% of startups fail. 

The weird thing to me is that we accept this massive failure rate as normal and inevitable, and in fact good — only the strong survive — and while we try to apply better engineering methodologies on a startup-by-startup basis, and develop norms and practices, there is insufficient discussion of how we can deal with this on a more global level, on a more fundamental level, that addresses root causes. Looking at these failure rates in a material way, as opposed to some weird venture capital negging and propaganda about the glory of startups, means confronting the underlying structure of how money for technological development is deployed. The reason for the astronomical failure rates of engineering projects and the often shoddy results of them, are things far beyond “bad communication” and “lack of product market fit”; something bigger is driving something as encompassing as this massive failure rate. 

Again, I must note that while people assume that our tech industry is top notch, amazing, etc., they are not aware of the parameters of that and do not consider the possibility that actually, the way we do engineering in the Valley is not good, poorly constructed, badly designed, intractably dysfunctional, doomed to massive wastes of time and money, and that we could have way better results, and waste far fewer resources, and have much better products, and better performance. 

As advanced as people think the industry is, most experienced systems engineers openly refer to the internet as being built out of glue and sticks and pipe cleaners… and they worry a lot about the state of the infrastructure. Hard problems here, but there are some big indicators here that we have a lot of room for improvement; as computer professionals, we must analyze, according to the practices and code of our field, the root cause conditions for failure scenarios; these are analyzed locally within startups themselves if, for example, there is a major outage; but that is much more localized than what I am talking about, which is fundamental, materially and widely agreed upon, problems in the industry. 

As most of my experience is with venture capital and startups, I know quite a bit less about how engineering is run at the giants; but I do have quite a bit of experience selling distributed infrastructure into large enterprises — although their bread and butter is not computing like we would see at a giant, it still offers sufficient evidence of the larger systems problem in technological development under venture capital. And people working at the big engineering teams report back that it is equally as much of a shitshow and there are just as many problems there as are within the industry itself.  

It is absolutely fascinating and bottomlessly distasteful to me that these horrific rates of failures, representing the loss of incalculable man hours and unthinkable capital, total waste of resources and energies, hideous mismanagement, is presented as some kind of virtue of the field; all because the failure of the 99% is a testament to the concentrated wealth and power of the 1%. There is actually a huge amount of culture fluff, ideology, folk wisdom, etc., that our failure rates are actually just signs of the natural merits of competition and the free market etc. These logics fail when you realize the extreme degree of market control and management of the industry by venture capital. This is not an organic environment and we are not seeing some nebulous organic failure condition inherent to “innovation” and “competition”; we are seeing the material results of material strategies. The failure rate is something that the industry PRIDES itself on: the gauntlet from which only a few emerge. This is directly connected to the general power law practices purposefully employed by venture capital, it is about concentration and distribution of industry proceeds. These failure rates means that huge amounts of totally competent and capable engineers, are working on stuff that will go nowhere, costing huge sums of money, and this is distorted into some kind of hyper masculine gladiator LARPing, a badge of courage. This total fucking disaster is accepted as industry norm and considered like, “the process” and how hard software engineering is. Fucking cry about it. We can do better than this. 

I got my start in this industry and specifically in hunting venture capital, while analyzing why engineering teams don’t work. Everywhere I went was engineering teams at the brink of collapse, everywhere was wasted code, replicated projects, massive bleeding out, the engineers were all mad as shit and hated the management, the communications between engineering and the rest of the company were reminiscent of a Cold War. Each startup is custom designing its own engineering practices, cobbling together their unique philosophy and the DNA of whatever startups led them to this juncture. Bizarre outgrowths of agile, misapplications of kanban, surprise surprise, the engineering team wants to use a product management software made by 4 fanatics from Burning Man. “Rage quitting” happens a lot on the job and that is when an engineer gets so fucking pissed off that they just leave. One kid I know did that literally a few months ago. Like your engineers will just walk out the door and be like fuck you, I’m gonna go work at Facebook for a 300 base. And they’ll have that job by the end of the day and you won’t have another engineer for like 3 months. So you’ll have to contract it out to someone who has no clue what they are doing, and so on. Of course, with the introduction of job precarity into even the American full time programming workforce, this ability to just leave and move around in the system, will be extremely constricted. The rage will just fester. 

 Working from product management, developer communities, marketing positions, and with a constant audience with C-suite going back 15 years, I have had a first row seat into how this functioned in many different companies, and I remember just looking around constantly like: this is fucking untenable. This is not working. This is no way to run engineering teams. 

People in tech do and are doing amazing work, but one thing people don’t believe after being brainwashed for so long by venture capital, is that technology could actually be way, way, way, way, way, better than it is right now. And that is a future we should fight for.  

I’ll skip to the lede here, let’s call it Kane’s Law: the reason why engineering sucks and the products suck, is because every engineering project is a battle between engineering and management, aka venture capital, which is laying in the wings getting rich as shit, with a direct incentive to squeeze as much out of engineers as possible, on their agenda, to cut them as much out of the ownership of the company as possible (stock), and further, to openly serve a fascist accelerationist schedule, creating the breakneck requirements that lead to “death marches”, widespread burnout and sometimes the failure of the entire company. 

Venture capital fundamentally compromises our technological development and far from being generative and productive to the human cause, the technological development has flowed directly from the incentives of the VC, orthogonal to the engineer, causing a fundamental rift between engineering and the rest of the startup body, and THIS is what destroys engineering teams. If we are truly looking to see 10x gains in the engineering capabilities, and if we are willing to literally steal the entire industry’s code without credit or compensation in order to make coding AI, one would think that we could also include some discussion of these more underlying economic management of engineering labor and the development process. Follow the money, etc. How capital is made, deployed, managed, increased, must always been on the table. 

How do we claim to be so “innovative” when the model of technological development for 30 fucking years has been the same tiny group of VCs and the leaders of the tech giants, using the same fucking business models and the same labor practices, the same bullshit, the same excuses, the exact same model of funding and growing startups. If we are so innovative and flexible and cutting edge, why is the discussion of these more underlying dynamics, always off the discussion field? These models should be constantly evaluated and iterated and changed to be better; we have stagnancy, entrenched bad engineering practices, and massive failure rates coming out of the system. Seriously: how do you plan to innovate when the plan for technological development has been the exact same, the exact same VCs, the same accelerator model, the same Delaware registration, the same cap table and term sheets, the same pitch decks, the same schedule for financial distribution to startups, the same metrics for engineering (developed by venture capital and for the motivation of VC returns on various goals). Nothing at the base is changing even though it can be connected to so many failures of the industry; it isn’t even on the table for discussion. We cheer 99% of projects and startups failing because the goal of venture capital is to achieve a massively disproportionate amount of money and concentration of resources; ensuring the failure of so many, ensures the concentration of wealth and power and encourages the monopolization of infrastructure by venture capital. 

This is absolutely nothing to be proud of in this industry. 

It is oft quoted in the Valley that most startups fall to internal forces, not to competition or other external causes. And indeed, it is suggested that these internal forces are many and diverse. Founder fall-outs (almost guaranteed, lol, never start a company with another person), inability to ship, ineffective enterprise sales teams, disruption in leadership, lack of community growth, running out of runway too soon. But I think there is something below that layer, that is in the fundamental set up of our startups and that spans almost all of them. 

The venture capitalists have been talking a lot about AI accelerationism, and its a place where we can see this dynamic playing out — after all, it is the software engineering labor that produces the AI, so outside of capital and monopoly on the instruments of production, labor is a fundamental target of technofascist accelerationism. 

Accelerationism is the great coercive force to apply to a workforce. Accelerationism is many things, more concretely than a philosophy, but it also about concentrating a giant amount of capital and labor around a particular area that the capitalist thinks will make them the most money. The POWER for accelerationist timeframes comes from workers and by pouring capital on the issue, causing growth. Therefore, the engineer is the natural target of the venture capitalist for extraction and exploitation when accelerationism is on the table. And this accords with what we are seeing around the industry, where engineering labor is being destabilized with AI, outsourcing is through the roof, stock grants are plummeting , and yet more emphasis is placed on militant delivery, and venture capitalists are running around crowing about the “parasite” engineers that come to feast on the winnings of the venture capitalists — you think I’m joking? 

Here’s what a16z is saying about its precious engineers in its marketing books, this via the thinly veiled fascist screed “Network State”, by a16z venture capitalists, in regards to technical workers who work at a startup after the initial and heavily gate-kept funding of it: 

“‘Strong men create good times’ … but as they scale… they also start attracting lazy parasites to the wealth they’ve created, people who want to join something great rather than build something great… no one wants to work as hard or be as ruthless as that early Spartan band, given the easy wealth available, so they enjoy themselves and busy themselves by fighting each other over trifles.” 

“Once the founder reaches the seemingly inevitable point where they need a bureaucracy… the parasites start entering. They don’t want the risk of a small or even mezzanine-size business. They want lots of perks, high salaries, low workload, and the minimum work for the maximum return… they know they don’t need to pull their weight…”

So in this case you have the biggest, most important and most capital-influential venture capital firm, a16z, openly referring to the engineers that build their fortunes as parasites. This is an open glimpse into the actual dynamic between the industry’s primary labor workforce, computer programmers, and the venture capitalists who must use them to extract products and capital. 

Engineers think that their beef is with the executive team. Often, the major beef is with marketing, around delivering according to release dates; or with sales, which takes times away from the engineering team for everything from technical pitches and support, while also pressuring them to deliver on new customer features. In the startup environment, things are overly personalized as we are wont to do as human, and so in general, engineers think of everyone in the company besides them, as being the enemy and further being, well, morons. This turns into “the CMO is an idiot” or “the VP sales doesn’t understand what we do here”, which are likely to be true, but also doesn’t explain why this exact same thing is happening over and over at every startup you go to, with the exact same conflicts happening, ending up with the engineering team being estranged from the rest of the company, leading to worse working conditions for everyone, but especially the engineers. And of course, this dynamic is associated with the massive failure rates we see across the industry.

Programmers don’t realize that the actual beef here is with venture capitalists, who are architecting the technical direction of the entire industry through large blockades of startups designed to work together on singular agendas. The chain of command goes directly from the VC to the founder/CEO, who then manages the executive team. The orders are still always coming from the top. Further, executives themselves are usually installed by venture capital and come from somewhere inside the venture capital conspiracy. The direction of the executive team is set by the venture capitalist and the executive team is the functional representation of the interests of capital (VC); the real beef is with the man behind the curtain, who is pushing the executive team in myriad ways and making sure they instantiate the model within the startup. 

Most of the engineers will never even meet a venture capitalist. There’s some more aggressive reach out now than there used to be from the mothership, but the VC is not involving itself with your line engineer; so the executive team is all the programmers see as material; it is hard to identify what you cannot see, and easy to assume that what you see is coming from what is before you. AKA, some random fucking asshole in a Brooks Brothers suit. I once had a boss who wore Mickey Mouse ties. I quit.

In reality, the executives and yes, even the founder, are hired and managed by the venture capitalists to carry their demands; thus, they are a communications node that is then charged with distributing the commands of VC and interpreting them across the company. They are in many ways just miniature clones of the venture capitalists and more aligned with the venture capitalists than with their employees; it is with venture capitalist whom they share the winnings except for scraps for the engineers.

Venture capital is the point of centralization in this entire system — where all of the money and power is hoarded. It communicates with its startups like a dictator, is reliant on exploitation of labor, and, via mechanisms beautifully described in Conway’s Law and Marxism, this is an original sin that contaminates the rest of the field. This is from the introduction to the Conway’s Law paper: 

“The basic thesis of this article is that organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. We have seen that this fact has important implications for the management of system design. Primarily, we have found a criterion for the structuring of design organizations: a design effort should be organized according to the need for communication.”

In all ways, the venture capital is the superstructure of the startup. I would add to Conway’s Law, in utter respect and in tribute, that capital flow is a communications structure that is even more fundamental than those of discourse, but in all areas, venture capital is the centralized tower from which strategy, capital, direction, command, communications, originates and then flows into the startup, where the capital and labor are used to produce technology projects of various kinds, including perhaps most relevantly, the instruments of production which we see are very much owned by venture capital in this model. 

Then the gains of such are distributed back primarily to the VC in terms of capital, data, customers/users, new surveillance mechanisms and other gains of this process. 

This is the true fundamental communications structure of a startup: we are the bitch of venture capital. 

We are too simplistic when we only look at the internal communications structure; there is an overarching system, of which startups must be correctly identified as a sub-system of the larger agenda — whether that is web 3.0 more broadly or work on the various crypto and AI initiatives, and most importantly, the superstructure of venture capital.

Marc Andreessen, the head of Silicon Valley, has done a great job of playing both sides on this debate. As part of ongoing efforts to radicalize the engineers and get them revved up on the fascism train, which, weird, is not only on time but arrived, Marc has tried to capitalize on the famous tension between engineering and its proxy management layer, by trying to inflame them about the “managerial class”, and promising to get rid of management, in a flattening of organizational charts; this is what we saw at Meta, where a very significant demonstration of the new model was made. The public firings were for the engineers — good times are over, we’re accelerationists now, its warp speed. But then you have this being done under the illusion of relieving the pressure on the engineering workforce that has driven our technological production into hell: management. AKA venture capital. 

 This is a great example of terrorism by the tech elite of the engineers, is then justified by “management bad”, with your common systems engineer being too daft to identify a common systems anywhere outside of his nap pod. But it venture capital that is the true managerial class in every sense of the word. 

The venture capitalist template is where the model flows, even though it is posited like decentralized technology development, that is actually not at all how the systems operate; no one with a straight face can say that technology is a decentralized place when the engineers at all the major companies have top-down, dehumanizing designations like “L8 engineer”, with each category according to a skill set, a realm, a base salary, an allocation of stock options. 

Or a place where all the engineers come out of the same small handful of prestige schools . Marc wants to bitch about the educational system while he has been harvesting from it successfully for decades without having to pay for it? This goes deep with him, its a multigenerational Stanford thing; another thing where we see just hypocrisy from lazy venture capitalists who don’t even bother to put out any more material analysis than “managers are bad!”, who are looking for the best ways to agitate more work out of their guinea pigs.

If we’re being honest, we have to consider that engineers are exploited. At the same time, engineers are often (minor) shareholders in the company, so they do have ownership on the instruments of production: in technology, the instrument of production is compute. That is very deeply monopolized area and you can see the vast wealth accumulates around the compute layer and just around platform companies in general, like we’ve seen in multiple generations of financial technology, I.e. The core mission, energy and capital from PayPal was used to plant Coinbase. At these platform layers you also are dealing with significant amounts of compute and monopolies on compute by vertical; I.e., consider Airbnb’s compute footprint, or Coinbase’s, you are looking at very large compute footprints and even in OpenAI, where they are resource constrained by access to GPU. All of these companies are funded by the same venture capitalist, Marc Andreessen, and represent the control and centralization points on various verticals. And indeed, the fact that engineers have had some ownership, however small, to this heart of capitalism itself, cannot be discounted.

However, the new startup model, as we’ve discussed on this blog, involves pushing engineers out of stock options, the primary way we get major financial gains to engineers. The fact that they HAVE received stock options and thus have actual ownership, even though paltry in the overall wealth distribution, has divided them from other “workers” and at times made them… not really fairly in the class of workers. But, that is a topic for another post; the point being, this field is being rapidly destabilized and even computer programmers are being turned into gig laborers with no stock. It will be interesting to see if at any point they realize this is happening to them, but I doubt it. Unfortunately, software engineers are notoriously bootlickers and you’ll note that the boots they lick, are venture capital boots.

Conway’s Law.

The vast majority of engineers do NOT have significant stock particularly outside of the privileged realm of blue-chip firms, and in both cases, we see this core disruption where antagonism between, if you would permit me to be so bold, labor and capital, disrupt and destroy startups. The venture capital model is built to profit on this model, and has absolutely no incentive except to continue the pressure, the conflict, the abuse, the discord and the failure that we see. 

As I’ve discussed in the past, I honestly and genuinely think that computer scientists could have gone either way on the moral spectrum, for whatever way of quantifying that you use. I would describe efforts to stop tech fascism that occurred within the industry during web 2.0, as a battle between two possibilities for the engineering worker body: whether they were to fall and be obedient guinea pigs for the venture capitalists, or if they were to develop their own identity as workers, and clash with the venture capital overlords. 

They picked venture capital. 

 However, that does NOT mean that the dynamic between VC and engineering labor, isn’t a massive rend in the technological progress of our field, or that this allegiance with the overlords isn’t heavily coerced through brainwashing, bribery, lying, stealing, manipulating, introducing artificial precarity, and overwork. 

The amount of discontent of programmers overall and particularly within the political landscape of the startup, is going to increase; but venture capital won’t be perceived as the problem. This gives the venture capitalists the opportunity to capitalize on the discontent they create, like we see with this whole “managerial class” talking point, which hides labor destabilization under the augur of addressing long-running, VC-created discontent. Discontented workers produce energy that can be harnessed to larger agendas. Venture capital is in the middle, right now, of evolving into a rising fascist state; their programmer labor force is the obvious first body of this party. They will continue to use programmer discontent, created by them, to drive this energy into fascism. This is the biggest problem we face right now. 

But it cannot be left unsaid, that people are so bought into technology as we see it, as representing some objective “bleeding edge”, that we don’t consider that we could be offering way better technology, that is way better for people and workers, that is much healthier for laborers, and which distributes gains back to the world. This is my contention in the article: the engineering sucks because of the venture capital. 

Venture capitalists aren’t the guardians of innovation, they are its destroyers.