In my Professional capacity, I often come into contact with various methodologies such as Agile, ISO and Six Sigma. I've seen it done Right and wrong -- and I do think that in general Agile is the best of them. However, it is worth saying right out of the gate that all of these things cannot magically increase productivity, or make the lumpen creative. All the process-oriented approaches can do is get you uniform output, and as such reduce the incidence of stellar catastrophes.
As to why Agile by and large hits the mark, I believe the reasons lie mostly with praexological principles. When I am exposed to new ideas, or go over old ones, I have the habit of seeing it through the praexological lens. It struck me how similar the reasons that the failures of the "waterfall" method of software development resemble those of central economic planning. Requirements are decided by wise overlords, which are then implemented by programmers, checked by QA, and sold by marketing/sales. If any one of those in the chain fail, the project fails -- and the longer the planning horizon is, the more catastrophic the failure becomes.
Agile solves this effectively through decentralization and redundancy, which is the place that most organizations fall down in implementing agile. It is worth noting that this is also the way to mitigate the the ill effects of malinvestment brought on by failed economic calculation. As above, so below.
Another interesting concept in Project Management is the Iron triangle of Cost, Scope and Schedule -- that if you insist on any two, one must give. This is fundamentally what economic calculation is all about. Agile recognizes that if you insist that all three requirements are met, no amount of effort can make it a reality. If only the state would realize that backing up it's insistence on project goals with a gun is similarly futile and deluded.
The reasons for this are largely due to the information paradox created by all economic valuation being subjective. Value is subjective due to the impossibility of any two individuals having an identical informational context; like a program, we cannot rule out differing output when run by different interpreters, even if we feed it identical inputs. Similarly, the possibility that a group of stakeholders will ever be able to come to perfect agreement on the three points of the triangle quickly approaches 0 as the number of stakeholders increases.
Anyways, most businesses are not actually built to maximize the company's revenue, but profits for the owners; and as such they tend to be arranged hierarchically. This inevitably conflicts with the notion of "self-directed teams" in such organizations, as anxious managers and info-hoarding Neddries will lose power. The radical transparency required for such a team to correctly function is often a bridge too far, usually due to a lack of trusted agents. Most organizations are simply incapable of providing sufficient incentives to attract and keep employees worthy of such trust.
Similarly, governments always tend towards centralization due to the fact that doing such institutionalized violence makes finding trusted agents practically impossible. One look at the state of politics bears this out; only those capable of great lies can bury the skeletons required for maintaining the aura of legitimacy. Similarly, the requirement to bury such skeletons inevitably kills the level of transparency required for effective cross-agency co-ordination. As such, de-centralization is seen as an existential threat rather than as the way to effectively achieve their stated goals; and governments default to self-preservation becoming the goal.
