A critical appraisal of the key deficiencies in risk workshop methodology that may lead to them producing of misleading or useless data

Tuesday 2 December 2008

From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
I have looked at this and all of these things are well understood social phenomenon that a qualified facillitator should be able to avoid or at least should plan for. I have found that risk based scheduling and risk based PERT systems work best if facillitated well from the beginning. Rather than seek to minimize expected costs they seek to minimize variance. The methods are nice because they can be rank based or subjective, as long as raters are consistent with themselves. The mechanism kills projects early by forcing failure early rather than delaying an inevitable failure late. Incentive neutral expected costs are higher because you are not minimizing against expectation, but rather against expected or in some cases median risk, but it is rare that the cost difference is large. Indeed a project killed early is far less costly than one that is permitted to continue so people keep their jobs. It puts the incentives on the table up front. Project planning is NEVER incentive neutral. One real risk is "who is going to get fired for telling the truth." That sets up powerful incentives to bias estimates, "rework" solutions and dig for further capital to keep the process running.

Some of your problems are not "problems," but rather incentive failures. Consider your post:

v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.

There is substantial research that shows that groups only learn through dissent. Overcoming organizational defenses and the associated incentives is the true problem and must be addressed prior to any risk workshop. If the facillitator has not addressed "the unspoken," the group process itself is a danger to the risk management process. Your list can be overcome by good process. Your list is a description of process failure, but not inevitable process failure. If an element of a process has known faults, like over valuing extreme outcomes, then the process should include that knowledge. Failure to include such knowledge is a failure to learn and implement learning.

I am also extremely aware that this is very hard to do. Group equilibriums are incentive equilibriums and these equilibriums are very powerful. Only the marginal decision makers can move them, which may not be members of the management chain you are working with and may not be members of management. Finding these leverage points requires substantial work that would appear to the customer as non-work because you are looking for which social contract(s) is driving all other social contracts, which is both threatening and does not directly appear to bear on the "work at hand."

All of this gets lost in discussions of "misaligned" incentives. Incentives are never misaligned, the subjective goals of the marginal deciders will in expectation be reached, or everyone will go down. The problem isn't "misalignment," but rather poor contracting system wide and system controls sitting somewhere other than in the self-interest of the principals who made themselves "non-marginal."

I disagree that "no facillitator, however good, can resolve," because that is more than a strong statement of certain process failure, it is a statement that learning is impossible. In some groups and work systems this is very true, but only because people are paying them to not learn. If you pay people to be stupid they will gladly be stupid for you.

I am concerned about the emphasis on "cognitive maps." They are tools and very good tools, but you should always question the assumptions behind a tool before using it. In the case of physical tools, like a Phillip's head screw driver, this is easy. Does your problem contain a screw with a Phillip's head? In the case of social tools whether individual tools like cognitive maps or whole systems of tools like Six Sigma, they contain assumptions that must be met in whole. Are they met?

____________________________________________________

No comments: