David Harris
Student at West Virginia University
dharris1@mix.wvu.edu

I am adding a few texts to help out for those who are interested in reading about this. I used to have an entire library of books on the topic, but they went with the job and now I only keep a few.

One of the better books is "Overcoming Organizational Defenses" by Chris Argyris at Harvard University. There are also a series of good dissertations run through the Center for Entrepreneurial Studies and Development through the Industrial Engineering Department at West Virginia University on risk based management of large scale projects and risk based scheduling. Last time I heard they were developing software to schedule projects based on risk.

I will look for a good paper on the bathtub curve. There was an excellent one in either IEEE's primary journal or in the management SIG journal.

The observations on equilibrium contracts comes out of basic economics, however there have been psychologists who have been very successfully using this observation to transform groups. If you can find the marginal actors then you have found the point where leverage can be applied to change the whole system. Again, there is literature but when I went to get a PhD I gave up a wealth of written knowledge on shelves at my prior employer.

Finally, there is of course the wealth of writing that comes from W Edwards Deming and the quality movement that followed. In particular the structured programs like Six Sigma or the one at Carnegie Mellon's Software Engineering Institute the Capability Maturity Model are more valuable as teaching tools than as programs. (I am sure they would strongly disagree). The reason I say that is that a good quality program has to be executable by management and most of these programs fail because the changes are too large for senior staff to manage. Such programs are supposed to change how managers manage not how workers work. By changing how managers manage, workers change their work. That is harder to do than it sounds. We behave as we do because we are satisfied and avoid contrary behaviors. If the behaviors are unsuccessful that does not mean we will change, it just means we have unsuccessful behaviors.

_________________________________________________

From: Matthew Leitch [mailto:m.leitch1@ntlworld.com]
Sent: 11 December 2008 16:08
To: Gavin Lawrence
Subject: Re: [riskanal] Pertaining to Risk based Cost Contingency

An excellent post. In addition to your sample of workshop problems I would add that I find it deeply depressing to see good ideas I have worked out (at considerable personal effort) misquoted on a flipchart, "mixed and mangled into useless oblivion" along with the dross. The quote is from my book, or at least a draft of it.

The tendency towards seeking consensus might be reduced slightly by saying early on that disagreement will be taken as a signal of uncertainty and uncertainties will be noted and responded to, not squeezed out in an attempt to reach agreement. That would go along with trying to get a view from everyone who is informed enough to hold on on an issue. Risk 'ratings' are estimates and some are supported by more evidence than others. Disagreement indicates that the ratings are likely to be flaky and we should think about finding out more.

Prediction markets could be a better way to get the inputs of lots of people on uncertain quantities. I suspect the software already exists and some clever schemes have been worked out so that even if you need lots of probabilities you can still make it work.

Matthew

_________________________________________________

LinkedIn
Date: 12/09/2008

Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information

Group: Enterprise Risk Management Association

Rakesh Dighe wrote:

Hi

I have used Risk Workshops for over 12 years in an Oil Company setting. There are some important pre requisites to obtaining a successful outcome e.g.

Prepare a strawman in advance using a Subject Matter Expert
Lay down Big Rules for the workshop
Define clearly the risk scoring criteria
One single person to document output
Not a good idea in some cultures to have the Big Boss present at the workshop as it stifles discussion or aggressively leads it in a certain dominant direction

Rgds

Rakesh
CEO
Risk Quotient

_________________________________________________

November 28, 2008

I'm impressed with your proposals on why risk workshops may be defective. I would add the obvious point that risk management is more science and less subjective than most other management disciplines. Thus, methodology becomes a matter of importance to the analysis process. And of course, with methodology come the issues of reliability and validity. All of these concepts are unwieldly, and so many learners lose interest in the presentation points of the given workshop. I'm still wrestling with this myself in front of my learners, but it does seem to detract from workshop success at times, depending on the participants and their backgrounds. Good luck!

William McKibbin
Consulting Financial Engineer & Risk Analyst
wjmc@mckibbinusa.com
____________________________________________________

My view is that workshops are great to bring out ideas about risk issues. However, they are hopeless at getting quantitative info. For that, I believe the 1 to 1 interview is better, and then compare estimates. Delphi method suck.

Also a workshop can be a place where people don’t really speak their mind, but will do (more often) if the 1 to 1 is anonymous.

Best wishes
David Vose, Senior partner, Vose Consulting Group.
david@voseconsulting.com

____________________________________________________

While the specifics of risk identification are tenuous, the more important element is what management does once a risk is observed. Mitigating or minimizing risk is a dynamic of business life. Yet, in the present economy, and the prior few years, the facts indicate that beneficial, post-recognition activity is non-existent.

I suggest that the risk training should also include some discussion and elevation of decision processes and their down-stream impacts. This would be analogous to both designing fire exits in an office building (risk assessment) and then actually constructing them for use (decision processes).

Greg Chenevert,
Independent Business Consultant, Manchester NH, USA

____________________________________________________

RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
From:Larry Cosgrove
Date:November 27, 2008
To: Gavin Lawrence
Group:Catastrophe Risk Modeling Group
Status:Pending
Gavin

I suspect that many of your comments are dead on. I do know that the "people translation" to those not familiar with numerical modeling and statistics is the critical issue here. If you know information to be incorrect, it is best to stand up and make the point. Remember that to those making decisions, sometimes the terminology (especially acronyms) and methodology can prove to be very boring when placed in an open discussion.

In weather prediction methods, I find it best to act as the "translator" while doing my best to overcome the perceived "risk of ridicule". For instance, if the numerical models point to a cold period and vendor forecasts are far warmer and/or calmer, I always provide pictorial proof of what I am describing.

So if I were to give advice for increasing the value or a risk workshop, I would offer these suggestions:

1) Keep the discussion lively; use expressions to go along with the data and engage questions or comments after your initial arguments

2) Graphics that exclaim will do the job far better than charts that induce sleep

3) Always end with a firm conclusion, not a nebulous correlation

Best Regards,
Larry Cosgrove
Chief Meteorologist at Avant Capital Management/Irish Exchange LP
Owner at WEATHERAmerica
Chief Meteorologist at WWJM-FM

____________________________________________________

RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
My post may appear critical and I have never been to the UK or used the risk groups there, but the failures are the failures to engineer a poorly controlled and undesigned device (humans and human groups) into the process. No one would design humans as we really exist. Human groups and humans as individuals would be, if we had been engineered, an embarrasment to the designer and never make it to market. Rather than fail at an exponential rate, like all intelligently designed systems, we experience "bath-tub" curve failures. Groups do not like to design around the "bath-tub" curve because we are all intelligent, caring, ethical and well meaning. Such a thing would imply something might be wrong with "us," and that is very threatening. If something is wrong with "us" we might lose our jobs. Being human is very difficult as it is, failing to design around our in built failures is process failure. Facillitators and managers, being human, share these faults. Process should work around this unforgivable fault in managers and facillitators so that the process is better than human, even if the defective machines implementing it are human.

____________________________________________________

From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
I have looked at this and all of these things are well understood social phenomenon that a qualified facillitator should be able to avoid or at least should plan for. I have found that risk based scheduling and risk based PERT systems work best if facillitated well from the beginning. Rather than seek to minimize expected costs they seek to minimize variance. The methods are nice because they can be rank based or subjective, as long as raters are consistent with themselves. The mechanism kills projects early by forcing failure early rather than delaying an inevitable failure late. Incentive neutral expected costs are higher because you are not minimizing against expectation, but rather against expected or in some cases median risk, but it is rare that the cost difference is large. Indeed a project killed early is far less costly than one that is permitted to continue so people keep their jobs. It puts the incentives on the table up front. Project planning is NEVER incentive neutral. One real risk is "who is going to get fired for telling the truth." That sets up powerful incentives to bias estimates, "rework" solutions and dig for further capital to keep the process running.

Some of your problems are not "problems," but rather incentive failures. Consider your post:

v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.

There is substantial research that shows that groups only learn through dissent. Overcoming organizational defenses and the associated incentives is the true problem and must be addressed prior to any risk workshop. If the facillitator has not addressed "the unspoken," the group process itself is a danger to the risk management process. Your list can be overcome by good process. Your list is a description of process failure, but not inevitable process failure. If an element of a process has known faults, like over valuing extreme outcomes, then the process should include that knowledge. Failure to include such knowledge is a failure to learn and implement learning.

I am also extremely aware that this is very hard to do. Group equilibriums are incentive equilibriums and these equilibriums are very powerful. Only the marginal decision makers can move them, which may not be members of the management chain you are working with and may not be members of management. Finding these leverage points requires substantial work that would appear to the customer as non-work because you are looking for which social contract(s) is driving all other social contracts, which is both threatening and does not directly appear to bear on the "work at hand."

All of this gets lost in discussions of "misaligned" incentives. Incentives are never misaligned, the subjective goals of the marginal deciders will in expectation be reached, or everyone will go down. The problem isn't "misalignment," but rather poor contracting system wide and system controls sitting somewhere other than in the self-interest of the principals who made themselves "non-marginal."

I disagree that "no facillitator, however good, can resolve," because that is more than a strong statement of certain process failure, it is a statement that learning is impossible. In some groups and work systems this is very true, but only because people are paying them to not learn. If you pay people to be stupid they will gladly be stupid for you.

I am concerned about the emphasis on "cognitive maps." They are tools and very good tools, but you should always question the assumptions behind a tool before using it. In the case of physical tools, like a Phillip's head screw driver, this is easy. Does your problem contain a screw with a Phillip's head? In the case of social tools whether individual tools like cognitive maps or whole systems of tools like Six Sigma, they contain assumptions that must be met in whole. Are they met?

____________________________________________________

The risk workshop methodology is deeply entrenched in the UK risk management process and has been for many years, nowhere more so than in project risk and safety risk. If you examine the international use of risk workshops you will note that they are far less and even minimally used as a means of eliciting risk knowledge.

An examination of risk British literature over the years has not revealed any critical analysis of the performance of risk workshops in delivering accurate risk quantification (an examination of major UK projects from infrastructure to IT will demonstrate the gross cost, performance and schedule deficiencies).

Careers are built on facilitation of risk workshops and consultancies charge substantial fees for undertaking them. I have sat in on many risk workshops and discussed the results with many risk managers who expressed concern and dissatisfaction with the results.

A critical appraisal is long overdue on the effectiveness of the risk workshop process and to this effect I drew up the list of points given in the 'blog' post. All these points are logically obvious from allied research in the socio-psychological fields. One particular area I haven't addressed is that of "Successful sociopathy" which has significant import to risk analysis e.g. Enron, and may be the contributing factor in the 'over-confidence' identified by Bent Flyvberg (Megaprojects and Risk) also cf. Network Rail West Coast Mainline upgrade, NHS Data base, Battlespace, CSA, Olympic Games 2012 etc. etc..

A number of risk managers have suggested to me that the camaraderie and "group hug" are important aspects of the risk workshop. It is my contention that this is total nonsense. If you want to build team spirit go to the pub or go paint-balling. A risk workshop has only one function and that is to deliver identification of risks and accurate estimates of their probability and impact. If it fails on this single measure it was a waste of time and money.

The solution I recommended but am not locked into (if anyone has a better solution let me know) is that of a soft systems methodology known as SODA, there are a wide variety of problem structuring methods for complexity, ambiguity and uncertainty (see Rational Analysis for a Problematic World, Ed. Jonathan Roeshead, Wiley). Some of these were developed by engineers in the aircraft industry for coping with complex engineering problems such as Concorde but have found applications in wide and disparate fields.

With regard to issues of consensus and 'buy-in: what exactly are these issues. In the first instance the purpose of risk analysis is to produce an accurate qualitative or quantitative assessment of the risks (a calculus of possibilities or probabilities) for management to make decisions under uncertainty. I contend that attendees of risk workshops may be keen to reach consensus for a wide variety of reasons that may have nothing to do with the importance of the risk assessments and for this reason the risk assessment process fails. Central to "buy-in" is the concept of sharing of responsibility and all that that entails - I addressed this point in the 'blog''.

I would be very happy to receive a point by point critique of my posting and particularly on how facilitators account for or overcome the issues raised.

N.B. Successful Sociopathy:- “Successful Sociopathy and corporations:- Robertson et al (Medical J. of Australia 1996) describe a pathological condition which they call "successful sociopathy". They suggest that many successful businessmen and entrepreneurs show some of the features of this condition.

More on this can be found on Brian Martin's web site at:- http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/health/sociopathy.html#Background

Gavin Lawrence

____________________________________________________

LinkedIn
Date: 11/29/2008

Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information

Group: Global EHS Online Software Solutions

Andy Evans wrote:

Gavin Some of the '19 possible defects' are hazards to the risk assessment process. Perhaps you might care to risk assess each one!

Certainly the potential for 'group-think', 'reluctance to challenge the status quo (or plan)', 'a desire to get the right answer (i.e all is ok) and 'complacency' can be problems.

Looking at your 19 though I struggle to see how a workshop ' damages the camaraderie of the team' so much thatt is better to see the team individually.

Equally you solution seems to me to involve an even greater requirement for a skilled facilitator and probably will involve more facilitator time (as they are working 1-1 with each participant) which would suggest more consultants should be favouring your technique on revenue grounds!

And you state: "There is research to indicate that workshops and group brain-storming sessions do not produce any more creative thinking or better results than that produced by the individuals alone." But what does the research say on the level of agreement with the results? To my mind one reason for a workshop is so that there will be buy-in where action is required to reduce risk.

Andy Evans
Senior Aviation Advisor at AviateQ International

I agree with the blogspot comments, they are risks or defects to a successful risk workshop. As a risk consultant I never approach the process as if the risk workshop is the only tool. Usually I allow all participants to participate in an anonymous survey to identify Strategic, Operational, Compliance and Reporting risks using a very detailed magnitude and vulnerability scale that they helped develop in prior sessions. The Magnitude and Vulenrability (aka Likelhood) scale take a long time to create and I used workshops to do so. In the end if a 5 point scale is being used one can end up with 50 paragraphs of text for the Magnitude scale (5 x 10 factors) and 60 paragraphs of text for the 5 x 12 vulnerability factors. Its a little hard later to dispute the results since the whole group participated in describing the Magnitude and Vulnerability paragraph descriptions, but 110 paragraphs is alot of work. After the Magnitude and Vulnerability scale is created by the client/organization, an anonymous survey can be sent out where they used the rating scale, I usually use SurveyMonkey or some other tool and then summarize results. Only after this is done is a workshop planned.

During the actual workshops usually 20 persons participate from every department in the global company. I believe my risk workshops have been successful because I start out with a statement to the effect that the participants need to be wary of a false sense of precision that some workshop facilitators and participants may believe can occur. I emphasize that a workshop is not precise and therefore is only one tool in the toolkit to prioritize risk response investments by the entity.

Other tools are OECD bribe payer indecies, corrupt practices indecises and software piracy indecis available from OECD which help with country risk assessments which are again only one tool. I used about twenty different sources of data for each top 20 risk statements and about 25 for business process risk by country when I do a global risk assessment. Prior audit experience, change, complexity etc.
Unfortunately I can't give more details because a US process patent is pending.

Bernice Lemaire
President Lemaire Consulting at Lemaire Consulting LLC
bernicecpa@aol.com
____________________________________________________

LinkedIn
Date: 11/28/2008

Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information

Group: Risk Management and Compliance Experts

Roger Miles wrote:

Good blog post - looks like a largely successful trawl for empirical evidence to back the social-behavioural effects modelled by our academic colleagues in Political Science and cognitive psychology. Such real-world manifestations of the effects are sadly familiar but not generally well documented beyond a stack of "anecdotals".

I strongly agree that the absence of cognitive and group-behavioural components from risk assessment (and indeed risk management as a whole... and most particularly from the public-policy end of it) is responsible for much of the markets' current difficulty. In fact I felt strongly enough about this to have spent three years writing it up as a PhD thesis - ironically enough, relating these effects to fragility in the banking system. Hate to have to say to them "I told you so", but...

Interested to hear more about what you plan to do with this.

Regards

Roger

Roger Miles
Risk analysis, Centre for Risk Mgmt.
roger.t.miles@kcl.ac.uk
_________________________________________________

LinkedIn
Date: 11/27/2008

Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information

Group: Risk Management Executives & Professionals

Franco Oboni wrote:

Dear Gavin,

Add me to the list of people who agree on the 18 points.
Ar Riskope we are totally against workshops, but unfortunately we get sometimes forced into them by clients.
We have expressed this view in our book, suggesting that one on one interviews are fare more constructive than workshop, defined an interview format, and a set of rules to determine probabilities.
To complete this discussion I would like to add that we generally develop tolerability criteria simultanously with the risk assessment, there too using specific methodologies and applications.
Visit our web iste to know more http://www.riskope.com

Let me know if you would like to join my network on LinkedIN.
Thank you for the list of 18 points. I will certainly use it, and reference you!

Bye
Franco
Franco Oboni
Oboni Riskope Associates Inc.,
foboni@riskope.com

_________________________________________________