David Harris
Student at West Virginia University
dharris1@mix.wvu.edu
I am adding a few texts to help out for those who are interested in reading about this. I used to have an entire library of books on the topic, but they went with the job and now I only keep a few.
One of the better books is "Overcoming Organizational Defenses" by Chris Argyris at Harvard University. There are also a series of good dissertations run through the Center for Entrepreneurial Studies and Development through the Industrial Engineering Department at West Virginia University on risk based management of large scale projects and risk based scheduling. Last time I heard they were developing software to schedule projects based on risk.
I will look for a good paper on the bathtub curve. There was an excellent one in either IEEE's primary journal or in the management SIG journal.
The observations on equilibrium contracts comes out of basic economics, however there have been psychologists who have been very successfully using this observation to transform groups. If you can find the marginal actors then you have found the point where leverage can be applied to change the whole system. Again, there is literature but when I went to get a PhD I gave up a wealth of written knowledge on shelves at my prior employer.
Finally, there is of course the wealth of writing that comes from W Edwards Deming and the quality movement that followed. In particular the structured programs like Six Sigma or the one at Carnegie Mellon's Software Engineering Institute the Capability Maturity Model are more valuable as teaching tools than as programs. (I am sure they would strongly disagree). The reason I say that is that a good quality program has to be executable by management and most of these programs fail because the changes are too large for senior staff to manage. Such programs are supposed to change how managers manage not how workers work. By changing how managers manage, workers change their work. That is harder to do than it sounds. We behave as we do because we are satisfied and avoid contrary behaviors. If the behaviors are unsuccessful that does not mean we will change, it just means we have unsuccessful behaviors.
_________________________________________________
Monday 15 December 2008
Thursday 11 December 2008
From: Matthew Leitch [mailto:m.leitch1@ntlworld.com]
Sent: 11 December 2008 16:08
To: Gavin Lawrence
Subject: Re: [riskanal] Pertaining to Risk based Cost Contingency
An excellent post. In addition to your sample of workshop problems I would add that I find it deeply depressing to see good ideas I have worked out (at considerable personal effort) misquoted on a flipchart, "mixed and mangled into useless oblivion" along with the dross. The quote is from my book, or at least a draft of it.
The tendency towards seeking consensus might be reduced slightly by saying early on that disagreement will be taken as a signal of uncertainty and uncertainties will be noted and responded to, not squeezed out in an attempt to reach agreement. That would go along with trying to get a view from everyone who is informed enough to hold on on an issue. Risk 'ratings' are estimates and some are supported by more evidence than others. Disagreement indicates that the ratings are likely to be flaky and we should think about finding out more.
Prediction markets could be a better way to get the inputs of lots of people on uncertain quantities. I suspect the software already exists and some clever schemes have been worked out so that even if you need lots of probabilities you can still make it work.
Matthew
_________________________________________________
LinkedIn
Date: 12/09/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Enterprise Risk Management Association
Rakesh Dighe wrote:
Hi
I have used Risk Workshops for over 12 years in an Oil Company setting. There are some important pre requisites to obtaining a successful outcome e.g.
Prepare a strawman in advance using a Subject Matter Expert
Lay down Big Rules for the workshop
Define clearly the risk scoring criteria
One single person to document output
Not a good idea in some cultures to have the Big Boss present at the workshop as it stifles discussion or aggressively leads it in a certain dominant direction
Rgds
Rakesh
CEO
Risk Quotient
_________________________________________________
Sent: 11 December 2008 16:08
To: Gavin Lawrence
Subject: Re: [riskanal] Pertaining to Risk based Cost Contingency
An excellent post. In addition to your sample of workshop problems I would add that I find it deeply depressing to see good ideas I have worked out (at considerable personal effort) misquoted on a flipchart, "mixed and mangled into useless oblivion" along with the dross. The quote is from my book, or at least a draft of it.
The tendency towards seeking consensus might be reduced slightly by saying early on that disagreement will be taken as a signal of uncertainty and uncertainties will be noted and responded to, not squeezed out in an attempt to reach agreement. That would go along with trying to get a view from everyone who is informed enough to hold on on an issue. Risk 'ratings' are estimates and some are supported by more evidence than others. Disagreement indicates that the ratings are likely to be flaky and we should think about finding out more.
Prediction markets could be a better way to get the inputs of lots of people on uncertain quantities. I suspect the software already exists and some clever schemes have been worked out so that even if you need lots of probabilities you can still make it work.
Matthew
_________________________________________________
Date: 12/09/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Enterprise Risk Management Association
Rakesh Dighe wrote:
Hi
I have used Risk Workshops for over 12 years in an Oil Company setting. There are some important pre requisites to obtaining a successful outcome e.g.
Prepare a strawman in advance using a Subject Matter Expert
Lay down Big Rules for the workshop
Define clearly the risk scoring criteria
One single person to document output
Not a good idea in some cultures to have the Big Boss present at the workshop as it stifles discussion or aggressively leads it in a certain dominant direction
Rgds
Rakesh
CEO
Risk Quotient
_________________________________________________
Monday 8 December 2008
November 28, 2008
I'm impressed with your proposals on why risk workshops may be defective. I would add the obvious point that risk management is more science and less subjective than most other management disciplines. Thus, methodology becomes a matter of importance to the analysis process. And of course, with methodology come the issues of reliability and validity. All of these concepts are unwieldly, and so many learners lose interest in the presentation points of the given workshop. I'm still wrestling with this myself in front of my learners, but it does seem to detract from workshop success at times, depending on the participants and their backgrounds. Good luck!
William McKibbin
Consulting Financial Engineer & Risk Analyst
wjmc@mckibbinusa.com
____________________________________________________
I'm impressed with your proposals on why risk workshops may be defective. I would add the obvious point that risk management is more science and less subjective than most other management disciplines. Thus, methodology becomes a matter of importance to the analysis process. And of course, with methodology come the issues of reliability and validity. All of these concepts are unwieldly, and so many learners lose interest in the presentation points of the given workshop. I'm still wrestling with this myself in front of my learners, but it does seem to detract from workshop success at times, depending on the participants and their backgrounds. Good luck!
William McKibbin
Consulting Financial Engineer & Risk Analyst
wjmc@mckibbinusa.com
____________________________________________________
Sunday 7 December 2008
My view is that workshops are great to bring out ideas about risk issues. However, they are hopeless at getting quantitative info. For that, I believe the 1 to 1 interview is better, and then compare estimates. Delphi method suck.
Also a workshop can be a place where people don’t really speak their mind, but will do (more often) if the 1 to 1 is anonymous.
Best wishes
David Vose, Senior partner, Vose Consulting Group.
david@voseconsulting.com
____________________________________________________
Also a workshop can be a place where people don’t really speak their mind, but will do (more often) if the 1 to 1 is anonymous.
Best wishes
David Vose, Senior partner, Vose Consulting Group.
david@voseconsulting.com
____________________________________________________
While the specifics of risk identification are tenuous, the more important element is what management does once a risk is observed. Mitigating or minimizing risk is a dynamic of business life. Yet, in the present economy, and the prior few years, the facts indicate that beneficial, post-recognition activity is non-existent.
I suggest that the risk training should also include some discussion and elevation of decision processes and their down-stream impacts. This would be analogous to both designing fire exits in an office building (risk assessment) and then actually constructing them for use (decision processes).
Greg Chenevert,
Independent Business Consultant, Manchester NH, USA
____________________________________________________
I suggest that the risk training should also include some discussion and elevation of decision processes and their down-stream impacts. This would be analogous to both designing fire exits in an office building (risk assessment) and then actually constructing them for use (decision processes).
Greg Chenevert,
Independent Business Consultant, Manchester NH, USA
____________________________________________________
Tuesday 2 December 2008
RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
From:Larry Cosgrove
Date:November 27, 2008
To: Gavin Lawrence
Group:Catastrophe Risk Modeling Group
Status:Pending
Gavin
I suspect that many of your comments are dead on. I do know that the "people translation" to those not familiar with numerical modeling and statistics is the critical issue here. If you know information to be incorrect, it is best to stand up and make the point. Remember that to those making decisions, sometimes the terminology (especially acronyms) and methodology can prove to be very boring when placed in an open discussion.
In weather prediction methods, I find it best to act as the "translator" while doing my best to overcome the perceived "risk of ridicule". For instance, if the numerical models point to a cold period and vendor forecasts are far warmer and/or calmer, I always provide pictorial proof of what I am describing.
So if I were to give advice for increasing the value or a risk workshop, I would offer these suggestions:
1) Keep the discussion lively; use expressions to go along with the data and engage questions or comments after your initial arguments
2) Graphics that exclaim will do the job far better than charts that induce sleep
3) Always end with a firm conclusion, not a nebulous correlation
Best Regards,
Larry Cosgrove
Chief Meteorologist at Avant Capital Management/Irish Exchange LP
Owner at WEATHERAmerica
Chief Meteorologist at WWJM-FM
____________________________________________________
From:Larry Cosgrove
Date:November 27, 2008
To: Gavin Lawrence
Group:Catastrophe Risk Modeling Group
Status:Pending
Gavin
I suspect that many of your comments are dead on. I do know that the "people translation" to those not familiar with numerical modeling and statistics is the critical issue here. If you know information to be incorrect, it is best to stand up and make the point. Remember that to those making decisions, sometimes the terminology (especially acronyms) and methodology can prove to be very boring when placed in an open discussion.
In weather prediction methods, I find it best to act as the "translator" while doing my best to overcome the perceived "risk of ridicule". For instance, if the numerical models point to a cold period and vendor forecasts are far warmer and/or calmer, I always provide pictorial proof of what I am describing.
So if I were to give advice for increasing the value or a risk workshop, I would offer these suggestions:
1) Keep the discussion lively; use expressions to go along with the data and engage questions or comments after your initial arguments
2) Graphics that exclaim will do the job far better than charts that induce sleep
3) Always end with a firm conclusion, not a nebulous correlation
Best Regards,
Larry Cosgrove
Chief Meteorologist at Avant Capital Management/Irish Exchange LP
Owner at WEATHERAmerica
Chief Meteorologist at WWJM-FM
____________________________________________________
RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
My post may appear critical and I have never been to the UK or used the risk groups there, but the failures are the failures to engineer a poorly controlled and undesigned device (humans and human groups) into the process. No one would design humans as we really exist. Human groups and humans as individuals would be, if we had been engineered, an embarrasment to the designer and never make it to market. Rather than fail at an exponential rate, like all intelligently designed systems, we experience "bath-tub" curve failures. Groups do not like to design around the "bath-tub" curve because we are all intelligent, caring, ethical and well meaning. Such a thing would imply something might be wrong with "us," and that is very threatening. If something is wrong with "us" we might lose our jobs. Being human is very difficult as it is, failing to design around our in built failures is process failure. Facillitators and managers, being human, share these faults. Process should work around this unforgivable fault in managers and facillitators so that the process is better than human, even if the defective machines implementing it are human.
____________________________________________________
From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
My post may appear critical and I have never been to the UK or used the risk groups there, but the failures are the failures to engineer a poorly controlled and undesigned device (humans and human groups) into the process. No one would design humans as we really exist. Human groups and humans as individuals would be, if we had been engineered, an embarrasment to the designer and never make it to market. Rather than fail at an exponential rate, like all intelligently designed systems, we experience "bath-tub" curve failures. Groups do not like to design around the "bath-tub" curve because we are all intelligent, caring, ethical and well meaning. Such a thing would imply something might be wrong with "us," and that is very threatening. If something is wrong with "us" we might lose our jobs. Being human is very difficult as it is, failing to design around our in built failures is process failure. Facillitators and managers, being human, share these faults. Process should work around this unforgivable fault in managers and facillitators so that the process is better than human, even if the defective machines implementing it are human.
____________________________________________________
From:David Harris
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
I have looked at this and all of these things are well understood social phenomenon that a qualified facillitator should be able to avoid or at least should plan for. I have found that risk based scheduling and risk based PERT systems work best if facillitated well from the beginning. Rather than seek to minimize expected costs they seek to minimize variance. The methods are nice because they can be rank based or subjective, as long as raters are consistent with themselves. The mechanism kills projects early by forcing failure early rather than delaying an inevitable failure late. Incentive neutral expected costs are higher because you are not minimizing against expectation, but rather against expected or in some cases median risk, but it is rare that the cost difference is large. Indeed a project killed early is far less costly than one that is permitted to continue so people keep their jobs. It puts the incentives on the table up front. Project planning is NEVER incentive neutral. One real risk is "who is going to get fired for telling the truth." That sets up powerful incentives to bias estimates, "rework" solutions and dig for further capital to keep the process running.
Some of your problems are not "problems," but rather incentive failures. Consider your post:
v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.
There is substantial research that shows that groups only learn through dissent. Overcoming organizational defenses and the associated incentives is the true problem and must be addressed prior to any risk workshop. If the facillitator has not addressed "the unspoken," the group process itself is a danger to the risk management process. Your list can be overcome by good process. Your list is a description of process failure, but not inevitable process failure. If an element of a process has known faults, like over valuing extreme outcomes, then the process should include that knowledge. Failure to include such knowledge is a failure to learn and implement learning.
I am also extremely aware that this is very hard to do. Group equilibriums are incentive equilibriums and these equilibriums are very powerful. Only the marginal decision makers can move them, which may not be members of the management chain you are working with and may not be members of management. Finding these leverage points requires substantial work that would appear to the customer as non-work because you are looking for which social contract(s) is driving all other social contracts, which is both threatening and does not directly appear to bear on the "work at hand."
All of this gets lost in discussions of "misaligned" incentives. Incentives are never misaligned, the subjective goals of the marginal deciders will in expectation be reached, or everyone will go down. The problem isn't "misalignment," but rather poor contracting system wide and system controls sitting somewhere other than in the self-interest of the principals who made themselves "non-marginal."
I disagree that "no facillitator, however good, can resolve," because that is more than a strong statement of certain process failure, it is a statement that learning is impossible. In some groups and work systems this is very true, but only because people are paying them to not learn. If you pay people to be stupid they will gladly be stupid for you.
I am concerned about the emphasis on "cognitive maps." They are tools and very good tools, but you should always question the assumptions behind a tool before using it. In the case of physical tools, like a Phillip's head screw driver, this is easy. Does your problem contain a screw with a Phillip's head? In the case of social tools whether individual tools like cognitive maps or whole systems of tools like Six Sigma, they contain assumptions that must be met in whole. Are they met?
____________________________________________________
Date:December 2, 2008
To: Gavin Lawrence
Group:Risk Economics
Status:Pending
I have looked at this and all of these things are well understood social phenomenon that a qualified facillitator should be able to avoid or at least should plan for. I have found that risk based scheduling and risk based PERT systems work best if facillitated well from the beginning. Rather than seek to minimize expected costs they seek to minimize variance. The methods are nice because they can be rank based or subjective, as long as raters are consistent with themselves. The mechanism kills projects early by forcing failure early rather than delaying an inevitable failure late. Incentive neutral expected costs are higher because you are not minimizing against expectation, but rather against expected or in some cases median risk, but it is rare that the cost difference is large. Indeed a project killed early is far less costly than one that is permitted to continue so people keep their jobs. It puts the incentives on the table up front. Project planning is NEVER incentive neutral. One real risk is "who is going to get fired for telling the truth." That sets up powerful incentives to bias estimates, "rework" solutions and dig for further capital to keep the process running.
Some of your problems are not "problems," but rather incentive failures. Consider your post:
v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.
There is substantial research that shows that groups only learn through dissent. Overcoming organizational defenses and the associated incentives is the true problem and must be addressed prior to any risk workshop. If the facillitator has not addressed "the unspoken," the group process itself is a danger to the risk management process. Your list can be overcome by good process. Your list is a description of process failure, but not inevitable process failure. If an element of a process has known faults, like over valuing extreme outcomes, then the process should include that knowledge. Failure to include such knowledge is a failure to learn and implement learning.
I am also extremely aware that this is very hard to do. Group equilibriums are incentive equilibriums and these equilibriums are very powerful. Only the marginal decision makers can move them, which may not be members of the management chain you are working with and may not be members of management. Finding these leverage points requires substantial work that would appear to the customer as non-work because you are looking for which social contract(s) is driving all other social contracts, which is both threatening and does not directly appear to bear on the "work at hand."
All of this gets lost in discussions of "misaligned" incentives. Incentives are never misaligned, the subjective goals of the marginal deciders will in expectation be reached, or everyone will go down. The problem isn't "misalignment," but rather poor contracting system wide and system controls sitting somewhere other than in the self-interest of the principals who made themselves "non-marginal."
I disagree that "no facillitator, however good, can resolve," because that is more than a strong statement of certain process failure, it is a statement that learning is impossible. In some groups and work systems this is very true, but only because people are paying them to not learn. If you pay people to be stupid they will gladly be stupid for you.
I am concerned about the emphasis on "cognitive maps." They are tools and very good tools, but you should always question the assumptions behind a tool before using it. In the case of physical tools, like a Phillip's head screw driver, this is easy. Does your problem contain a screw with a Phillip's head? In the case of social tools whether individual tools like cognitive maps or whole systems of tools like Six Sigma, they contain assumptions that must be met in whole. Are they met?
____________________________________________________
The risk workshop methodology is deeply entrenched in the UK risk management process and has been for many years, nowhere more so than in project risk and safety risk. If you examine the international use of risk workshops you will note that they are far less and even minimally used as a means of eliciting risk knowledge.
An examination of risk British literature over the years has not revealed any critical analysis of the performance of risk workshops in delivering accurate risk quantification (an examination of major UK projects from infrastructure to IT will demonstrate the gross cost, performance and schedule deficiencies).
Careers are built on facilitation of risk workshops and consultancies charge substantial fees for undertaking them. I have sat in on many risk workshops and discussed the results with many risk managers who expressed concern and dissatisfaction with the results.
A critical appraisal is long overdue on the effectiveness of the risk workshop process and to this effect I drew up the list of points given in the 'blog' post. All these points are logically obvious from allied research in the socio-psychological fields. One particular area I haven't addressed is that of "Successful sociopathy" which has significant import to risk analysis e.g. Enron, and may be the contributing factor in the 'over-confidence' identified by Bent Flyvberg (Megaprojects and Risk) also cf. Network Rail West Coast Mainline upgrade, NHS Data base, Battlespace, CSA, Olympic Games 2012 etc. etc..
A number of risk managers have suggested to me that the camaraderie and "group hug" are important aspects of the risk workshop. It is my contention that this is total nonsense. If you want to build team spirit go to the pub or go paint-balling. A risk workshop has only one function and that is to deliver identification of risks and accurate estimates of their probability and impact. If it fails on this single measure it was a waste of time and money.
The solution I recommended but am not locked into (if anyone has a better solution let me know) is that of a soft systems methodology known as SODA, there are a wide variety of problem structuring methods for complexity, ambiguity and uncertainty (see Rational Analysis for a Problematic World, Ed. Jonathan Roeshead, Wiley). Some of these were developed by engineers in the aircraft industry for coping with complex engineering problems such as Concorde but have found applications in wide and disparate fields.
With regard to issues of consensus and 'buy-in: what exactly are these issues. In the first instance the purpose of risk analysis is to produce an accurate qualitative or quantitative assessment of the risks (a calculus of possibilities or probabilities) for management to make decisions under uncertainty. I contend that attendees of risk workshops may be keen to reach consensus for a wide variety of reasons that may have nothing to do with the importance of the risk assessments and for this reason the risk assessment process fails. Central to "buy-in" is the concept of sharing of responsibility and all that that entails - I addressed this point in the 'blog''.
I would be very happy to receive a point by point critique of my posting and particularly on how facilitators account for or overcome the issues raised.
N.B. Successful Sociopathy:- “Successful Sociopathy and corporations:- Robertson et al (Medical J. of Australia 1996) describe a pathological condition which they call "successful sociopathy". They suggest that many successful businessmen and entrepreneurs show some of the features of this condition.
More on this can be found on Brian Martin's web site at:- http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/health/sociopathy.html#Background
Gavin Lawrence
____________________________________________________
An examination of risk British literature over the years has not revealed any critical analysis of the performance of risk workshops in delivering accurate risk quantification (an examination of major UK projects from infrastructure to IT will demonstrate the gross cost, performance and schedule deficiencies).
Careers are built on facilitation of risk workshops and consultancies charge substantial fees for undertaking them. I have sat in on many risk workshops and discussed the results with many risk managers who expressed concern and dissatisfaction with the results.
A critical appraisal is long overdue on the effectiveness of the risk workshop process and to this effect I drew up the list of points given in the 'blog' post. All these points are logically obvious from allied research in the socio-psychological fields. One particular area I haven't addressed is that of "Successful sociopathy" which has significant import to risk analysis e.g. Enron, and may be the contributing factor in the 'over-confidence' identified by Bent Flyvberg (Megaprojects and Risk) also cf. Network Rail West Coast Mainline upgrade, NHS Data base, Battlespace, CSA, Olympic Games 2012 etc. etc..
A number of risk managers have suggested to me that the camaraderie and "group hug" are important aspects of the risk workshop. It is my contention that this is total nonsense. If you want to build team spirit go to the pub or go paint-balling. A risk workshop has only one function and that is to deliver identification of risks and accurate estimates of their probability and impact. If it fails on this single measure it was a waste of time and money.
The solution I recommended but am not locked into (if anyone has a better solution let me know) is that of a soft systems methodology known as SODA, there are a wide variety of problem structuring methods for complexity, ambiguity and uncertainty (see Rational Analysis for a Problematic World, Ed. Jonathan Roeshead, Wiley). Some of these were developed by engineers in the aircraft industry for coping with complex engineering problems such as Concorde but have found applications in wide and disparate fields.
With regard to issues of consensus and 'buy-in: what exactly are these issues. In the first instance the purpose of risk analysis is to produce an accurate qualitative or quantitative assessment of the risks (a calculus of possibilities or probabilities) for management to make decisions under uncertainty. I contend that attendees of risk workshops may be keen to reach consensus for a wide variety of reasons that may have nothing to do with the importance of the risk assessments and for this reason the risk assessment process fails. Central to "buy-in" is the concept of sharing of responsibility and all that that entails - I addressed this point in the 'blog''.
I would be very happy to receive a point by point critique of my posting and particularly on how facilitators account for or overcome the issues raised.
N.B. Successful Sociopathy:- “Successful Sociopathy and corporations:- Robertson et al (Medical J. of Australia 1996) describe a pathological condition which they call "successful sociopathy". They suggest that many successful businessmen and entrepreneurs show some of the features of this condition.
More on this can be found on Brian Martin's web site at:- http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/health/sociopathy.html#Background
Gavin Lawrence
____________________________________________________
LinkedIn
Date: 11/29/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Global EHS Online Software Solutions
Andy Evans wrote:
Gavin Some of the '19 possible defects' are hazards to the risk assessment process. Perhaps you might care to risk assess each one!
Certainly the potential for 'group-think', 'reluctance to challenge the status quo (or plan)', 'a desire to get the right answer (i.e all is ok) and 'complacency' can be problems.
Looking at your 19 though I struggle to see how a workshop ' damages the camaraderie of the team' so much thatt is better to see the team individually.
Equally you solution seems to me to involve an even greater requirement for a skilled facilitator and probably will involve more facilitator time (as they are working 1-1 with each participant) which would suggest more consultants should be favouring your technique on revenue grounds!
And you state: "There is research to indicate that workshops and group brain-storming sessions do not produce any more creative thinking or better results than that produced by the individuals alone." But what does the research say on the level of agreement with the results? To my mind one reason for a workshop is so that there will be buy-in where action is required to reduce risk.
Andy Evans
Senior Aviation Advisor at AviateQ International
Date: 11/29/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Global EHS Online Software Solutions
Andy Evans wrote:
Gavin Some of the '19 possible defects' are hazards to the risk assessment process. Perhaps you might care to risk assess each one!
Certainly the potential for 'group-think', 'reluctance to challenge the status quo (or plan)', 'a desire to get the right answer (i.e all is ok) and 'complacency' can be problems.
Looking at your 19 though I struggle to see how a workshop ' damages the camaraderie of the team' so much thatt is better to see the team individually.
Equally you solution seems to me to involve an even greater requirement for a skilled facilitator and probably will involve more facilitator time (as they are working 1-1 with each participant) which would suggest more consultants should be favouring your technique on revenue grounds!
And you state: "There is research to indicate that workshops and group brain-storming sessions do not produce any more creative thinking or better results than that produced by the individuals alone." But what does the research say on the level of agreement with the results? To my mind one reason for a workshop is so that there will be buy-in where action is required to reduce risk.
Andy Evans
Senior Aviation Advisor at AviateQ International
I agree with the blogspot comments, they are risks or defects to a successful risk workshop. As a risk consultant I never approach the process as if the risk workshop is the only tool. Usually I allow all participants to participate in an anonymous survey to identify Strategic, Operational, Compliance and Reporting risks using a very detailed magnitude and vulnerability scale that they helped develop in prior sessions. The Magnitude and Vulenrability (aka Likelhood) scale take a long time to create and I used workshops to do so. In the end if a 5 point scale is being used one can end up with 50 paragraphs of text for the Magnitude scale (5 x 10 factors) and 60 paragraphs of text for the 5 x 12 vulnerability factors. Its a little hard later to dispute the results since the whole group participated in describing the Magnitude and Vulnerability paragraph descriptions, but 110 paragraphs is alot of work. After the Magnitude and Vulnerability scale is created by the client/organization, an anonymous survey can be sent out where they used the rating scale, I usually use SurveyMonkey or some other tool and then summarize results. Only after this is done is a workshop planned.
During the actual workshops usually 20 persons participate from every department in the global company. I believe my risk workshops have been successful because I start out with a statement to the effect that the participants need to be wary of a false sense of precision that some workshop facilitators and participants may believe can occur. I emphasize that a workshop is not precise and therefore is only one tool in the toolkit to prioritize risk response investments by the entity.
Other tools are OECD bribe payer indecies, corrupt practices indecises and software piracy indecis available from OECD which help with country risk assessments which are again only one tool. I used about twenty different sources of data for each top 20 risk statements and about 25 for business process risk by country when I do a global risk assessment. Prior audit experience, change, complexity etc.
Unfortunately I can't give more details because a US process patent is pending.
Bernice Lemaire
President Lemaire Consulting at Lemaire Consulting LLC
bernicecpa@aol.com
____________________________________________________
During the actual workshops usually 20 persons participate from every department in the global company. I believe my risk workshops have been successful because I start out with a statement to the effect that the participants need to be wary of a false sense of precision that some workshop facilitators and participants may believe can occur. I emphasize that a workshop is not precise and therefore is only one tool in the toolkit to prioritize risk response investments by the entity.
Other tools are OECD bribe payer indecies, corrupt practices indecises and software piracy indecis available from OECD which help with country risk assessments which are again only one tool. I used about twenty different sources of data for each top 20 risk statements and about 25 for business process risk by country when I do a global risk assessment. Prior audit experience, change, complexity etc.
Unfortunately I can't give more details because a US process patent is pending.
Bernice Lemaire
President Lemaire Consulting at Lemaire Consulting LLC
bernicecpa@aol.com
____________________________________________________
LinkedIn
Date: 11/28/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Risk Management and Compliance Experts
Roger Miles wrote:
Good blog post - looks like a largely successful trawl for empirical evidence to back the social-behavioural effects modelled by our academic colleagues in Political Science and cognitive psychology. Such real-world manifestations of the effects are sadly familiar but not generally well documented beyond a stack of "anecdotals".
I strongly agree that the absence of cognitive and group-behavioural components from risk assessment (and indeed risk management as a whole... and most particularly from the public-policy end of it) is responsible for much of the markets' current difficulty. In fact I felt strongly enough about this to have spent three years writing it up as a PhD thesis - ironically enough, relating these effects to fragility in the banking system. Hate to have to say to them "I told you so", but...
Interested to hear more about what you plan to do with this.
Regards
Roger
Roger Miles
Risk analysis, Centre for Risk Mgmt.
roger.t.miles@kcl.ac.uk
_________________________________________________
Date: 11/28/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Risk Management and Compliance Experts
Roger Miles wrote:
Good blog post - looks like a largely successful trawl for empirical evidence to back the social-behavioural effects modelled by our academic colleagues in Political Science and cognitive psychology. Such real-world manifestations of the effects are sadly familiar but not generally well documented beyond a stack of "anecdotals".
I strongly agree that the absence of cognitive and group-behavioural components from risk assessment (and indeed risk management as a whole... and most particularly from the public-policy end of it) is responsible for much of the markets' current difficulty. In fact I felt strongly enough about this to have spent three years writing it up as a PhD thesis - ironically enough, relating these effects to fragility in the banking system. Hate to have to say to them "I told you so", but...
Interested to hear more about what you plan to do with this.
Regards
Roger
Roger Miles
Risk analysis, Centre for Risk Mgmt.
roger.t.miles@kcl.ac.uk
_________________________________________________
LinkedIn
Date: 11/27/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Risk Management Executives & Professionals
Franco Oboni wrote:
Dear Gavin,
Add me to the list of people who agree on the 18 points.
Ar Riskope we are totally against workshops, but unfortunately we get sometimes forced into them by clients.
We have expressed this view in our book, suggesting that one on one interviews are fare more constructive than workshop, defined an interview format, and a set of rules to determine probabilities.
To complete this discussion I would like to add that we generally develop tolerability criteria simultanously with the risk assessment, there too using specific methodologies and applications.
Visit our web iste to know more http://www.riskope.com
Let me know if you would like to join my network on LinkedIN.
Thank you for the list of 18 points. I will certainly use it, and reference you!
Bye
Franco
Franco Oboni
Oboni Riskope Associates Inc.,
foboni@riskope.com
_________________________________________________
Date: 11/27/2008
Subject RE: Reasons for Risk Workshops failing to deliver accurate quantitative information
Group: Risk Management Executives & Professionals
Franco Oboni wrote:
Dear Gavin,
Add me to the list of people who agree on the 18 points.
Ar Riskope we are totally against workshops, but unfortunately we get sometimes forced into them by clients.
We have expressed this view in our book, suggesting that one on one interviews are fare more constructive than workshop, defined an interview format, and a set of rules to determine probabilities.
To complete this discussion I would like to add that we generally develop tolerability criteria simultanously with the risk assessment, there too using specific methodologies and applications.
Visit our web iste to know more http://www.riskope.com
Let me know if you would like to join my network on LinkedIN.
Thank you for the list of 18 points. I will certainly use it, and reference you!
Bye
Franco
Franco Oboni
Oboni Riskope Associates Inc.,
foboni@riskope.com
_________________________________________________
Thursday 27 November 2008
Reasons for Risk Workshops failing to deliver accurate quantitative information
In the UK risk workshops play a central role in the elicitation of risks and their probabilities and impacts. Below I give 19 reasons for risk workshops failing to deliver accurate and meaningful results. The list is not exhaustive and the reasons are not mutually exclusive. I would welcome some feedback on how these should be best dealt with.
i. Participants believe that their view is 'out on a limb' from the rest of the group.
ii. Participants may believe that they will be subject to ridicule for expressing an alternative view.
iii. Participants may think that others have already said 'it', or thought about it because it seemed so obvious, and that the idea must have been rejected for "good" reasons.
iv. They may have 'trading agreements' with the others in the group that would be broken if they expressed a view that opposes that of their trading partners - to do so would have consequences for support on other issues.
v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.
vi. It damages the camaraderie of the team.
vii. Participants may be frightened of reprisals for expressing a particular view that is thought to be counter to the prevailing view of those in power.
viii. May demonstrate weaknesses or knowledge/experience etc., deficiencies of the participant.
ix. Drawing on research in cognitive sciences on pattern recognition and visual perception, the practice of plotting, on a white-board, of probability (y-axis) and £-impact - log scaled (x-axis) may influence the risk assessments vis a vis relative positioning and density of the plot points for reasons such as symmetry and balance.
x. Major infrastructure cost over-runs are regularly in the range of 25 - 600%, Flyvberg identifies over-optimism as the single greatest because of over-estimation; this may flow from the points listed above.
xi. Group assessments tend to give extreme values greater credence i.e. approaching 0 or 1.
xii. Social norms of the group can discourage the extent to which the thinking of each of individuals in the group is used in the group decision making.
xiii. Existing social processes and social relationships encourage only shallow thinking to surface.
xiv. Group decision making often results in higher risk decisions being taken because of the shared responsibility and hence, less liability, than that of the individual decision maker.
xv. There is research to indicate that workshops and group brain-storming sessions do not produce any more creative thinking or better results than that produced by the individuals alone.
xvi. Participant selection is a key issue to be explored further in the context of management v. Front-line executive. Also how expert are the experts?
xvii. Over-emphasis placed on the value and out-put of workshops by managers and decision makers.
xviii. Risk consultants over-emphasise the value of risk workshops because they make money from running them!
XiX. Attendees of risk workshops are keen to achieve a consensus, right or wrong, in order to get back to work that they feel is more important i.e. risk workshops are perceived time wasting etc...
I have presented 18 points to a wide variety of risk academics and risk practitioners around the world and none have disputed the issues raised with regard to the 'risk workshop's' deficiencies. In fact most have offered total agreement that these workshops, no matter how well facilitated, do not deliver the quantitative assessments that are comprehensive and accurate enough. Aside from the personal anecdotal evidence there is more than enough research evidence to draw a conclusion that workshops do not deliver.
Assuming that the risk workshop has attendees that have been appropriately selected, something like a dozen key members of staff are taken out of work, effectively for a day - perhaps representing a loss of US$ 15,000 add to this the facilitators cost, plus everybody’s travel expenses to the workshop – perhaps totalling as much as US$25,000.
I would like to suggest that the most effective approach to resolving the risk workshop deficiencies would be that known as "Cognitve Mapping for Strategic Options and Development Analysis" (SODA). The concept and practice of SODA is extensively described in documents on http://www.banxia.com/ , and In the book, Making Strategy: The Journey of Strategic Management, Eden C and Ackermann F., so I won't dwell on the detail.
The advantage of SODA is multi-fold:- the risk analyst can interview the "experts" individually in their own time, revisit them for additional information. The experts are not, therefore, taken out of their work situation for more than an hour at a time. The individual's risk perceptions are explored in depth without any influence from colleagues. Cognitive maps are prepared of each experts perspectives on risk with regard to the project.
There is no limit to the number of experts that can be interviewed as is the case with attendees of a workshop where 12 is pretty much the maximum.
The benefits:-
1. The individual interview process has removed the points raised in points i to viii and xii.
2. The cognitive mapping process removes issues in points ix and xi.
3. Larger number of interviews reduces selection problems, point xvi.
4. More creative and deeper thinking may be encouraged, points xiii and xv.
I submit that the risk analyst will have achieved far more at this stage using SODA, more efficiently and cost effectively than any risk workshop would have done. Not only will a wider range of problems have been described along with the quantitative assessments but insights in to mitigation strategies and value management issues would be becoming apparent.
If you use the Decision Explorer software package each interviewee's cognitive map can be stored, in addition the interviewee's individual maps can be merged in to one. The aggregation of the interviews cognitive maps is very important:-
Consider the risky events that have been identified and individual assessments of probability P(E) x likely outcome distribution from a wide variety of individuals of varying expertise. How to combine these different risk assessments, I have suggested a method in the Bayesian Statistics Forum; it is also important to check out issues and methodologies cited in Roger Cooke;s book, Experts in Uncertainty (see Recommended Books forum). Certainly from the combined risk weighted estimates of the interviews a comprehensive and detailed risk handling matrix can be developed.
Prioritisation of risky events normally flows from a straight forward cost ranking of the simulated risk register. The development of risk mitigation strategies follows from this. What can a risk workshop contribute to this – answer nothing!
Yet, not only do the aggregated cognitive maps contain the embryonic strategic solutions to risk handling and mitigation and are worthy of further detailed analysis and development, they are also fundamental in the analysis and understanding of complex problems.
It is only after the SODA process has been undertaken should a review 'risk workshop' be held at which all the issues raised can be discussed and analysed.
i. Participants believe that their view is 'out on a limb' from the rest of the group.
ii. Participants may believe that they will be subject to ridicule for expressing an alternative view.
iii. Participants may think that others have already said 'it', or thought about it because it seemed so obvious, and that the idea must have been rejected for "good" reasons.
iv. They may have 'trading agreements' with the others in the group that would be broken if they expressed a view that opposes that of their trading partners - to do so would have consequences for support on other issues.
v. To dissent from the view of the group may put team cohesiveness at risk - threatening established order.
vi. It damages the camaraderie of the team.
vii. Participants may be frightened of reprisals for expressing a particular view that is thought to be counter to the prevailing view of those in power.
viii. May demonstrate weaknesses or knowledge/experience etc., deficiencies of the participant.
ix. Drawing on research in cognitive sciences on pattern recognition and visual perception, the practice of plotting, on a white-board, of probability (y-axis) and £-impact - log scaled (x-axis) may influence the risk assessments vis a vis relative positioning and density of the plot points for reasons such as symmetry and balance.
x. Major infrastructure cost over-runs are regularly in the range of 25 - 600%, Flyvberg identifies over-optimism as the single greatest because of over-estimation; this may flow from the points listed above.
xi. Group assessments tend to give extreme values greater credence i.e. approaching 0 or 1.
xii. Social norms of the group can discourage the extent to which the thinking of each of individuals in the group is used in the group decision making.
xiii. Existing social processes and social relationships encourage only shallow thinking to surface.
xiv. Group decision making often results in higher risk decisions being taken because of the shared responsibility and hence, less liability, than that of the individual decision maker.
xv. There is research to indicate that workshops and group brain-storming sessions do not produce any more creative thinking or better results than that produced by the individuals alone.
xvi. Participant selection is a key issue to be explored further in the context of management v. Front-line executive. Also how expert are the experts?
xvii. Over-emphasis placed on the value and out-put of workshops by managers and decision makers.
xviii. Risk consultants over-emphasise the value of risk workshops because they make money from running them!
XiX. Attendees of risk workshops are keen to achieve a consensus, right or wrong, in order to get back to work that they feel is more important i.e. risk workshops are perceived time wasting etc...
I have presented 18 points to a wide variety of risk academics and risk practitioners around the world and none have disputed the issues raised with regard to the 'risk workshop's' deficiencies. In fact most have offered total agreement that these workshops, no matter how well facilitated, do not deliver the quantitative assessments that are comprehensive and accurate enough. Aside from the personal anecdotal evidence there is more than enough research evidence to draw a conclusion that workshops do not deliver.
Assuming that the risk workshop has attendees that have been appropriately selected, something like a dozen key members of staff are taken out of work, effectively for a day - perhaps representing a loss of US$ 15,000 add to this the facilitators cost, plus everybody’s travel expenses to the workshop – perhaps totalling as much as US$25,000.
I would like to suggest that the most effective approach to resolving the risk workshop deficiencies would be that known as "Cognitve Mapping for Strategic Options and Development Analysis" (SODA). The concept and practice of SODA is extensively described in documents on http://www.banxia.com/ , and In the book, Making Strategy: The Journey of Strategic Management, Eden C and Ackermann F., so I won't dwell on the detail.
The advantage of SODA is multi-fold:- the risk analyst can interview the "experts" individually in their own time, revisit them for additional information. The experts are not, therefore, taken out of their work situation for more than an hour at a time. The individual's risk perceptions are explored in depth without any influence from colleagues. Cognitive maps are prepared of each experts perspectives on risk with regard to the project.
There is no limit to the number of experts that can be interviewed as is the case with attendees of a workshop where 12 is pretty much the maximum.
The benefits:-
1. The individual interview process has removed the points raised in points i to viii and xii.
2. The cognitive mapping process removes issues in points ix and xi.
3. Larger number of interviews reduces selection problems, point xvi.
4. More creative and deeper thinking may be encouraged, points xiii and xv.
I submit that the risk analyst will have achieved far more at this stage using SODA, more efficiently and cost effectively than any risk workshop would have done. Not only will a wider range of problems have been described along with the quantitative assessments but insights in to mitigation strategies and value management issues would be becoming apparent.
If you use the Decision Explorer software package each interviewee's cognitive map can be stored, in addition the interviewee's individual maps can be merged in to one. The aggregation of the interviews cognitive maps is very important:-
Consider the risky events that have been identified and individual assessments of probability P(E) x likely outcome distribution from a wide variety of individuals of varying expertise. How to combine these different risk assessments, I have suggested a method in the Bayesian Statistics Forum; it is also important to check out issues and methodologies cited in Roger Cooke;s book, Experts in Uncertainty (see Recommended Books forum). Certainly from the combined risk weighted estimates of the interviews a comprehensive and detailed risk handling matrix can be developed.
Prioritisation of risky events normally flows from a straight forward cost ranking of the simulated risk register. The development of risk mitigation strategies follows from this. What can a risk workshop contribute to this – answer nothing!
Yet, not only do the aggregated cognitive maps contain the embryonic strategic solutions to risk handling and mitigation and are worthy of further detailed analysis and development, they are also fundamental in the analysis and understanding of complex problems.
It is only after the SODA process has been undertaken should a review 'risk workshop' be held at which all the issues raised can be discussed and analysed.
Subscribe to:
Posts (Atom)
