Multi-Stakeholder Risk Assessment of Socio-Technical Interventions
Citation: R. Clarke and K. Michael, "Multi-Stakeholder Risk Assessment of Socio-Technical Interventions," in IEEE Transactions on Technology and Society, doi: 10.1109/TTS.2026.3703434.
Photo by fauxels
Abstract
IT-enabled interventions into economic and social systems of broad scope and large scale are likely to have substantial impacts, and suffer high failure rates. Conventional approaches to devising and evaluating such interventions are heavily committed to the perspective of the sponsoring organization, with standardized analysis, design, and implementation processes typically disregarding broader stakeholder participation. The interests of other stakeholders are commonly considered only if they are perceived to represent constraints on the achievability of the sponsor’s objectives. This narrows the focus to only those stakeholders that are perceived to have sufficient power, and overlooks other legitimate stakeholders. This article adopts a socio-technical approach to emphasize the importance of the social dimension in all IT-enabled interventions. This leads to the conception of risk assessment (RA) being refined, with the objective of addressing the challenges that arise with large-scale, high-impact socio-technical interventions. A new form of RA is proposed, incorporating the perspectives of multiple stakeholders. It is intended to reduce both harm to stakeholders and the incidence of intervention failure. The Multi-Stakeholder Risk Assessment (MSRA) process is presented, adapting the traditional risk assessment processes in a minimally disruptive manner. An overview of the steps within MSRA is provided, as well as practical guidance regarding its application. Prior exemplars and motivators for the adoption of MSRA are identified, and its efficacy is demonstrated through two example projects conducted by a large-scale government agency.
Open Access
I. INTRODUCTION
IN PERFORMING their functions, organizations initiate interventions into existing social and economic processes. Interventions can be of many kinds, including new or amended legislation, regulatory measures of some other kind, adap tation of institutional or sectoral infrastructure, changes in business processes, and applications of transformative or disruptive technologies. Interventions in large-scale systems are inevitably socio-technical in nature, incorporating social, technical and environmental dimensions, encapsulating people, processes and the policies and procedures that regulate interac tions between humans, machines and other components, such as geographical settings.
During the seven decades since computing began to be applied to data processing, applications of information tech nology (IT) have migrated far beyond the conversion of data into information. Information Systems (IS) draw inferences from clusters of information, to support decision-makers. IS may also make decisions based on information and inferences, support individuals taking actions driven by those decisions, and even directly implement decisions. Systems have also expanded beyond organizational boundaries, linking pairs, chains and networks of organizations, and extending out to individuals. Contemporary IS are capable of substantial impacts and implications, ranging from the highly beneficial to the very harmful, sometimes foreseen and managed, but in many cases unintended, unanticipated, or foreseen but ignored. The growth in scale and scope of IT-enabled systems has been accompanied by greater potential value to at least some participants, but also by considerable increases in the likelihood and seriousness of harm.
To cope with the expanding scale and scope of IT enabled projects, more rapid development techniques have been deployed. These feature even less quality assurance effort than was applied to the simpler systems of the past. The result has been an ongoing, poor record of plannability, of delivery to time and budget, and of reliability, maintainability and adaptability. Academics have studied the reasons under lying project shortfalls, especially outright project failure; but failures persist, and the quality of service of many ongoing systems remains low [1].
The ongoing high rates of failure and quality shortfalls make clear that contemporary approaches are not adequate to address the challenges arising from large-scale chained and networked inter-organizational systems, and extra-organizational systems in which individuals outside the organization are deeply involved. In studies of complex IS, it is important to reflect researcher perspective theory as presented in [2]. This notes the very strong focus within the IS discipline on single perspective research, and very limited reflection of the interests of stakeholders other than the sponsor of the intervention.
With most interventions, one organization is recognizable as the primary driver or sponsoring organization. In the context of interventions in which IS play a significant role, the term system sponsor is appropriate [2, p. 484]. Reduction in the frequency and scale of IS failures depends on the encouragement of system sponsors to recognize the existence of other stakeholders and to enable and encourage IS practi tioners to adopt the perspectives of those other stakeholders when conceiving, designing and implementing IS. A problem that must be confronted is that most evaluation techniques have a very tight focus on the interests of a single stakeholder. The stakeholder whose interests are prioritized is almost always the sponsoring organization [3].
A significant mismatch exists between the current, strong commitment by researchers to system sponsors’ interests, and the economic and social need for research into the impacts and implications of IT-enabled interventions on a great many other parties as well. Researchers need to take account of the accumulated knowledge that points to the need to reflect the interests of all affected parties in system designs.
II. STAKEHOLDER THEORY
The term stakeholders was coined as a counterpoint to shareholders, to bring into focus the interests of parties other than the corporation’s owners [4], [5]. Users of IS have long been recognized as stakeholders, commencing with employees in the 1970s, in the context of intra-organizational systems. With the emergence of inter-organizational systems [6], and then extra-organizational systems [7], IT services extended beyond organizations’ boundaries, and hence many suppliers and customers, both corporate and individual, became users as well.
The notion of stakeholders is broader than just users, however. It comprises not only participants in IS but also “any other individuals, groups or organizations whose actions can influence or be influenced by the development and use of the system whether directly or indirectly” [8, p. 3]. The term usees is usefully descriptive of stakeholders who are not participants in a system, but are influenced by it [7], [9, p. 388], [10], [11], [12]. A comprehensive taxonomy of stakeholders is provided by [13], in the form of an onion ring of three layers (the information system, the containing system and the wider environment), containing 13 classes of roles.
The stakeholder notion has been subjected to further anal ysis and discussion during the four decades since it emerged. One analysis distinguishes stakeholders’ salience based on power, legitimacy and urgency [14]. An approach based on ethics might dictate that legitimacy has primacy. On the other hand, a strong tendency has been evident, in industry and government practice, and in research publications, for attention to be paid only to the interests of those stakeholders that are capable of significantly affecting the success of the project–as perceived by the project sponsor [15]. The interests of the sponsoring organization are treated as paramount, the interests of powerful participants and usees are relegated to the role of constraints on the achievement of the primary organization’s objectives, and the interests of participants and usees that lack power are commonly marginalized or ignored.
Ackermann and Eden [16], for example, recommend that an organization sponsoring an intervention concentrate primarily on players who have high power and high interest. An eye is to be kept open for context-setters (with high power but low interest) and subjects (with low power but high interest), whose potential to oppose may need to be neutralized, but who might be able to be enlisted as allies [18, p. 183]. In that highly-cited article [17], the notion of stakeholder legitimacy is sublimated as interest, and the authors acknowledge that “strategic man agement of stakeholders [is] problematic...because it seems a manipulative– and thus somehow ‘illegitimate’– activity” [18, p. 180].
Similar marginalization of all but the most powerful stake holders is evident in other movements within the project management arena. In regard to the benefits management approach, for example [18], the practical guidance delivered to IS professionals is that the purpose of understanding a stakeholder’s motives and agenda is “to influence the change process positively and to address issues that may be potential barriers to change”. The key players are those with both a high level of power and a high level of interest, but the focus of the project sponsor is on “resolving senior stakeholder issues”, not fulfilling stakeholder needs, even of the key players [19].
Stakeholder theory has been applied and extended in [2], which presents a theory relating to Researcher Perspec tive. Research that is concerned with real-world phenomena involves carefully-conducted observation of some object of study. Generally, the parts of the real world that researchers select as objects of study involve multiple entities, each of which perceives the phenomena in their own way. A Stake holder Perspective is the viewpoint adopted by a stakeholder in a particular activity, reflecting that stakeholder’s perception of phenomena within the relevant context, the stakeholder’s value-set, and the interests that the stakeholder seeks to protect and advance.
When changes occur, each stakeholder observes them from their own particular viewpoint, and seeks to protect and advance their own interests. The actions of, and interactions among, stakeholders may have a significant impact on the outcomes. Hence a researcher who seeks to present credible research, even merely describing phenomena, let alone predict ing or making normative judgements about behavior, needs to achieve a sufficient understanding of the relevant stakeholder perspectives.
The central proposition of researcher perspective theory is that research is seldom conducted in a holist or universalist manner, reflecting the interests of all stakeholders at once. It is very challenging for a researcher to convincingly claim omni cognizance. It is accordingly much more common to adopt the perspective of one party, or the perspectives of a small number of the parties, involved in or affected by the events.
Systematic studies of IS research literature have shown that the large majority of published IS research adopts the perspective of a single stakeholder [2], [20], [21]. The entity whose interests are privileged by the design of any particular research project could be in principle any of the system spon sor, users of or participants in the system, or usees. In practice, however, the evidence makes clear that the large majority of IS research has its focus on the system sponsor’s interests. Researcher perspective theory identifies benefits arising from single-perspective IS research that reflects the interests of a stakeholder other than the system sponsor. Even greater benefits are achievable by dual-perspective research designed to benefit both members of a dyad. Broader, multi-perspective research is challenging, but it is highly desirable, and, in some contexts at least, feasible.
III. OPTIONS FOR REFLECTING MULTIPLE STAKEHOLDERS’ INTERESTS
Misconceived, poorly designed and poorly implemented interventions give rise to high costs, high risk of inadequate performance and failure, and collateral damage. The impacts and implications of these interventions are consequently recog nized as requiring evaluation, preferably at several checkpoints along the way. They are also framed as socio-technical interventions, encompassing technical, social and environmen tal dimensions [22]. Many evaluation techniques exist, with widely varying approaches and foci.
Mainstream project evaluation techniques within organi zations include soft or justificatory forms of Business Case Development (BCD) [23], and more exacting techniques such as Discounted Cash Flow Analysis (DCF) [24], Net Present Value Analysis (NPV) [25], financial sensitivity analysis and financial risk assessment. All of these depend on quantifica tion, and on the expression of costs and benefits in financial terms. Such analyses are generally undertaken from the per spective of a single legal entity or government agency. In the case of interventions in which IT plays a major role, that single entity or agency is typically the system sponsor [2].
Some single-organization assessment techniques extend to factors that cannot be readily reduced to financial values, whose representations are referred to as non-quantifiable fac tors expressed as qualitative data. One such approach is internal Cost-Benefit Analysis (CBA) [26]. Another is Risk Assessment (RA) [27], [28]. All of these techniques are highly organization-centric, and generally reflect interests of some other party only if that stakeholder is perceived by the organization to be sufficiently powerful to be able to affect the organization’s capacity to achieve its objectives.
Some other evaluation techniques adopt a broader frame of reference. Technology Assessment (TA) is concerned with the evaluation of potential impacts and implications of a par ticular technical capability [29], [30]. Environmental Impact Assessment (EIA) [31] evaluates the effects on the physical environment (commonly summarized as air, land and water) of a development project such as a mine, transport infrastruc ture, a manufacturing facility or a campus. Privacy Impact Assessment (PIA) considers effects on individuals’ privacy interests–although often only the privacy interests involved in personal data [32], [33]. Social impact assessment [34] has a broad remit, child rights impact assessment [35] a highly specific focus, and surveillance impact assessment [36], [37] combines technological with psychological and social impact assessment.
It is feasible to apply some form of impact assessment approach to address the concerns of stakeholders gener ally. However, the approach is seldom consistent with any organization’s perceived needs, and is in direct conflict with the conventionally-assumed legal responsibility of Board directors to serve the interests of shareholders. Moreover, impact assessment of all kinds demands considerable depth and breadth of analysis, and hence considerable resources with specialist expertise.
IV. STAKEHOLDER INCLUSIVITY
Contemporary large-scale information systems (LIS) can no longer be considered merely from a technical viewpoint. They are increasingly key elements within broader strate gic interventions into communities, societies, economies or polities. While IT-enabled interventions commonly have a primary driver (sponsor), many stakeholders are involved. Some are participants, conventionally termed users of the system, including the staff of the sponsoring organization and of other organizations with access to the system, and individuals such as service-subscribers. Meanwhile, other enti ties that are not participants are nonetheless affected by it, and are usefully referred to as usees. LIS have substantial impacts on and implications for both users and usees. Those impacts and implications need to be considered during the design and adaptation of the IS.
Project methods may provide for user participation in sys tems analysis and design (e.g. [38]). However, the penetration of effective participatory techniques has been limited, and usees are seldom integrated into project work. Various forms of evaluation are applied to projects, and they may incorporate consideration of stakeholder concerns. Yet, techniques such as business case development, net present value analysis and cost benefit analysis seldom incorporate other stakeholders in any effective manner. Their focus is strongly on the interests of the sponsor of the IT-enabled intervention.
Fig. 1. The conventional security model. Adapted from [48, pp. 547-549]
The purpose of this article is to propose a practicable mechanism whereby the interests of relevant parties can be reflected in the assessment of interventions. None of the mainstream evaluation techniques fulfils that need. This article accordingly considers Risk Assessment (RA), and proposes adaptation of the technique. The paper’s purpose is: To seek a practicable mechanism whereby the interests of relevant parties can be reflected in the risk assessment of socio technical interventions. The aim of the research is to show that considerable benefits can be achieved by extending a well established business process to address the requirements of the multiple stakeholders in impactful socio-technical interven tions. The primary contribution of the paper is to IS practice, through the establishment of an adapted form of business process. This is in accord with the need for enhanced practice impact [39]. In addition, the work contributes to practice relevant IS research, in relation to theory in the areas of project evaluation, stakeholder management, and the conduct of impactful IS projects.
The paper commences by summarizing the established busi ness process of RA. The article then reports on the application of a design research approach, addressing in succession the problem, the objectives, design, demonstration and evaluation of the design’s potential. First, an adapted form of RA is articulated, referred to as Multi-Stakeholder Risk Assessment (MSRA). As preliminary tests of the proposition, exemplars of the technique, and drivers for its adoption, are identified.
The efficacy of the approach is then discussed in the con text of a very large-scale IS failure. The proposed adapted socio-technical artefact is then evaluated by examining its application to a project to implement voice authentication at scale.
A. Conventional Risk Assessment
The term risk refers to the perceived likelihood of harm arising to an asset as a result of a threatening event impinging on a vulnerability. The term security refers to the desirable condition in which harm is prevented or mitigated because threats and vulnerabilities are subject to effective safeguards. The assessment of risk therefore needs to be based on a model of security, and meaningful conversations depend on an established terminology and sufficiently clear definitions.
Figure 1 illustrates the conventional security risk model, situating it within the broader landscape of Governance, Risk, and Compliance (GRC) as well as information assurance and security management [40]. For readers less familiar with the risk assessment domain, this model provides a foundational view of how threats, vulnerabilities, safeguards, and harmful outcomes relate to one another in a structured risk-management process. Governance establishes policies, decision rights, and accountability structures that determine how risk should be understood and managed, while compli ance ensures these processes adhere to regulatory, legal, and standards-based requirements, such as ISO/IEC 27001 [41], NIST CSF [42], COBIT [43], and SABSA [44]. Within this broader context, RA is the analytical mechanism that evaluates whether organizational controls are appropriate, proportional, and effective. It also provides the evidence base for risk treatment decisions and governance reporting.
In terms of information assurance and security management, risk assessment is the bridge between business-level objectives and operational security actions [45]. It ensures that protective measures, i.e., “safeguards”, are justified by demonstrable threats and vulnerabilities, and that they mitigate risks in ways aligned to organizational priorities, stakeholder needs, and asset criticality. Risk assessment is therefore not simply a technical exercise; it is a mechanism for aligning security with business value, trust, resilience, and compliance obligations. Risk Assessment has been in use for a sufficiently long period that it has become the subject of industry standardization (NIST 2012 [46], EC 2016 [47], IEC 31010:2019 [48], ISO 27005:2022 [27] which superseded ISO 27005:2011 [28], pp. 7-17, 33-49].
In the fields of data and IT security, a substantial literature exists, and this provides a useful framework that can be drawn on in other contexts. A depiction of the security model conventional in the data and IT security fields is in Figure 1. Definitions of the terms are provided in [49]. The RA process uses a short sequence of steps to apply the concepts in Figure 1 to identify and understand residual, inadequately-addressed risks, and hence lay the foundation for Risk Management (RM) activities to address them. These steps are embodied within an extended process model presented in the following section. An aspect of particular significance for the analy sis presented in this article is the recognition that Assets, perceptions of value in those Assets, and Harm to them, are relative to the particular Stakeholder whose interests are affected by a Security Incident. In conventional applications of Risk Assessment, however, it is unusual for the interests of any Stakeholder to be considered other than those of the sponsor of the intervention.
The risk process flow begins with a generic threat, which gives rise to a threatening event– an occurrence that can interact with, or be contrived to exploit, a vulnerability. A security incident may then occur, which can lead to harm to assets important to stakeholders. At each stage, safeguards may prevent, deter, detect, or mitigate undesirable outcomes. Like wise, ineffective or absent safeguards, sometimes undermined by countermeasures or conflicting controls, illustrate how risk can materialize when protection is inadequate or poorly managed. Risk identification involves recognizing threats, vul nerabilities, stakeholders, and assets, corresponding to the elements shown on the left side of the figure. Risk assessment evaluates the likelihood that threatening events will interact with vulnerabilities, and the potential consequences (harm). The relationships depicted in the center and right of the figure (e.g., “impinges on,” “results in,” “leads to”) represent these causal pathways foundational to risk assessment. Finally, risk mitigation refers to the selection and implementation of safeguards, highlighted in green, which aim to break or weaken the chain linking threats to harmful outcomes. This may involve prevention, detection, response, or resilience mechanisms. By clarifying how these core risk assessment activities map onto the elements of Figure 1 and how they fit within GRC and information assurance, the figure serves not merely as a depiction of causal relationships but as a conceptual framework for understanding the security risk lifecycle.
V. RESEARCH METHOD
In order to develop and demonstrate an adapted form of conventional risk assessment, the research presented here adopts the design science approach. In this approach the term design is used to denote “the purposeful organization of resources to accomplish a goal” [51]. An amended form of a business process is a socio-technical artefact as understood in this context of design science [52], [53, p. 337]. However, design science has a strong orientation towards the system sponsor perspective. Thus, the opportunity exists to extend conventional risk assessment to support multiple perspectives. This way has already been demonstrated by action research which adopts “the idiographic viewpoint [whereby] any mean ingful investigation must consider the frame of reference and underlying social values of the subjects” [54]. The accumu lated wealth of socio-technical thinking [22], [55], [56] can be applied to articulate what might be usefully described as participative design science. Following Hevner et al.’s Guideline 3 [51] by asking ‘What is a feasible and effective process for the design of a particular system or category of systems?’ is one line of inquiry. Another deeper line of inquiry is to ask ‘What is a feasible and effective process for reflecting the perspectives of all parties in the design of a particular system or category of systems?’ [2, p. 492].
A similar line of development has occurred in the field of ‘design thinking’. This was coopted from architecture into management, and typified as “a discipline that uses the designer’s sensibility and methods to match people’s needs with what is technologically feasible and what a viable busi ness strategy can convert into customer value and market opportunity” [57, p.2]. Subsequentlly, less emphasis has been placed on the designer, and more on encompassing the many players in the contexts within which a design emerges, and the dynamics of interactions among them [58].
The process adopted in this paper reflects Peffers’ [59] Design Science Research Methodology (DSRM). This dis tinguishes steps for problem identification and definition of objectives, followed by design and development, demonstra tion and evaluation. The remainder of this article implements a process modelled on DSRM. Section IV addresses the early steps of the process, articulating the problem definition and objectives, and proposing an amended business process. Section V demonstrates the adapted process’s practicability, drivers for its adoption, and its efficacy. Section VI evaluates the approach by applying it to a project to implement voice authentication in a very large public sector context.
VI. MULTI-STAKEHOLDER RISK ASSESSMENT
Fig. 2. An Open Socio-Technical System incorporating External Users and Usees. Adapted version of [68, p. 25, Fig. 2].
This section defines the proposed business process, reports the findings of a supplementary literature review undertaken in relation to broader conception and interpretation of RA, then further articulates the features of the MSRA process.
A. Definition
The RA technique was conceived to serve the interests of the organization that conducts the assessment, or that commissions the intended intervention. Standards documents and guidelines reflect those origins. On the other hand, the authors have argued, in the specific context of Artificial Intelligence (AI) [60, p. 413], [61], that:
(a) The responsible application of AI is only possible if stakeholder analysis is undertaken not only to identify the categories of entities that may be affected by the project, but also to gain insight into those entities’ needs and interests;
(b) Risk Assessment processes that reflect the interests of stakeholders need to be broader than those commonly undertaken within organizations; and
(c) The responsible application of AI depends on Risk Assessment processes being conducted from the perspective of each stakeholder group, to complement that undertaken from the organization’s perspective.
In the present paper, the authors argue that those assertions apply not only to applications of AI, but to interventions gen erally, particularly those that have substantial IT-based compo nents. Such interventions are intrinsically socio-technical [62], [63], [64]). Socio-technical systems theory and related design methods are built on open systems thinking [65] and acknowl edge that systems comprise technical components working in combination with social and/or human elements [22]. Analyses must therefore consider the interactions between the social, technical and environmental dimensions [66], [67]. Relevant elements within these dimensions include amendments to statutes, codes and contracts, adjustments to business pro cesses that cross organizational boundaries, and the application of change management techniques to cause users who are affected by it to adjust their behaviors. Bostrom and Heinen [68] provided conceptual guidance on how to integrate the technical and social dimensions. However, that guidance pre dates the contemporary dominance of inter-organizational and extra-organizational IS. In Figure 2, we propose an enhanced model that reflects the users internal to the IS but external to any of the organizations involved (bounded by the square), and the usees, who are external to the system as a whole (bounded by the circle) [49].
Applying these socio-technical notions, the authors contend that RA is capable of adaptation to accommodate stakeholders additional to the system sponsor, in a form referred to here as Multi-Stakeholder Risk Assessment (MSRA): an adaptation of conventional, organization-internal RA that encompasses multiple, parallel assessments that draw on a common base of information, but each of which is undertaken from the perspective of a particular stakeholder. The results of the various assessments are integrated into a consolidated multi perspective form. That integrated body of understanding is then applied during the Risk Management phases of the process.
B. Supplementary Literature Review
A further scan was undertaken of the risk assessment literature, to identify theoretical and practical contributions that extend RA’s scope beyond its conventional focus on the interests of the system sponsor. Searches were conducted in mainstream IS journals and the AIS Electronic Library (AISeL). These were supplemented by more open-ended searches across Google Scholar. The intention was to identify, acquire and inspect papers that contain “risk assessment” in Title and/or Abstract, and “stakeholder” anywhere in the text. Because the capabilities of the available search facilities vary considerably, it was necessary for some of the searches to use variants of that strategy.
The topic of risk assessment has not generated a large corpus of work. For example, among the over 17,000 refereed papers in AISeL, only about 500 (c. 3%) contain the primary term anywhere in the text, and only 32 (c. 0.2%) of those con tain it in Title and/or Abstract. Similarly, searches in specific journal collections found about 3-4% of papers mentioning the term, with well under 1% also containing “stakeholder”. Further, the large majority of articles that satisfy the search criteria treat stakeholder interests only as constraints on the achievement of the system sponsor’s objectives, not as objec tives in their own right.
Given the limited literature found on multi-perspective risk assessment, a review of broader perspectives on IS analysis and design processes appeared to be necessary. Much of the early work in the field was strongly oriented towards technical systems, e.g. [69]. Even where some emphasis was placed on man/machine interaction, transaction processing and information extraction were designed to serve the interests of the sponsoring organization (e.g. [70]). As the scope of IS expanded over the next quarter-century, the interests of stakeholders other than the sponsoring organization became apparent. This is evident even within the business-school context, where three of the top five software project risk factors were perceived to be “Failure to gain user commitment”, “Lack of adequate user involvement” and “Failure to manage end user expectations” [71, p. 78]. Yet studies of IS failure continued to adopt organization-centric approaches, with the interests even of users, let alone usees, attracting limited attention [72], [73].
Broader vision is found in the work of Checkland on soft systems methods. This recognizes the existence of conflicting worldviews, and the need to act purposefully to negotiate a design that each worldview can live with [74] and [75]. Parallel threads were the ETHICS approach, dating to 1983 [76], and the early years of Scandinavian and British tradi tions in participative design [38], [77]. Participative design, however, has gravitated towards dominance of the sponsoring organization’s interests, with the focus on the effectiveness of the resulting system, reduced resistance to change, and support for functions to be performed and return on investment to be achieved.
Impetus for a broadening of the beneficiaries of research has been provided by several recent initiatives. Reference [78], writing in the context of fintech research, invoked ‘making a better world with ICTs’ [79]. Other authors have referred to “understanding and tackling societal grand challenges through management research” [80], and “responsible IS research for a better world” [81], which resulted in a Special Issue in ISJ [82], [83]. To these can be added [84], [85], which consolidate the legitimacy of IS research relating to societal change, public policy and regulation, and the Responsible Research in Business and Management movement [86], whose first three Principles are “Service to society”, “Stakeholder involvement” and “Impact on stakeholders”, (e.g., [87] on ethical and inclusive digitized society).
Among the very small pool of relatively recent papers on risk assessment that have at least hinted at a broader view of interests, many include only small contributions, such as a brief statement that “risk is socially constructed through a process of articulating, negotiating, and contesting by various stakeholders in the project” [88] and recognition that “research subjects, consumers, or other stakeholders...are likely to assess risks differently” [89]. Keil et al. [71, p. 107] note that “Although the information systems literature recognizes the significance of integrating multiple stakeholders’ knowledge during the course of design and coding...there has been no attempt to integrate multiple stakeholder perspectives in the area of IT project risk”. Their own work, however, is limited to intra-organizational users, fails to discriminate categories of users, and excludes usees.
A paper that is superficially close to the proposal in the present work is [90]. The authors describe ‘a multi perspective modeling method for IT risk assessment’ that they call RISKM, the first requirement of which is to “provide perspectives specific to (groups of) stakeholders involved in the group process [that] should...be integrated with each other to foster cross-perspective communication and cooperation” [88, p. 604]. However, the notion of stakeholder is limited to an organizational role, i.e. it is concerned with sub-organizations of the system sponsor, the examples being IT and project management, line and top management, and internal and external auditing. No further mention of RISKM has been found in the literature.
More positively, ethicists [91, p. 733] argue that “risk resolution should consider...the different stakeholder expec tations about how to judge a project as a success or a failure”. The contexts considered by Gotterbarn and Rogerson [91, p. 735] include software to support bridge design and “A patient waiting to be identified by software as a heart transplant recipient”. They also distinguish what they refer to as direct and latent stakeholders– the latter being equivalent to usees. The lead author of this article in [92] presents a risk assessment framework for mobile payments, in which the harm model includes financial loss to users, merchants, card issuers, transaction-processing organizations and the various service providers involved. In proposing practicable backup arrangements for individuals and small organizations, [21] identifies as potential stakeholders: members of associations; members of enterprises such as investment clubs; employees, contractors, customers and suppliers; and individuals’ families and clients, in both an economic and a social sense.
Stakeholder identification is a primarily pragmatic process rather than one supported by robust theory. This is because of its heavy dependence on context. Effectiveness of both design and risk assessment requires deep appreciation of the envi ronment into which the intervention is being injected. This in turn depends on having an adequate model of the players and sufficient understanding of the political economy dynamics among those players. However, social network analysis may be of value in some circumstances [93].
C. Articulation of the Process
Fig. 3. Multi-Stakeholder risk assessment and risk management. Adapted from [60, p. 414].
Our intention is that MSRA be minimally disruptive to whatever flavor of RA each organization currently utilizes internally. An outline of the conventional RA process is provided by the sequence of steps down the left-hand side of Figure 3. This depicts RA as a series of nine steps making up the first phase of a larger process. This analysis phase is followed by a design phase that prepares a Risk Management (RM) plan in sufficient detail to be executable, and an implementation phase that brings the plan to fruition.
In principle, MSRA could be performed as a single, integrated, collaborative process, with all parties actively par ticipating at the table. This approach may be appropriate where the system sponsor recognizes the advantage of direct engage ment with the relevant parties. The sponsoring organization can thereby gain a deep understanding, enabling the reflection of stakeholders’ interests in the project design criteria and features. In practice, however, the considerable asymmetry of information, resources and power among stakeholders is challenging to overcome. Moreover, substantial disparity exists among stakeholders’ perspectives, values and dialects. Few organizations’ executives, analysts and designers are likely to be able to meet the challenges involved. The technique proposed here accordingly features parallel RAs, undertaken independently, working from a common understanding of the proposal.
The primary extension involved in transitioning from RA to MSRA is the performance of additional instances of steps 1.4 to 1.9. This may be done:
• by stakeholders themselves (which is appropriate where the stakeholders are organizations with access to sufficient resources and expertise);
• by advocacy organizations on behalf of stakeholders (which is more suitable where stakeholders are individuals or small organizations); or
• by proxies for stakeholders (such as consultants, or, in the weakest form, staff-members of the sponsoring organization, playing a role by adopting an external perspective).
The advantage of advocacy organizations over consultants and role-playing staff lies in their experiential base and the authenticity of their input, as evidenced by the richness of anecdotes, examples and exceptions available to them. Advocacy organizations are in many cases poorly resourced, and in some cases entirely dependent on volunteer resources, and hence the sponsoring organization may need to provide support, such as funding for travel to enable participation in key events.
To facilitate the performance of multiple, parallel risk assessments undertaken from different perspectives, it is necessary to make a small adaptation to step 1.3. A document needs to be produced and distributed that describes the socio technical intervention at a sufficient level of detail to enable each stakeholder group to undertake its own assessment. The document needs to be introduced to participants through a briefing, and its implications articulated through discussion.
To provide effective feedforward to the RM phase, the results of the various risk assessments need to be integrated into a consolidated document (step 2.1). This needs to be introduced by a briefing and articulated by discussion, this time for the internal project team. The adequacy of the resulting Risk Management plan needs to be evaluated against this consolidation of the outcomes of the MSRA (step 2.3). To ensure that the stakeholders gain the intended benefits, and that the sponsoring organization enjoys public support from the stakeholders, the loop is best closed by means of some form of stakeholder participation in the Risk Management implementation phase.
The term Multi-Stakeholder Risk Assessment (MSRA) was adopted in [58]. Although it is a straightforward descriptor, searches for previous usage in the refereed literature have found only a few uses of the expression, of which [94] appears to be the most relevant, and [95] the most recent. The term Multi-Stakeholder Risk Management (MSRM), applicable to the subsequent phases during which the consolidated risk assessment report is applied, has also scantly appeared in the literature, notably in [96] and [97].
VII. MSRA PROSPECTS
The previous sections have addressed the design science steps of problem identification, definition of objectives, and design. The next DSRM phase identified by [57] is Demon stration. In their discussion of the Demonstration phase, those authors include the depictions “Find suitable context” and “Use artefact to solve problem” [59, Fig. 1, p. 54], and “use in experimentation, simulation, case study, proof, or other appropriate activity” [57, p. 55]. They include several different instantiations appropriate to the context of each of their four case studies.
MSRA is a business process description, in design science terms a socio-technical artefact, an adaptation of the widely used and standardized business process of risk assessment. In these circumstances, key aspects that require demonstration are the technique’s practicability, the existence of drivers for its adoption, and its potential efficacy. The first sub-section below considers practicability by identifying exemplars of some of the elements of MSRA in prior practices. The second sub section identifies economic and social drivers for adoption of MSRA. The third sub-section provides a brief review of a recent case study to demonstrate MSRA’s efficacy.
A. Prior Exemplars
A first test of the proposition that MSRA has potential is whether circumstances exist in which at least some of its important features are apparent. Uses of MSRA-like features exist outside the IS field, particularly in environmental con texts. Small-scale Environmental Impact Assessment (EIA) activities, e.g. for the storage or use of chemicals in such places as fire stations, are often the subject of informal interactions with groups representing community, environ mental and perhaps indigenous interests. These assist the organization to appreciate the possibly complex, contested, vague and/or conflicting concerns of such groups [98]. In economic contexts, major regional employers that are prepar ing to close operational facilities often consult with interest groups in the affected communities. This is particularly the case with government organizations, but also with corporations in, for example, the resource-extraction, steel-making and car assembly industries.
Another area in which aspects of MSRA are visible is investigation of and experimentation with inherently high-risk products. Medical implant designers undertake carefully designed pilot studies, with active participation by multiple health care professionals, patient categories, and patient advo cacy organizations. In such contexts, developers’ awareness of stakeholders’ needs, and of the scope for collateral damage is institutionalized by regulatory requirements (ISO 14971:2019 [99]).
Inter-organizational IS in highly-networked industry sectors such as health and international trade involve many specialized enterprises, and many inter-linkages and flows among them, overlaid by regulatory interventions to satisfy such public interest needs as hygiene, public safety, service quality and tax-collection. Changes to architecture, infrastructure, data flows and business processes depend on effective negotiation among many players. Some aspects of MSRA are evident in such contexts ([100], [101]).
A further cluster of examples exists in the area of efficient electronic markets. A government may break open a monopoly marketspace to overcome entrenched, excessive profit-making and stultification of innovation, by exposing the impacts of the current arrangements, gaining public support, and in effect shaming the beneficiaries of the monopoly into accepting the need for change [102]. One side of a marketspace may drive such a change. Livestock producers, distant from major population-centers, and facing high costs to get their stock to sale yards, and low prices offered in so-called ‘farm gate private treaties’, involved multiple stakeholders in the development of an online auction scheme to reduce informa tion asymmetry and better balance the market-participants’ interests [105]. Geographically dispersed producers faced entrenched information asymmetries and dependency on dom inant intermediaries. The shift to online auction platforms involved not only producers and buyers but also stock agents, regulators, transport operators, and rural communities. A crucial design feature was the creation of a profession of livestock inspectors, whose function was to assure buyers of the appropriateness of descriptions of livestock job-lots. Meaningful stakeholder involvement, particularly consulta tion with producers’ associations and rural advocacy groups, helped ensure transparency in pricing mechanisms and reduced opportunities for exploitation. In contrast, earlier attempts to reform market practices without such inclusion had failed, as the interests of smaller producers and remote communities were insufficiently represented.
A contrasting class of examples arises in platform-economy transformations such as ride-sharing or home rental platforms [103]. Disruptors from eBay (since 1995) to Uber (2009), and many other aspirants, have taken advantage of new technology and the start-up’s lack of legacy technology and a well-paid labor force, by deploying new services at a speed that large, established corporations simply cannot match. Case studies have highlighted failures by regulatory agencies to enforce existing laws, to the deep disadvantage of heavily-regulated incumbents. In some jurisdictions, the exclusion of incumbents from discussions resulted in capitulation by regulators, disloca tion and financial losses suffered by investors and employees, and loss of amenity by communities. Where broad consultation took place, on the other hand, transition strategies enabled progress without dramatic negative consequences [104], [105], [106].
In several jurisdictions, including New York City, taxi regulators engaged only with new platform entrants (ride-share companies), sidelining incumbent operators, local councils, disability advocates, and community groups [106]. This lack of inclusive stakeholder engagement resulted in sudden income shocks and capital losses for regulated incumbents (some even committing suicide), community safety concerns (where women were harassed or assaulted), and litigation (in not providing for those requiring wheelchair access). These cases illustrate how stakeholder exclusion can amplify harm and produce irreversible system destabilization.
B. Impetus for Adoption
A further important test of the viability of the adapted process is whether mechanisms exist that provide impetus for its adoption. Applying innovation diffusion theory [107], important characteristics of MSRA are: relative advantage the degree to which it is perceived to be better than what it supersedes, compatibility- consistency with conventional RA processes, complexity- difficulty of understanding and use, trialability- the degree to which it can be experimented with on a limited basis, and observability- the visibility of its results.
Circumstances arise in which the system sponsor is subject to a requirement to reflect the interests of multiple stakehold ers, with a sufficiently credible threat of enforcement action to motivate compliance. The impositions might derive from formal statutory authority, contractual or licensing conditions, moral suasion by a powerful regulatory agency, or perhaps aspects of the institutional environment, such as widely adopted industry standards or cultural conventions within the industry sector or country.
Even where no formal obligation exists, circumstances arise in which a sponsoring organization perceives advan tages in gaining an understanding of the various perspectives, accommodating the interests of other parties without bearing disproportionate cost, and keeping harm to its own interests within manageable bounds. This is most likely where the system sponsor recognizes moral obligations on the grounds of public policy, business ethics, corporate social respon sibility (CSR) [108], or a commitment to public interest technology (PIT) [22] or ‘environmental, social, and corporate governance’ (ESG), as codified in the UN Principles for Responsible Investment (UNPRI) [109]. This appears most likely where the institutional context reinforces collaborative and human-wellbeing values. This is the case, for example, with a public agency whose mission is to deliver economic and/or social value to particular population segments.
System sponsors are more likely to respond positively when the benefits of multi-stakeholder engagement are framed in terms of risk reduction and reputational protection. Structured engagement, such as staged, suitably-resourced and time bounded consultation rounds, consolidated briefing packs, and clear agendas, reduces the perceived burden on providers.
Further, involving established advocacy organizations pro vides a bridge between the sponsor and diffuse stakeholder groups, offering experiential legitimacy and reducing informa tion asymmetry. Commitments to transparency (e.g., disclosure of biometric thresholds, data retention rules, or inter-agency sharing constraints) have been effective in building trust.
C. Efficacy
Beyond providing evidence of the feasibility of MSRA being adopted by organizations, DRSM calls for its efficacy to be demonstrated. This section draws on a deep case study of a large-scale IS failure in order to suggest how different the outcomes would have been had MSRA been applied.
In Australia, the processes involved in paying welfare bene fits are conducted by a specialist agency called the Department of Human Services (DHS). Its Centrelink Division runs a vast database that supports the transfer of over AUD150 billion p.a. to over 5 million clients who receive support from about 100 different schemes. Some of the largest schemes require clients to report on a fortnightly basis such income as they receive from part-time or casual employment. Errors and delays in reporting, and in some cases fraudulent behavior, result in incorrect amounts being paid. DHS invests consid erable resources in detecting where this has occurred, and arranging recovery of overpayments. During the period 2015 22, DHS sought to further automate the overpayment detection and recovery processes. The project became widely-known as ‘Robodebt’. This brief review of the project draws on a comprehensive case study in [73].
The key feature of the initiative, referred to as income averaging, involved the extraction of annual data from each benefit-recipient’s taxation return, and division by 26 to pro duce the average fortnightly income. DHS in effect asserted this to be the correct figure for each fortnight, and clients whose direct reports to DHS had been materially different during any period were sent demands to produce specific documentation to justify their claims, in default of which a debt was automatically raised against them and recovery processes were automatically commenced [110]. Given the elapsed time of multiple years, many notices going to out of-date addresses, many clients’ poor comprehension of the DHS letter, many clients’ poor record-keeping, and many clients’ inability to extract copies from the relevant employers, well over half of the more than 400,000 people subjected to demands were unable to prove their innocence. Over a 3-year period, DHS recovered nearly AUD 0.8 billion from people who were, by definition, among the nation’s poorest and most welfare-dependent [111].
If MSRA had been applied, diverse client-segments would have been identified as stakeholders, particularly the unem ployed, aged, partially disabled, tertiary students, and first nations people. An engagement process would have been instigated with the advocacy organizations that represent their interests. This sector is well-established in Australia, and the relevant organizations are easily located. Sufficient information would have been provided to those organizations in advance of the agency committing to the initiative. Multiple advo cacy organizations would have identified the multiple serious deficiencies in the proposed scheme, and communicated the problems to DHS. DHS would have had the opportunity to assimilate the information. The agency’s senior executives would have been unable to avoid recognizing that the proposal was seriously flawed and unviable, and harbored very substan tial risks to the agency’s reputation. A more energetic search for variants and alternatives would have been conducted.
The full case study in [73] shows that DHS did the opposite. It obfuscated the details, did not institute any engagement, rebuffed attempts by advocacy organizations to engage with it, suppressed internal legal advice, avoided seeking external legal advice, ignored unrest among frontline staff, and applied pressure to its staff in an endeavor to prevent leaks to the media. After 5 years of agony for clients and staff alike, the scheme eventually collapsed, the repayments extorted from clients were repaid, with total wastage of public funds of the order of AUD 3 billion. Massive LIS failures continue to occur [1]. MSRA has demonstrable potential to nip them in the bud.
VIII. APPLICATION OF MSRA
In [59, p. 56], the Evaluation step of DSRM involves observing and measuring “how well the artifact supports a solution to the problem...Depending on the nature of the problem venue and the artifact, evaluation could take many forms.” We apply the same approach to the Evaluation as that used in study by the Peffers et al. [59, pp. 57-73], by examining the application of the proposed socio-technical artefact, in our case MSRA, to a documented case study. This demonstrates how a succession of disclosures, discussions and recapitulations built mutual understanding among multiple advocates and sponsor representatives, leading to a mutually acceptable design outcome.
To represent an effective test-bed for MSRA, a case needs a number of key characteristics. The intervention has to be of significance and scale, and to be injected into an environment that involves multiple stakeholders with compelling interests, thereby ensuring a sufficiently rich contextual setting. Ade quate documentation needs to be available. It is desirable that a number of points of potential failure exist, of a kind that might or might not be amenable to prevention or mitigation by exploiting opportunities provided by the MSRA technique.
A project conducted by the lead author was selected, for the same large Australian government agency as appeared in the vignette in the previous section. It dates back some years, however, to a time when the agency was far more attuned to the need to reflect stakeholder interests. To support its interactions with its more than 5 million clients, it operates over 300 physical service centers, web-sites, and call-centers handling over 30 million calls per annum. Interactive voice response (IVR) was already used to pre-process calls and, in an increasing proportion of relatively simple transactions, to enable fully-automated handling, to reduce costs and improve speed of response.
On their first contact with Centrelink, clients establish their identity face-to-face. Thereafter, various authentication pro cesses are used when customers contact the agency, depending on the channel and the nature of the transaction. Centrelink was considering the addition of voice authentication as a further factor, with the intention of reducing the time spent by staff, and of enabling additional self-service transactions. It commissioned a consultant to assist it to understand and address its clients’ privacy and other concerns in relation to the initiative. One of the authors was lead-consultant on the project. The description and quotations used here are taken from publicly-available documents, although some insights and interpretations take advantage of the participation by the lead author in the project [112].
A. The Project Launch
At the commencement of the project, the consultant con ducted an Issues Analysis, to provide Centrelink with a broad view of the field, and to assist in devising a strategy to surface and articulate issues that could affect the success of the undertaking. Generic privacy concerns were distinguished from issues that may arise with given client segments. The purpose was to lay the foundation whereby the design could avoid or accommodate concerns, or at least ameliorate the potentially harmful effects of necessary design features.
Centrelink agreed to the consultant’s recommendation of an engagement process with consumer and privacy advocacy organizations, and to the process being iterative through the phases of the project. A long list of 17 non-government advocacy organizations was identified, with the intention of constructing a sufficiently diverse, but workably small, col lection of organizations and individuals, each with interest in the subject-area, and either sufficient expertise or the capacity to quickly develop it. The list was winnowed down to a priority list of 5 individuals and 5 reserves. Invitation and joining instructions for the first face-to-face meeting were supplemented by a 10-page Briefing Paper on what technology-providers called Speaker Verification Technology.
The Briefing Paper was supplemented by a tutorial session by a consultant technologist. The basis of the technology was explained as being that the shape of the vocal tract conditions the speech sounds a person makes, and that each individual’s speech differs sufficiently from everyone else’s that it is possible, at least in principle and with a sufficient and sufficiently clear speech sample, to confirm or deny an identity assertion with considerable confidence. At enrolment, the client is stimulated to provide enough utterances that a composite model or ‘voiceprint’ can be built, which is used as the reference-point against which future samples are assessed.
The advocates stated their appreciation of the briefing, quickly assimilated the technical material and homed in on a set of concerns about the technology and its application. One related to the criteria used to decide on acceptance / rejection decisions, which were declared as the False Acceptance Rate (FAR), the False Rejection Rate (FRR) and the conventional (but arbitrary) assumption that an Equal Error Rate (EER) is desirable. One advocate asked “Shouldn’t the balance between anticipated FAR and anticipated FRR be based on a risk assessment that reflects the concerns of both Centrelink and its customers?”. This placed on the table the expectation of advo cates that risk perceptions are dependent on the perspective of the observer, that the many client-categories have perspectives different from the agency and, to some extent at least, from one another, and that their perspectives are legitimate and need to be reflected in the design.
Another early-emerging issue was expressed variously as “Do you keep raw audio-samples?” and “I’d have a huge problem with [retention of] the raw audio-stream. I’m not sure about the voice-print because we don’t have enough information on the table yet”. Over the course of the three meetings, this developed into express concerns about two aspects, one of which was the likelihood of data-sharing among agencies, and the co-option of the scheme into the recurrent dream-project of the Australian Public Service to implement a national identification scheme.
The other and substantial concern was about the scope for, and the agency’s intention of enabling, the use of voice prints for identification (through one-to-many comparison) rather than, as indicated in the background information, for authentication (which involves a one-to-one comparison between a current voice-sample and the previously-generated voiceprint for the client who the speaker purports to be). To the question “There’s a dissonance in the message: is it about identification or not?”, the agency responded “with say 1 million enrolled, [a 1% ERR] implies that 10,000 might be within the standard threshold setting [i.e. 10,000 potential matches within the database, and hence generally difficult to apply to identification]”, but also “The primary policy concern is authentication; a secondary interest that we have to have in mind is addressing fraud [which depends on identification]”.
A third fundamental issue was the risk of masquerade, to some extent mischievousness within families and households, but also at the level of changing bank account details to commit financial fraud: “As with any biometric, there is a risk of ’compromised once, compromised forever”’. Another question was whether there are “biological characteristics in the generated [voiceprint]” or “personal characteristics, such as ethnicity, unAustralian accent, Downs syndrome, cleft palate,. ... voice-disabled with synthesized speech”.
Multiple, less contentious aspects also appeared early in the conversation. One was that “People will feel alienated and intimidated if they don’t readily authenticate. And Centrelink clients are already alienated and intimidated”. This resulted in the agency appreciating the need to both design for tolerance and to study the circumstances that give rise to Failure to Enrol (FTE) in the first place, and to repeated False Rejections for the same person. Another related to the foreignness, and to many clients the apparent pointlessness, of lengthy exercises talking to a machine to enable it to create a voiceprint. During subsequent pilots, this did indeed appear to be a material factor in a significant minority of recruits failing to complete the enrolment process.
Another concern was about equity, in particular, “differential levels of false rejections or false acceptances, depending on some characteristic of the client or client-group, [resulting in] differential levels of service [e.g. longer or more arduous processes through a different communications channel]”. The first meeting also reinforced a particular consultant recom mendation, which was that tenderers be required to provide sufficient information to Centrelink that throws light on these issues, and that can be disclosed to the consultees.
B. The Second Meeting
The second meeting 18 months later was again supported by a Background Paper sent in advance, this time provid ing a considerable amount of (non-confidential but relevant) technical information from tenderers. There was moderate turnover in the composition of the advocates, with two of the previous reserves joining, but bringing with them reasonable background in authentication.
The client target-group had in the meantime been refined to comprise frequent callers, not occasional ones, and especially clients with a fortnightly income self-reporting obligation. In response to an advocate’s question, the agency stated its motivations were “cost-savings in call-centres (time-reduction for operators, and increased percentage of calls performed entirely with IVR); but also improved authentication quality”. Breakeven on the investment was said to be achievable with a 10% take-up by volume-clients.
The possibility was declared of Centrelink providing an authentication service for other agencies. The concern was again clearly stated that “the voice template could leak to other organizations. Other agencies are known to be involved in some way already”. Transparency to clients was identified as an issue: “Underplaying the significance of [data re-use and disclosure to other agencies] falls short of informed consent. ... Centrelink needs to encourage adoption, and to do that it needs to engender trust by fully and openly declaring”. A further issue was that “conversions from raw voice-print to template do not really represent a one-way hash”, implying the possibility of the reverse-engineering of voices, thereby enabling masquerade.
As regards use identification, the agency stated that “It’s not tenable to use the data for a fishing expedition, because of quality reasons, and the considerable processing requirements: at 200 msec for each matching test, a comparison against 6 million records would require 1.2 million seconds, or 14 days of processing-time. That might be tenable in investigating a multiple fraud, but not otherwise”.
Responding to a question about mandating enrolment for voice authentication, the agency replied “it will not be compul sory as such, but it will be required for the self-service channel, i.e. there will be some transactions which can only be per formed with voice authentication”. The statement that “There’s little knowledge available about the characteristics of ‘goats’ [an industry term for people who suffer multiple authentication failures]” gave rise to concerns about discrimination against some categories of client (e.g. the poorly-educated, the voice impaired, and those with English as a poor second language, or with significant accents arising from their lingual background). The agency confirmed that “we don’t really understand the nature of the outliers”, including the seriously vulnerable, which includes, for example, alcoholics and people with per manent or episodic psychological conditions such as psychosis and depression.
A concern was raised about whether agents (typically family-members) would continue to be able to call on behalf of clients. Overall, it was clear that “choice is critical”, because “voice can’t be silent, and hence can be intercepted at origin”. “Any move to impose voice authentication as a condition of using the telephone would be of great concern”.
C. The Third Meeting
The third and final meeting was held a further 18 months later, with 4 of the previous 5 advocates participating. The agency replayed the background and outlined the pilot imple mentation that was under way at the time, with some hundreds of participants. Failure to Enrol had begun very high but had been reduced to 16%, with measures being taken to lower that further. False Rejections averaged 7%. No instances of False Acceptance had been detected. Noisy environs of the caller, and noisy communications channels, appeared to be major factors.
The agency advised that “The [system-integrator] would like raw voice streams to be retained in the SV Engines. Subject to some remaining clarifications, Centrelink’s current position is that there is insufficient justification for doing so, and no legal requirement to do so; and it intends to not permit retention of raw voice streams in the SV Engine databases”.
The advocates accepted the proposition that the data stored in the system could be used for fraud investigation and enforcement processes, but “only where suspicion had arisen from some other source”. They were very concerned about “the use of the data as a means of generating suspicion, not least because voice is suitable for authentication, but not for identification”.
Ongoing concerns were expressed about the possibility of retention of raw-voice data, and of re-use, sharing, disclo sure or unauthorized access to the voice-prints. The agency acknowledged some of the concerns “and concurred, therefore, pending legal confirmation, that it will continue to be the case that the biometric samples are not retained”. Further, given the use of a third-party service-provider, the risk existed of Centrelink not having full control over design, operation, data management, and quality and security assurance. Further issues remained about control over scope creep and de facto mandation, and the fact that there was “as yet no ability to withdraw from the scheme [after earlier enrolment]”.
D. Evaluation of MSRA Applied to the Voice Authentication Project
This sub-section reviews the evidence from the case study against the key features of MSRA. Despite the seriousness of the concerns expressed, the advocates were experienced in managing conflicts of interests and conducting discussions at worst civilly, and at best in an engaging manner. For their part, the agency’s executives and project staff were calm and forthcoming, and advocates, in each meeting, were very positive about the transparency and the engagement levels shown by staff. A pertinent example was their joint statement that “We’re impressed with Centrelink’s willingness to engage in this dialogue, but are concerned about whether the initiative is or will be subject to regulation”. Overall, advocates expected that key protective provisions would be legislated rather than merely left as undertakings by one generation of Centrelink executives that can be reneged on by a later generation.
The client-advocacy participants were uniformly supportive of the consultative process used. They stated that they were “not aware of any other processes that had considered voice authentication in any depth”. They were uniformly “supportive of the design, subject to two areas of qualification...They iden tified the most critical factors underlying the PCG members’ support as use for authentication and not for identification, consent-based/not mandated, no serious disadvantage for those unwilling or unable to use it, and legislated protections”.
The identification of stakeholders was achieved on the basis of the consultant’s interactions with Centrelink, combined with the consultant’s knowledge of the advocacy organization landscape and the snowball effect of referrals among non government organizations (NGOs). The power of the various advocacy organizations was at no stage a focus, with the key factors applied by the agency being the relevance of the NGO’s perspective and the expertise of the individual invitees.
The agency agreed to, and implemented, a commitment to transparency. This included statements about, and progressive refinements to, the agency’s motivations. Information about the technology was provided by the agency and its technical consultants, and, as a condition of tendering, by the potential suppliers to the agency. This enabled the advocates to conduct risk assessment exercises individually, collectively, and by workshopping within NGOs, and to feed the results into the agency. Unlike the MSRA process described in this article, the separate, parallel risk assessments were not formalized, and not directly resourced by the project sponsor. On the other hand, the stakeholder groups benefited from the open, interactive process, in that each could draw on the insights of other advocates.
Centrelink executives evidenced assimilation of the feed back, variously informally within meetings, more formally in subsequent-round documents and pre-considered verbal state ments, and in commitments for and against design features. Safeguards and mitigation measures were expressly discussed, in relation to the client-segments most likely to be disadvan taged by the scheme.
From the advocacy perspective, there were two key short falls in the process. Even though the engagement took place in three intensive episodes over a 3-year period, the consultation process did not continue into implementation. One reason was that full implementation of the technology occurred fully 2 years later, and another that staff turnover, and parallel changes in the agency and its programs, degraded the qual ity of the agency’s corporate memory. A second problem was that, despite the advocates’ strong representations that formalized protections be legislated, no such step was ever taken.
From Centrelink’s perspective, on the other hand, the agency’s interests were very well served. Advance warning of potential roadblocks and adoption barriers was gained. A number of these could be avoided by design, and in other cases measures could be built into the design to ameliorate the potential harm. The case study evidences many of the features of MSRA, and the outcomes evidence many of the advantages identified earlier in this article.
A meta-observation arising from the case study is that the pool of advocates who can participate at this level is limited. Many are independent consultants, who face conflicts of interest in relation to the gratis provision of their stock in-trade, which is their expertise, and who can offer only limited pro bono time; and many others are employees of advocacy NGOs which face competing priorities. For most individuals, the issues arise of unfunded travel and travel-time, and advocate fatigue. The availability of sitting fees and travel support (not necessarily advertised, but available on request) demonstrably improved participation, and this appears likely to be the case more generally.
IX. DISCUSSION
Technical interventions into socio-economic systems involve many elements. In the 21st century, these generally include the application of powerful IT, which is increasingly being extended to automated inferencing, decision-making and action. However, the mindset around large-scale, impactful interventions is that they are socio-technical by their very nature and thus need to be evaluated, not just implemented in that way. Technical interventions cannot run autonomously, even if they use techniques such as artificial intelligence (AI); and they are operational within a given regulatory environment, physical location and organizational policies, and for this reason MSRA can be considered a socio-technical artefact applied to socio-technical phenomena. Additionally, many assessment techniques have a tight focus on the interests of the system sponsor. There are few drivers for the inclusion of other stakeholders in assessment processes. To the extent that other stakeholders’ interests are reflected, it is because those parties have sufficient power.
The scope of IT-based systems has expanded from single functions within an organization, to multiple functions across organizational boundaries, and beyond those boundaries, out to individuals. Such systems are inherently socio-technical and cannot be meaningfully considered as mere technological artefacts. Meanwhile, stakeholders that have legitimacy but lack power suffer depredations from technical interventions, because system sponsors are not attuned to their interests. In many cases, harm to their interests could be avoided, or at least mitigated, and even some benefits delivered to them, with only limited compromise to the sponsor’s objectives, provided that legitimate stakeholders’ needs are factored into the design. Further, if those stakeholders’ needs are understood at an early stage, the costs of addressing them may not be great.
Identifying usees requires mapping not only the formal system boundary but also the indirect impact pathways rep resented in Figure 2. Analysts should trace data flows, dependency relationships, secondary service interactions, and foreseeable downstream effects [113]. Stakeholder snow balling techniques, particularly via consumer-rights and social service NGOs are useful in revealing latent stakeholders who may be indirectly affected by the intervention. Common cate gories of usees include people subject to automated decision making without direct participation, households affected by another individual’s enrolment failure, persons who care for others with accessibility limitations, linguistically diverse communities, the unhoused who may not be eligible for benefit payments because they do not have a fixed address and are thus invisible, and groups exposed to algorithmic or biometric error rates that differ systematically from the majority population [114].
Concrete examples from our case materials further illustrate this. In the Robodebt context [111], usees would have included those members of the community who were family members, or who had direct guardianship responsibility, or power of attorney of benefit recipients who had volatile employment patterns, or persons representing first nations peoples, support interpreters who help young people who have low literacy, those experiencing homelessness, and individuals unable to respond to correspondence delivered to outdated addresses. Another category of usee was legal aid organisations that supported Robodebt victims in formally challenging auto mated decisions and actions. The diversion of scarce resources created further problems by denying the capacity to assist other needy clients.
In the voice-authentication project, usees encompass those persons who are reliant on intermediaries such as carers or family members, individuals with cognitive or speech impair ments, callers operating in noisy environments or with hearing loss, people with strong accents or dialectal features. Early identification of these groups is essential to ensuring that their perspectives are reflected in the parallel assessments that make up MSRA. Dependent on the context, the user may be the benefit recipient, and the usee a member of society who supports an individual who is a benefit recipient, or lives within their household. This could extend to a doctor, lawyer, justice of peace, social worker, teacher, carer, or any provider who could speak on behalf of the benefit recipient or provide evidence in a given context.
Even if the will exists to reflect stakeholders’ legitimacy as well as their power, there are challenges in finding suitable assessment techniques to apply. Business Case Development is primarily driven by the prospects of profit or cost-savings, or the need to demonstrate compliance with regulatory require ments. All variants of Impact Assessment are foreign to corporations, and to many government agencies [115]. While more is being said regarding inclusion, usually this is related to financial inclusion (i.e., affordability), and does not address matters related to social inclusion (e.g., ageism, dis/ability, accessibility) that can exacerbate stereotypes, discriminatory practices and justice issues.
The research reported here has discussed and articulated a serious weakness within contemporary business processes. The interests of stakeholders other than the sponsoring orga nization are excluded, except to the extent that a stakeholder is seen as being powerful enough to represent a threat to the achievement of the sponsoring organization’s objectives. This is an understandable simplification in the case of relatively unsophisticated IS that operate entirely within a single orga nization, and use data to generate information. It is untenable in the contemporary context of LIS that are chained and net worked inter-organizational systems, and extra-organizational systems in which individuals outside the organization are deeply involved or otherwise significantly affected. This kind of thinking also falls short of the “social” in the “socio-technical”, because it prioritises the interests of one stakeholder above all others.
This paper proposes a new form of risk assessment process. Rather than a revolutionary, alternative business process, it is an adaptation of an existing and familiar business process. Initial assessment of the potential of the proposal has been undertaken and reported. The authors contend that adoption of this adapted business process delivers practice impact that can advantage rather than harm stakeholders, while satisfying the needs of the system sponsor.
A design science approach was applied. Despite being a creature of rational enterprise management, risk assessment has been shown to harbor good prospects for reflecting the interests of multiple stakeholders. Scans for instances of, and potential drivers for, the adoption of the MSRA technique identified a variety of contexts in which some of its hallmarks are evident. The potential efficacy of MSRA was illustrated by considering a large-scale government initiative whose mis conceptions gave rise to considerable cost to the public purse, and seriously harmed many users, and many usees.
The adoption of MSRA will not bring in the millennium. Sponsors and stakeholders will continue to be suspicious of one anothers’ motives. Transparency may be feigned. Under takings may not be documented, and heads of agreements may not be respected. Facilitators may bias agendas and the flow of discussion. Graphics may be used to ‘snow’ ’others’. Genuine zero-sum games will still exist. Sometimes, advocacy organizations will compete with one another; and tensions among the differing interests of sub-organizations and individuals within the sponsoring organization will affect the process. Some advocacy organizations, and some sponsors, will game the system, and will take advantage of goodwill displayed on the other side.
On the other hand, MSRA creates the scope for the various sides of arguments to grasp something about one another’s perspectives, develop inter-personal trust, and iterate their way towards outcomes that are better, or at least less-worse, than what would have arisen under blind, adversarial schemes. The final step in the design science process of Peffers et al. [59] is evaluation of the new artefact. This was undertaken by using MSRA as a lens through which to observe a documented case study of a government agency undertaking risk assessment and management of an application of inherently intrusive biometric authentication technology.
On the basis of the demonstration and evaluation steps that have been undertaken, exposure of the proposal is now appropriate, followed by experimentation and trialing, in order to gather field evidence as to whether MSRA can be a practicable mechanism to reflect the interests of the relevant players and to be effective for and acceptable to sponsoring organizations.
X. CONCLUSION
The research reported in this article has implications for researcher perspective theory. It provides evidence in sup port of the idea that it is feasible to adapt a standardized and widely-adopted single-perspective business process into a multi-perspective technique. This can be drawn on by academics who are performing practice-relevant research to inform the development of research techniques that articulate the notion of multi-perspective design science.
Further work is required to investigate the scope for integra tion of MSRA with systems analysis, design, implementation and continual improvement processes. Consideration also needs to be given to the application of MSRA in contexts in which the notion of a singular sponsoring organization or system sponsor is not appropriate, such as where the sponsor is a joint venture, an industry-wide scheme, a public-private part nership, or a broad multi-organizational network or ecosystem.
ACKNOWLEDGMENT
The term Multi-Stakeholder Risk Assessment was first used in a refereed article by the lead author relating specifically to the responsible application of AI [60]. Aspects of this article were presented at [49]. The authors acknowledge contributions to their thinking by their colleague and friend Roba Abbas.
References
[1] Y. K. Dwivedi et al., “Research on information systems failures and successes: Status update and future directions,” Inf. Syst. Frontiers, vol. 17, no. 1, pp. 143–157, Feb. 2015, doi: 10.1007/s10796-014-9500 y.
[2] R. Clarke and R. M. Davison, “Through whose eyes? The critical concept of researcher perspective,” J. Assoc. Inf. Syst., vol. 21, no. 2, pp. 483–501, Apr. 2020. [Online]. Available: https://aisel.aisnet.org/ jais/vol21/iss2/1
[3] R. Clarke, R. M. Davison, and W. Jia, “Researcher perspective in the IS discipline: An empirical study of articles in the basket of 8 journals,” Inf. Technol. People, vol. 33, no. 6, pp. 1515–1541, Oct. 2020, doi: 10.1108/itp-04-2019-0189.
[4] R. E. Freeman and D. L. Reed, “Stockholders and stakeholders: A new perspective on corporate governance,” California Manage. Rev., vol. 25, no. 3, pp. 88–106, Apr. 1983, doi: 10.2307/41165018.
[5] A. O. Laplume, K. Sonpar, and R. A. Litz, “Stakeholder theory: Reviewing a theory that moves us,” J. Manage., vol. 34, no. 6, pp. 1152–1189, Dec. 2008, doi: 10.1177/0149206308324322.
[6] S. Barrett and B. Konsynski, “Inter-organization information sharing systems,” MIS Quart., vol. 6, no. 4, pp. 93–105, Dec. 1982, doi: 10.2307/248993.
[7] R. Clarke, “Extra-organizational systems: A challenge to the soft ware engineering paradigm,” in Proc. IFIP Congr., no. 2, 1992, pp. 564–570.
[8] A. Pouloudi and E. A. Whitley, “Stakeholder identification in inter organizational systems: Gaining insights for drug use management systems,” Eur. J. Inf. Syst., vol. 6, no. 1, pp. 1–14, Mar. 1997, doi: 10.1057/palgrave.ejis.3000252.
[9] J. Berleur and J. Drumm, “Information technology assessment,” in Proc. 4th IFIP-TC9 Int. Conf. Hum. Choice Comput., Dublin, Ireland, Jul. 1990.
[10] S. F. H¨ubner and H. Lindskog, “Teaching privacy-enhancing technologies,” in Proc. IFIP WG 11.8 2nd World Conf. Inf. Secur. Educ. (WISE2), Perth, WA, Australia, Jul. 2001.
[11] K. Wahlstrom and G. Quirchmayr, “A privacy-enhancing archi tecture for databases,” J. Res. Pract. Inf. Technol., pp. 151–162, Aug. 2014. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.454.3815&rep=rep1&type=pdf
[12] E.P. S. Baumer, “Usees,” in Proc. 33rd Annu. ACM Conf. Hum. Factors Comput. Syst., Apr. 2015, pp. 3295–3298.
[13] I. Alexander, “A taxonomy of stakeholders: Human roles in sys tem development,” Int. J. Technol. Hum. Interact., vol. 1, no. 1, pp. 23–59, 2005. [Online]. Available: http://www.scenarioplus.org.uk/ papers/stakeholder taxonomy/stakeholder taxonomy.htm
[14] R. K. Mitchell, B. R. Agle, and D. J. Wood, “Toward a theory of stakeholder identification and salience: Defining the principle of who and what really counts,” Acad. Manage. Rev., vol. 22, no. 4, pp. 853–886, Oct. 1997, doi: 10.2307/259247.
[15] J. M. Bryson, “What to do when stakeholders matter: Stakeholder iden tification and analysis techniques,” Public Manage. Rev., vol. 6, no. 1, pp. 21–53, Mar. 2004. [Online]. Available: https://citeseerx.ist.psu.edu/ document?repid=rep1&type=pdf&doi=3aa8cd83844f0e07a93a995317 ca2a18d885fb99
[16] F. Ackermann and C. Eden, “Strategic management of stakehold ers: Theory and practice,” Long Range Planning, vol. 44, no. 3, pp. 179–196, Jun. 2011, doi: 10.1016/j.lrp.2010.08.001.
[17] M. C. Achterkamp and J. F. J. Vos, “Investigating the use of the stakeholder notion in project management literature, a meta-analysis,” Int. J. Project Manage., vol. 26, no. 7, pp. 749–757, Oct. 2007, doi: 10.1016/j.ijproman.2007.10.001.
[18] J. Ward and E. Daniel, Benefits Management: Delivering Value From is IT Investments. Hoboken, NJ, USA: Wiley, 2006.
[19] DoF. (Apr. 2020). Introduction to Benefits Management for Pro grammes and Projects and Stakeholder Engagement Within a Programme or Project. [Online]. Available: https://www.financeni.gov.uk/articles/programme-and-project-benefits-management
[20] R. Clarke, “Not only horses wear blinkers: The missing perspectives in IS research,” 2016, arXiv:1611.04059.
[21] R. Clarke, “Practicable backup arrangements for small organisations and individuals,” Australas. J. Inf. Syst., Sep. 2016, doi: 10.3127/ ajis.v20i0.1250.
[22] R. Abbas and K. Michael, Socio-Technical Theory: A Review. TheoryHub Book, 2022. [Online]. Available: https://open.ncl.ac.uk/ theories/9/socio-technical-theory/
[23] M. J. Schmidt. (2005). Business Case Essentials: A Guide to Struc ture and Content. [Online]. Available: http://www.solutionmatrix.de/ downloads/Business Case Essentials.pdf
[24] SoW.(2013). Discounted Cash Flow Analysis. [Online]. Available: https://www.streetofwalls.com/finance-training-courses/investment banking-technical-training/discounted-cash-flow-analysis/
[25] D. Dikov. (2020). Using the Net Present Value (NPV) in Financial Analysis. [Online]. Available: https://magnimetrics.com/net-present value-npv-in-financial-analysis/
[26] T. Stobierski. (Sep. 2019). How to Do a Cost-Benefit Analysis & Why It’s Important. [Online]. Available: https://online.hbs.edu/blog/ post/cost-benefit-analysis
[27] Information Security, Cybersecurity and Privacy Protection—Guidance on Managing Information Security Risks, Standard ISO 27005:2022, 2022. [Online]. Available: https://www.iso.org/standard/80585.html
[28] Information Technology-Security Techniques-Information Security Risk Management, Standard ISO 27005:2011, 2011. [Online]. Avail able: https://www.iso.org/standard/56742.html
[29] OTA. (Jan. 1977). Technology Assessment in Bus. and Government. [Online]. Available: http://www.princeton.edu/
[30] L. Garcia, “The U.S. office of technology assessment,” in Information Technology and Assessment (eds.). Amsterdam, The Netherlands: North Holland, 1991, pp. 177–180.
[31] R. K. Morgan, “Environmental impact assessment: The state of the art,” Impact Assessment Project Appraisal, vol. 30, no. 1, pp. 5–14, Mar. 2012, doi: 10.1080/14615517.2012.661557.
[32] R. Clarke, “Privacy impact assessment: Its origins and development,” Comput. Law Secur. Rev., vol. 25, no. 2, pp. 123–135, Apr. 2009, doi: 10.1016/j.clsr.2009.02.002.
[33] D.Wright and P. De Hert, Privacy Impact Assessments. Cham, Switzer land: Springer, 2012.
[34] H.Becker and F. Vanclay, The International Handbook of Social Impact Assessment. Cheltenham, U.K.: Edward Elgar, 2003.
[35] UoO.(2013). Bringing Children in From the Margins: Symposium on Child Rights Impact Assessments, Child Rights Impact Assessment: A Tool to Focus on Children. [Online]. Available: http://www.unicef.ca/sites/default/files/ imce uploads/report from cria symposium may2013 canada.pdf
[36] D. Wright and C. D. Raab, “Constructing a surveillance impact assessment,” Comput. Law Secur. Rev., vol. 28, no. 6, pp. 613–626, Dec. 2012, doi: 10.1016/j.clsr.2012.09.003.
[37] D. Wright, M. Friedewald, and R. Gellert, “Developing and testing a surveillance impact assessment methodology,” Int. Data Privacy Law, vol. 5, no. 1, pp. 40–53, Feb. 2015, doi: 10.1093/idpl/ipu027.
[38] D. Schuler and A. Namioka, Participatory Design: Principles Prac tices. Boca Raton, FL, USA: CRC Press, 1993.
[39] S. L. Pan and L. G. Pee, “Usable, in-se, and useful research: A 3U framework for demonstrating practice impact,” Inf. Syst. J., vol. 30, no. 2, pp. 403–426, Mar. 2020, doi: 10.1111/isj.12274.
[40] N. Racz, E. Weippl, and A. Seufert, “A frame of reference for research of integrated governance, risk and compliance (GRC),” in Communications and Multimedia Security, vol. 6109. Berlin, Germany: Springer, 2010, doi: 10.1007/978-3-642-13241-4 11.
[41] Information vacy Security, Cybersecurity Protection—Information Security and Pri Management Systems—Requirements, Standard ISO/IEC 27001, 2022, p. 2022. [Online]. Available: https://www.iso.org/standard/27001
[42] The NIST Cybersecurity Framework (CSF) 2.0, NIST, Gaithersburg, MD, USA, Feb. 26, 2024.
[43] ISACA.(2025). COBIT: An ISACA Framework. [Online]. Available: https://www.isaca.org/resources/cobit
[44] SABSA,. Enterprise Security Architecture. Accessed: Nov. 6, 2018. [Online]. Available: https://sabsa.org/sabsa-executive-summary/#how is-sabsa-used
[45] D. Landoll, The Security Risk Assessment Handbook: A Complete Guide for Performing Security Risk Assessments. Boca Raton, FL, USA: CRC Press, 2021.
[46] Guide for Conducting Risk Assessments, document NIST SP 800-30 Rev., NIST, Sep. 2012. [Online]. Available: https://csrc.nist.gov/ publications/detail/sp/800-30/rev-1/final
[47] European Commission. (Jun. 2016). EU General Risk Assess ment Methodology. [Online]. Available: https://ec.europa.eu/docsroom/ documents/17107/attachments/1/translations/en/renditions/native
[48] Risk Management—Risk Assessment Techniques, Standard IEC 31010:2019, 2019. [Online]. Available: https://www.iso.org/standard/ 72140.html
[49] R. Clarke and K. Michael, “Multi-stakeholder risk assessment of socio-technical system projects,” in Proc. Australas. Conf. Inf. Syst., Univ. Canberra, Dec. 2024, pp. 1–17. [Online]. Available: https:// aisel.aisnet.org/acis2024/1
[50] R. Clarke, “The prospects of easier security for small organisations and consumers,” Comput. Law Secur. Rev., vol. 31, no. 4, pp. 538–552, Aug. 2015, doi: 10.1016/j.clsr.2015.05.004.
[51] A. R. Hevner, S. T. March, J. Park, and S. Ram, “Design science in information systems research,” MIS Quart., vol. 28, no. 1, pp. 75–105, Jan. 2004. [Online]. Available: https://misq.umn.edu/design-science-in information-systems-research.html
[52] F. Niederman and S. T. March, “Design science and the accumulation of knowledge in the information systems discipline,” ACM Trans. Manage. Inf. Syst., vol. 3, no. 1, pp. 1–15, Apr. 2012, doi: 10.1145/ 2151163.2151164.
[53] S. Gregor and A. R. Hevner, “Positioning and presenting design science research for maximum Impact1,” MIS Quart., vol. 37, no. 2, pp. 337–355, Jun. 2013. [Online]. Available: https://aisel.aisnet.org/ misq/vol37/iss2/3
[54] R. L. Baskerville, “Investigating information systems with action research,” Commun. AIS, vol. 2, no. 19, pp. 2–32, 1999. [Online]. Available: https://aisel.aisnet.org/cais/vol2/iss1/19
[55] F. E. Emery and E. L. Trist, “Socio-technical systems,” in Management Sciences, Models and Techniques, vol. 2. Oxford, U.K.: Pergamon Press, 1960, pp. 83–97.
[56] E. Mumford, “A socio-technical approach to systems design,” Require ments Eng., vol. 5, no. 2, pp. 125–133, Sep. 2000, doi: 10.1007/ pl00010345.
[57] T. Brown. (Jun. 2008). Design Thinking. Harvard Bus. Review. [Online]. Available: https://shala4lio.wordpress.com/wp-content/ uploads/2015/03/design-thinking.pdf
[58] L. Kimbell, “Rethinking design thinking,” Annu. Rev. Policy Design, vol. 8, no. 1, pp. 1–20, 2011. [Online]. Available: https://ojs.unbc.ca/ index.php/design/article/download/1812/1369
[59] K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A design science research methodology for information systems research,” J. Manage. Inf. Syst., vol. 24, no. 3, pp. 45–77, Dec. 2007, doi: 10.2753/mis0742-1222240302.
[60] R. Clarke, “Principles and business processes for responsible AI,” Comput. Law Secur. Rev., vol. 35, no. 4, pp. 410–422, Aug. 2019, doi: 10.1016/j.clsr.2019.04.007.
[61] K. Michael, R. Abbas, and J. Pitt, “Maintaining control over AI,” Issues Sci. Technol., Forum, vol. 37, no. 3, pp. 1–3, 2021. [Online]. Available: https://issues.org/debating-human-control-over artificial-intelligence-forum-shneiderman '
[62] F. Emery, “Characteristics of socio-technical systems,” in The Social Engagement of Social Science, a Tavistock Anthology, Volume 1: The Socio-Psychological Perspective. Philadelphia, PA, USA: University of Pennsylvania Press, 2016, pp. 157–186.
[63] E. Mumford, “The story of socio-technical design: Reflections on its successes, failures and potential,” Inf. Syst. J., vol. 16, no. 4, pp. 317–342, Oct. 2006, doi: 10.1111/j.1365-2575.2006.00221.x.
[64] R. Abbas, J. Pitt, and K. Michael, “Socio-technical design for pub lic interest technology,” IEEE Trans. Technol. Soc., vol. 2, no. 2, pp. 55–61, Jun. 2021, doi: 10.1109/TTS.2021.3086260.
[65] L. von Bertalanffy, General System Theory: Foundations, Development, Applications. New York, NY, USA: George Braziller, 1968.
[66] K. Michael, K. M. Vogel, J. Pitt, and M. Zafeirakopoulos, “Artificial intelligence in cybersecurity: A socio-technical framing,” IEEE Trans. Technol. Soc., vol. 6, no. 1, pp. 15–30, Mar. 2025, doi: 10.1109/ TTS.2024.3460740.
[67] W. Xu and Z. Gao, “An intelligent sociotechnical systems (iSTS) framework: Enabling a hierarchical human-centered AI (hHCAI) approach,” IEEE Trans. Technol. Soc., vol. 6, no. 1, pp. 31–46, Mar. 2025, doi: 10.1109/TTS.2024.3486254.
[68] R. P. Bostrom and J. S. Heinen, “MIS problems and failures: A socio technical perspective,” MIS Quart., vol. 1, no. 3, pp. 17–32, Sep. 1977, doi: 10.2307/248710.
[69] B. Langefors, Theoretical Analysis of Information Systems. New York, NY, USA: Auerbach, 1973.
[70] G. B. Davis, Management Information Systems: Conceptual Founda tions, Structure, and Development. New York, NY, USA: McGraw-Hill, 1974.
[71] M. Keil, P. E. Cule, K. Lyytinen, and R. C. Schmidt, “A framework for identifying software project risks,” Commun. ACM, vol. 41, no. 11, pp. 76–83, Nov. 1998, doi: 10.1145/287831.287843.
[72] C. Sauer, Why Information Systems Fail: A Case Study Approach. London, U.K.: Alfred Waller, 1993.
[73] R. Clarke, K. Michael, and R. Abbas, “Robodebt: A socio-technical case study of public sector information systems failure,” Australas. J. Inf. Syst., vol. 28, pp. 1–42, Sep. 2024, doi: 10.3127/ajis.v28.4681.
[74] P. Checkland, Systems Thinking, Systems Practice. Chichester, U.K.: Wiley, 1981.
[75] P. Checkland and J. Scholes, Soft Systems Methodology in Action. Chichester, U.K.: Wiley, 1990.
[76] E. Mumford, Effective Systems Design and Requirements Analysis: The ETHICS Method. London, U.K.: Macmillan, 1995.
[77] F. F. Land and R. A. Hirschheim, “Participative systems design: Ratio nale, tools and techniques,” J. Appl. Syst. Anal., vol. 10, pp. 91–107, Jul. 1983.
[78] A. Lagna and M. N. Ravishankar, “Making the world a better place with fintech research,” Inf. Syst. J., vol. 32, no. 1, pp. 61–102, Jan. 2022, doi: 10.1111/isj.12333.
[79] G. Walsham, “Are we making a better world with ICTs? Reflections on a future agenda for the IS field,” J. Inf. Technol., vol. 27, no. 2, pp. 87–93, Jun. 2012, doi: 10.1057/jit.2012.4.
[80] G. George, J. Howard-Grenville, A. Joshi, and L. Tihanyi, “Understanding and tackling societal grand challenges through man agement research,” Acad. Manage. J., vol. 59, no. 6, pp. 1880–1895, Dec. 2016, doi: 10.5465/amj.2016.4007.
[81] ISJ.(2019). Call for Papers: Special Issue on Responsible IS Research for a Better World. [Online]. Available: https://onlinelibrary.wiley.com/ page/journal/13652575/homepage/special issues.htm
[82] R. M. Davison, A. Majchrzak, A. Hardin, and M.-N. Ravishankar, “Special issue on responsible IS research for a better world,” Inf. Syst. J., vol. 23, no. 1, pp. 1–7, Jan. 2023, doi: 10.1111/isj.12405.
[83] S. Ahuja, Y. E. Chan, and R. Krishnamurthy, “Responsible innovation with digital platforms: Cases in India and Canada,” Inf. Syst. J., vol. 33, no. 1, pp. 76–129, Jan. 2023, doi: 10.1111/isj.12378.
[84] A. Majchrzak, M. L. Markus, and J. Wareham, “Designing for digital transformation: Lessons for information systems research from the study of ICT and societal challenges,” MIS Quart., vol. 40, no. 2, pp. 267–277, Jun. 2016, doi: 10.25300/misq/2016/40:2.03.
[85] J. Patel, L. Giddens, P. Wang, A. G. Young, and S. Ahuja. (2024). Fostering Responsible IS Research. [Online]. Available: https:// aisel.aisnet.org/amcis2024/panels/panels/2
[86] RRBM.(2017). A Vision of Responsible Research in Bus. and Man agement: Striving for Useful and Credible Knowledge: Position Paper. [Online]. Available: https://rrbm.network/wp-content/uploads/2020/04/ Position-Paper revised 8April2020.pdf
[87] D. Dennehy et al., Eds., “Responsible AI and analytics for an ethical and inclusive digitized society,” in Proc. 20th IFIP WG 6.11 Conf. E Bus., E-Services E-Soc., 2021, pp. 1–787. [Online]. Available: https:// inria.hal.science/hal-03648105/document
[88] W.-K. Lim, S. Sia, and A. Yeow, “Managing risks in a failing IT project: A social constructionist view,” J. Assoc. Inf. Syst., vol. 12, no. 6, pp. 414–440, Jun. 2011, doi: 10.17705/1jais.00269.
[89] J. Pries-Heje, J. Venable, and R. Baskerville, “RMF4DSR: A risk man agement framework for design science research,” Scandin. J. Inf. Syst., vol. 26, no. 1, p. 3, 2014. [Online]. Available: http://aisel.aisnet.org/sjis
[90] S. Strecker, D. Heise, and U. Frank, “RiskM: A multi-perspective modeling method for IT risk assessment,” Inf. Syst. Frontiers, vol. 13, no. 4, pp. 595–611, Sep. 2011, doi: 10.1007/s10796-010-9235-3.
[91] D. Gotterbarn and S. Rogerson, “Responsible risk assessment with software development: Creating the software development impact statement,” Commun. Assoc. Inf. Syst., vol. 15, no. 40, pp. 703–750, 2005, doi: 10.1145/265684.265699.
[92] R. Clarke, “A risk assessment framework for mobile payments,” in Proc. 21st Bled eCommerce Conf., Jun. 2008, pp. 3–77. [Online]. Available: https://domino.fov.um.si/proceedings.nsf/0/ fc5aa5c853a1cf3dc1257481003d0293/
[93] C. Prell, K. Hubacek, and M. Reed, “Stakeholder analysis and social network analysis in natural resource management,” Soc. Natural Resour., vol. 22, no. 6, pp. 501–518, Jun. 2009. [Online]. Available: https://www.idahoecosystems.org/sites/default/files/ literature resource/Prell StakeholderSNAinResourceManagment 2009 1.pdf
[94] R. Molarius, M. R¨aikk¨onen, K. Forss´en, and K. M¨aki, “Enhancing the resilience of electricity networks by multi-stakeholder risk assessment: The case study of adverse winter weather in Finland,” J. Extreme Events, vol. 3, no. 4, Dec. 2016, Art. no. 1650016, doi: 10.1142/ s2345737616500160.
[95] M. Mollaeefar, A. Siena, and S. Ranise, “Multi-stakeholder cyber security risk assessment for data protection,” in Proc. 17th Int. Joint Conf. E-Bus. Telecommun., 2020, pp. 349–356, doi: 10.5220/ 0009822703490356.
[96] R. C. Young and E. Jordan, “IT governance and risk management: An integrated multi-stakeholder framework,” Asia Pacific Decis. Sci. Inst., Bangkok, Thailand, 2002.
[97] S. J. Shackelford and S. Russell, “Operationalizing cybersecu rity due diligence: A transatlantic case study,” South Carolina Law Rev., vol. 67, no. 3, p. 7, 2016. [Online]. Available: https:// scholarcommons.sc.edu/sclr/vol67/iss3/7
[98] H. T. Nelson, S. Hass, K. Sarle, and A. Renirie, “Communities of place vs communities of interest in the United States: Citizen information and locally unwanted land uses in EIA,” Environ. Impact Assessment Rev., vol. 87, Mar. 2021, Art. no. 106534, doi: 10.1016/j.eiar.2020.106534.
[99] Medical Devices—Application of Risk Management to Medical Devices, Standard ISO 14971:2019, 2019. [Online]. Available: https:// www.iso.org/standard/72704.html
[100] R. Clarke and M. Jenkins, “The strategic intent of on-line trading systems: A case study in national livestock marketing,” J. Strategic Inf. Syst., vol. 2, no. 1, pp. 57–76, Mar. 1993, doi: 10.1016/0963 8687(93)90023-4.
[101] J. Cameron, “An integrated framework for managing eBusiness col laborative projects,” Ph.D. thesis, School Comput. Sci. Eng., UNSW Sydney, Sydney, NSW, Austrlia, Sep.2009. [Online]. Available: https:// doi.org/10.26190/unsworks/2716
[102] B. S. Neo, “The implementation of an electronic market for pig trading in Singapore,” J. Strategic Inf. Syst., vol. 1, no. 5, pp. 278–288, Dec. 1992.
[103] D. Davlembayeva, R. Abbas, D. Laffey, K. Michael, and S. Papagiannidis, “The current state and future trajectory of the sharing economy: A multi-stakeholder perspective,” Int. J. Bus. Sci. Appl. Manage., vol. 19, no. 1, pp. 1–29, 2024. [Online]. Available: https://www.business-and-management.org/papers/ the-current-state-and-future-trajectory-of-the-sharing-economy-a multi-stakeholder-perspective/180
[104] K. M. Wyman, “Taxi regulation in the age of uber,” N.Y.U. J. Legislation Public Policy, vol. 2, no. 1, pp. 1–100, Apr. 2017. [Online]. Available: https://www.nyujlpp.org/wp-content/uploads/2017/ 04/Wyman-Taxi-Regulation-in-the-Age-of-Uber-20nyujlpp1.pdf
[105] R. Clarke, “Research opportunities in the regulatory aspects of elec tronic markets,” Electron. Markets, vol. 32, no. 1, pp. 179–200, Mar. 2022, doi: 10.1007/s12525-021-00469-0.
[106] K. Michael, “Modern indentured servitude in the gig economy: A case study on the deregulation of the taxi industry in the United States,” IEEE Technol. Soc. Mag., vol. 41, no. 2, pp. 30–41, Jun. 2022, doi: 10.1109/MTS.2022.3173306.
[107] E. M. Rogers, Diffusion of Innovations, 3rd ed., New York, NY, USA: Free Press, 1962.
[108] J. Hedman and S. Henningsson, “Developing ecological sustainability: A green IS response model,” Inf. Syst. J., vol. 26, no. 3, pp. 259–287, May 2016, doi: 10.1111/isj.12095.
[109] UNPRI.(2006). What Are the Principles for Responsible Invest ment?. [Online]. Available: https://www.unpri.org/about-us/what-are the-principles-for-responsible-investment
[110] S. Akter et al., “Algorithmic bias in data-driven innovation in the age of AI,” Int. J. Inf. Manage., vol. 60, Oct. 2021, Art. no. 102387, doi: 10.1016/j.ijinfomgt.2021.102387.
[111] K. Michael, “In this special section: Algorithmic bias—Australia’s robodebt and its human rights aftermath,” IEEE Trans. Tech nol. Soc., vol. 5, no. 3, pp. 254–263, Sep. 2024, doi: 10.1109/ TTS.2024.3444248.
[112] R. Clarke, “Appendix E: Privacy impact assessments: international study of their application and effects,” Xamax Counsultancy, Canberra, ACT, Australia, Oct. 2007, pp. 1–53. [Online]. Available: https:// www.rogerclarke.com/DV/ICOStudy-2007-AppE Aus.pdf
[113] R. Abbas, K. Michael, and M. Michael, “The regulatory considerations and ethical dilemmas of location-based services (LBS): A literature review,” Inf. Technol. People, vol. 27, no. 1, pp. 2–20, Feb. 2014, doi: 10.1108/itp-12-2012-0156.
[114] K. Michael, S. Kobran, R. Abbas, and S. Hamdoun, “Privacy, data rights and cybersecurity: Technology for good in the achievement of sustainable development goals,” in Proc. IEEE Int. Symp. Technol. Soc. (ISTAS), Medford, MA, USA, Nov. 2019, pp. 1–13, doi: 10.1109/ ISTAS48451.2019.8937956.
[115] R. Clarke, “Technological impact assessment: A framework for design ing a TIA project method,” IEEE Trans. Techno. Soc., early access, Jul. 18, 2025, doi: 10.1109/TTS.2025.3574863.
Authors
Roger Clarke received the degrees in information systems and information technology from The University of New South Wales and The Australian National University. He is currently a Consultant in strategic and policy aspects of transformational technologies and a Principal of Xamax Consultancy Pty Ltd., Canberra, Australia. He is also a Visiting Professor in computer science with The Australian National University and in technology law with UNSW Law & Justice. He is a Fellow of the Australian Computer Society and the Association for Information Systems.
Xamax Consultancy Pty Ltd., Chapman, ACT, Australia
The Australian National University, Canberra, ACT, Australia
UNSW Law & Justice, Sydney, NSW, Australia
(Senior Member, IEEE) is a Professor of Strategy, Innovation and Technology at the University of Sydney Business School where she is the Inaugural Director of the MBA (Technology and Digital Strategy) program. Previously, she was a joint tenured Professor with the School for the Future of Innovation in Society and the School of Computing and Augmented Intelligence at Arizona State University, and was also the Associate Dean International at the University of Wollongong, Aus tralia, where she began her academic career in 2001. She is also the founding Editor-in-Chief of the IEEE TRANSACTIONS ON TECHNOLOGY AND SOCIETY
The University of Sydney, Darlington, NSW, Australia
University of Sydney Business School, Sydney, Australia
Citation: R. Clarke and K. Michael, "Multi-Stakeholder Risk Assessment of Socio-Technical Interventions," in IEEE Transactions on Technology and Society, doi: 10.1109/TTS.2026.3703434.