Facial Emotion Recognition in the Future of Work
Citation: E. Hajric, F. N. Arevalo, L. Bruce, F. A. Smith and K. Michael, "Facial Emotion Recognition in the Future of Work: Social Implications and Policy Recommendations," in IEEE Transactions on Technology and Society, vol. 6, no. 3, pp. 295-304, Sept. 2025, doi: 10.1109/TTS.2024.3477512.
Abstract
Facial biometric systems potentially allow for the overt and covert detection of a person for a range of use case scenarios. This article considers a human resource management (HRM) workplace scenario where employees are monitored through cameras on personal electronic devices for the purposes of facial emotion recognition. The applications described pertain broadly to the “future of work” context. The article considers how employers, would use employee facial emotion data for data-driven decision-making in, for example, the construction and optimization of virtual teams, appropriateness for promotion to leadership positions, and fitness-to-task in mission critical work. Building on the outcomes of a socio-technical study, the initial component of which was an FER prototype, this paper considers the social implications and policy recommendations of the deployment of the technical system. Findings indicate that coded biases in determinations of FER include possible discrimination against women, racial minorities, undocumented immigrants and refugees, and people with visible and invisible disabilities.
SECTION I.
Introduction: Cameras, Biometrics, Ai and Emotion
Facial biometric systems potentially allow for the covert detection of a person for a range of use case scenarios. Here we are referring to more than facial recognition systems for identity management and authentication purposes. We consider human emotions being detectable, readable, and interpretable by a machine and corresponding visual interfaces [1]. It is contestable that machines, through vision systems, will lip read as HAL9000 was once depicted doing in Stanley Kubrick’s classic interpretation of Arthur C. Clarke’s 2001: A Space Odyssey[2]. And whether machines absorb emotion detection through facial expressions and hand gestures, as depicted in José Padilha’s 2014 Robocop[3] remains to be seen. But computers using artificial intelligence (AI) are now touted as having the human traits of understanding emotion behavior, and possibly even inferring reason as a result. In the recent launch of GPT-4o we noted its computer vision capabilities when it analyzed a selfie of host Zoph and inferred his emotional state as “pretty happy and cheerful… with a big smile and maybe a touch of excitement” [4].
Taking the HAL9000 and Robocop metaphors and applying them to a human resource management (HRM) workplace scenario where employees are monitored through cameras on personal electronic devices for the purposes of facial emotion recognition (FER) allows for the examination of the possible social policy implications. The applications described in this article pertain broadly to the “future of work” (FOW) context. This article considers how employers, would use employee facial emotion data for data-driven decision-making in, for example, the construction and optimization of virtual teams, appropriateness for promotion to leadership positions, and fitness-to-task in mission critical work. The authors have previously conducted a socio-technical study, the initial component of which was a FER prototype [1]. This paper considers the social implications of FER and presents some policy recommendations for the deployment of the technical system. Findings indicate that coded biases in determinations of FER include possible discrimination against women, racial minorities, undocumented immigrants and refugees, and people with disabilities.
A. Facial Emotion Recognition, HRM and the Future of Work
Biometric systems are designed and developed for the purpose of detecting, measuring, and analyzing both the physical and behavioral characteristics of individuals, with facial characteristics providing unique opportunities for unobtrusive and covert collection of identifying information pertaining to gender, race, and relevant to this article, one’s emotional state, in addition to other identifying details. Automated biometric systems based on facial recognition can be traced back to the 1980s. Since that time, systems have become increasingly sophisticated, with application in a variety of contexts inclusive of public spaces for surveillance purposes. In corporate HRM and work-related applications, biometrics are now being used to gauge employee productivity, mood, and emotion [5], the latter of which is the focus of this paper.
The impact of the affective state of the employees in a workplace on job outcomes is well studied [6]. However, as this information can be very subjective, it is difficult to accurately analyze the state of an individual in a workplace context. Traditionally, a slew of employee engagement and human resource management strategies have been utilized to address this issue. The attitude of employees towards collective tracking of emotional self-reports has been shown to be positive in some studies [5], but it is important to note that this is contingent on the consent of the employees and protection of their individual identities.
Using enterprise software to gauge employee mood is possible, however, these programs do not provide factual evidence of collective mood [7]. Direct tracking of physiological data of employees via wearables is also possible but may lead to additional user experience and privacy concerns. Many behavioral studies by psychologists have taken place, for instance, in understanding how human performance may be impacted by the duration of a shift, particularly overnight or in complex settings. Facial expressions are a dominant form of human communication and facial microexpressions can be used to detect emotions [8]. Using low-cost webcams to obtain these facial expressions and extracting features to estimate the underlying emotions has previously been demonstrated [9]. For instance, EmEx is a system that has been developed to recognize human emotions of individuals living with a disability and enable better communication, specifically in healthcare scenarios [10], which can potentially be applied in the workplace to make communication more accessible. Applying facial pattern recognition for cognitive ergonomics has also been demonstrated to prevent workplace accidents stemming from human error [11].
It is suggested that by using an automated FER to gauge the mood of the employees in a digitally-enabled workplace, a more empathetic work environment could be designed. Furthermore, developing and deploying a system that is seamless, inexpensive, and secure, could lead to improved employee satisfaction and performance. Despite these benefits, and the identified opportunities of FER in the workplace, there is the need to determine the social implications and corresponding policy requirements or recommendations, if these systems are to be considered for current and future of work HRM applications. This article seeks to present the social implications and policy recommendations to balance and complement technical studies, following on from the initial technical phase of the study [1]. As such, the background in view of ethics and workplace surveillance will be presented in the following section and will be elaborated on using several surveillance metaphors thereafter. An overview of the research design is then presented along with the findings of the study. Lastly, policy recommendations are provided as outcomes of the work with a brief conclusion.
SECTION II.
Background
A. Ethics and Workplace Surveillance
Facial recognition technologies, in general, have raised ethical concerns as bias, politics, and existing socio-technical dynamics become exacerbated in their negative impacts on society and human rights. Researchers, Congressional hearings, cities, and the public have all unified in demonstrating the seriousness of these ethical concerns and instituted bans on public uses of facial recognition systems in several cities such as Oakland, California, and Portland, Oregon. These systems, however, are still being developed and employed for private use, including disturbing plans for workplace surveillance, for example, Microsoft’s productivity score monitoring work behavior [12]. Additionally, some companies are seeking to include systemic uberveillance in the form of emotional recognition monitoring of their employees [13]. Technologies, such as those developed in the context of this study [1], can be applied in intrusive and unjust ways beyond their original intent at the point of development. When technologies are developed overlooking the social implications, merely focused on top-down data-driven strategies by management, the outcome will be fraught with complexities and uncertainties, that may eventuate in employee complaints, penalties for breaching legislation, and even high rates of staff turnover. The implementation of any new technology requires the consultation of diverse stakeholder groups for input at the forefront of any systems/design initiative to ensure not only the awareness and consent of workers, but also how to mitigate against unintended harmful effects.
There are many elements of worker’s rights and human rights that need to be considered and centered before use of such emotion monitoring technology is accepted or normalized in both future of work contexts and more broadly in society. As insider attacks in organizations are on the rise, more evidence-based methodologies are being sought as a means to conduct human behavior analytics within the function of electronic human resource management (e-HRM). Some of these approaches are yet to be proven and require further testing, and in other cases the approaches would be an obvious breach in human rights and ones autonomy. Furthermore, the interdisciplinary requirements of FER systems necessitate a thorough assessment of the underlying technology to ensure that disciplinary expertise is incorporated and that the resulting system or application is both technically and scientifically sound. This is critical as these technologies are being touted as having “emotion detection” capabilities and associated inferences have been proven to be scientifically unfounded, have been debunked, and are based on disputed phrenology studies at odds with current psychological expertise. One of the key origins of this emotion-reading framework is psychologist Paul Ekman and Wallace V. Friesen’s Facial Action Coding System (FACS) developed in 1978 [45]. Under FACS, facial muscle movements are individually measured as Action Units (AUs) presuming seven universal human emotions. This contests the robustness and complexity of the human experience, and undercuts the reliability of FER systems reducing emotions.
As another example of possible contention in the use of FER, a recently published study utilizing AI to determine trustworthiness based on facial features, received ample backlash, rendering it a practice of racial profiling and superficial understanding [14]. Emerging technologies must incorporate social scientists and other disciplinary perspectives in the design and development process to avoid siloed thinking that results in inaccurate understandings of human emotions and overestimates AI capabilities to compute and predict behavior. Additionally, there are ethical aspects to consider when designing such systems that exhibit surveillance mechanisms. Humans have more than seven emotions. These emotions are complex and cannot be accurately read by human beings without direct communication, making it increasingly challenging to replicate these emotions in systems.
Several important questions must be posed, specifically in a workplace or future of work context: Why would a workplace or private business need to inspect the emotions of employees? Is there a need to focus on interpersonal and managerial skills rather than technosolutionism? How do we consult in the design and development of these systems? Is there the potential for socio-technical studies that seek to balance the social and technical aspects? What will the future of work and of society look like if these systems are designed and implemented at scale? These questions can only be answered with a balanced assessment of the technological potential and the social implications, the latter of which includes recognition of the probable intrusiveness of these systems and the requirement for the protection of worker rights [13]. If these technologies are driven by a capitalist focus on productivity, they can create environments where this focus may outweigh workers’ rights, resulting in the conceivable act of continual employee surveillance. This form of watching may even extend to other stakeholders, including interactions between employees and customers and suppliers.
Workplace surveillance is an existing phenomenon that primarily benefits corporations in productivity monitoring with the purpose of meeting production needs. As pointed out by Jathan Sadowski workers sometimes rebel against these forces of surveillance in micro ways, termed “microresistance,” for example, disrupting a battery that is charging so a worker can get a short break while the manager gets a new one [15, p. 168]. Surveillance in the workplace deteriorates trust between a corporation and its employees by assuming employees cannot be trusted when left unsupervised. Levels of technological surveillance already exist in forms of tracking and monitoring logins, keystrokes and mouse movements in software and computing devices, email monitoring, and Web browsing history. There are also more traditional physical surveillance such as clocking-in, frequency and time spent during restroom or lunch breaks, existing surveillance cameras, and monitoring of vehicle use and parking. The addition of FER poses a potential challenge of over-surveillance in the workplace and is detrimental on workplace health.
The denigration of workers as if they are machines rather than humans, additionally creates low morale in the workplace. As a result of poor treatment, there are cascading effects on mental and physical health, and the overall health of the workplace environment [16]. Ultimately, the focus on productivity and surveillance ends up costing productivity gains long-term because employees are not taken care of, nor respected. Implementing technological surveillance devices for near continuous monitoring exacerbates these dynamics. A way to disrupt this is to include workers in the decision-making process for designing and implementing these technologies, and to rethink whether they are needed in the first place, or whether specific challenges being faced are social and as such require social remediation.
Building monitoring tools in a workplace or future of work context can potentially create a panopticon of surveillance, explained in the following section. It is possible that in such a future work context, management may wish to capture every word uttered, every movement made by a human in the name of process optimization and intellectual property, and greater productivity through speech-to-text conversions. Such data can be stored and processed on the premise of a better understanding of a workplace’s pulse, mood, potential to share, learn and succeed. Human resource managers may claim that by analyzing individual and team dynamics, the best configuration of worker skill sets might be achieved toward the maximum gains for the organization. Enter productivity scoring tools that are changing the face of human resources but not always obtrusive to the worker [13]. For many who are already concerned with the watering down of workplace surveillance laws (if they exist at all), the proliferation of fixed or handheld devices that can capture audio-visual content and people surreptitiously recording another through sousveillant activity is alarming [17], [18], [19]. Is a workplace a public space? And if so, can there be an expectation of privacy in public? The argument made by executives of organizations is that with the use of better analytics, workforce readiness toward dynamic capabilities in the organization can be better captured and that workforce planning can be improved aligning with business transformation strategies.
B. Big Brother Vs Little Brothers
There are two overarching conceptual framings that may be used to explain the potential for pervasive and persistent workplace surveillance through the adoption of modern technologies: (1) “Big Brother”, a term popularized by George Orwell’s 1984[20]; and (2) “Little Brothers”, an idea that likely emerged in the mid-70s with varying attribution [21]. Surveillance cameras affixed to the exterior of buildings protecting businesses are said to be a type of “big brother” overseeing the entry of a physical premises, whereas “little brothers” may be mobile cameras that can move around capturing anything in their direct surrounds.
Basic surveillance cameras have been used for decades to ensure that expensive assets and company inventory are not stolen by employees or professional thieves (e.g., in the protection of an organization’s intellectual property). Computer vision systems are increasingly being used to detect cases of fraud and industrial espionage in white collar worker settings. Even emergency services personnel are monitored on deployment. In the latter case, we have gone from first responder in-building surveillance, through to the use of in-vehicle dashcams, to wearable technologies providing a first-person point-of-view with audio-visual evidence. More than anything else, the new technologies are not just present for worker occupational health and safety (OH&S) but for security, incident handling, company liability, and reporting.
Artificial intelligence (AI) and machine learning (ML) have provided a means by which to analyze the data that is collected from surveillance cameras, particularly where the recorded footage has limited “noise” (e.g., is taped in controlled settings with good lighting and no rain or wind). In some instances, the recorded visual evidence is sent straight to the cloud for further processing, and in other cases annotation may happen offshore displaying worker movements and tracking other kinds of interactions. Today, however, some companies are claiming they can detect emotions by studying the facial expressions of the human subject recorded [22], [23], [24].
Jeremy Bentham’s “Panopticon” is the prominent conceptual framework used to describe workplace surveillance. The 18th century philosopher published the idea in 1791, in the Panopticon Or the Inspection House. Fig. 1 depicts a circular prison interior within a regular shaped building. Situated at the central observation tower is a guard who can oversee all the prisoners in their gated cells. Nothing is hidden, nothing is private from that vantage point. The geographic layout of the prison somehow maintains the behavior of inmates, as it will keep them guessing whether the guard is in actual fact observing their cell or another. This description stands as a metaphor for maintaining control among the masses. Studies have shown, that the introduction of closed-circuit television (CCTV) in public spaces, causes a chilling effect among gangs and those wishing to break the law (e.g., drug dealers) [25], [26], [27].
Fig. 1. Illustration of the Panopticon, as drawn by Willey Rively in 1791. Source: Jeremy Bentham - The works of Jeremy Bentham vol. IV, 172–3 https://en.wikipedia.org/wiki/Panopticon#/media/File:Panopticon.jpg.
A contrast to the Panopticon is a metaphor made prominent in 2001 with the publication of the article titled: “Privacy and Power: Computer Databases and Metaphors for Information Privacy” by law professor Daniel Solove [28]. It relies on a Kafkaesque interpretation of surveillance taken from The Trial, and aligns better to the emergence of pinhole cameras prevalent on most computing devices (e.g., tablets and laptops) in addition to mobile devices (e.g., smartphones and wearables). Solove writes: “[u]nderstood with the Kafka metaphor, the problem is the powerlessness, vulnerability, and dehumanization created by the assembly of dossiers of personal information where individuals lack any meaningful form of participation in the collection and use of their information… conceptualizing the problem with the Kafka metaphor has profound implications both for the law of information privacy and for choosing legal approaches to solve the problem” (cited in [29, p. 3]). Interconnectivity today means that data can be gathered by a single organization that can share access of the footage using rights controls among its members, both internal or external to the organization. The idea of the Kafka metaphor [46] aligns from the perspective that individual employees know that data can be gathered about them in the future of work context from a variety of vectors, and a variety of camera recording devices with the potential to be analyzed in many different ways.
Steve Mann has previously noted that the recording of one’s human activities of which they are a participant means that one has the capacity to self-correct behavior, even if they had thoughts of wrong-doing. Known as sousveillance, Mann has demonstrated in the past, through initiatives like Glogger.Mobi, that publishing one’s video feed live to the Internet further maintains positive behavior and also a continuous alibi. However, is this entirely true? Is it possible that some employees might forget the cameras are rolling, desensitized by the pervasive nature of too many recording devices, relegating their fundamental freedoms not to incriminate themselves during times of unexpected provocation? Video footage can lie [47]. Audio-visual recordings do not capture the extent of the full story. They have a start and stop time that may preclude a given incident that explains actions of given actors. Cameras also has a limited field of vision.
How might this scenario provide for us a modern context of FER and the incorporation of the Kafka metaphor superseding that of Bentham? Without a doubt, employees have a free will to sign a contract with a firm which practices workplace surveillance, for example, every time a company laptop is opened but there are instances that an employee might be unaware what they are subject to by emergent e-HRM. Some of these systems have begun to resemble policing applications using point scoring systems. Furthermore, with the deployment of FER systems, an additional dimension is being captured beyond productivity; the worker’s emotional state. These newly considered monitoring mechanisms, purportedly for health and wellbeing purposes, are prone to secondary uses in e-HRM systems in diverse work scenarios, for example in cybersecurity (e.g., loss prevention), talent management (e.g., insider threats), and group dynamics (e.g., groupthink). Implementation of FER technologies may also affect team dynamics, exacerbating the power asymmetry in the workplace between employers and employees. It may result in a one-way mirror, or ‘oligopticon’, where employees are further placed under scrutiny while managers and supervisors remain exempt from these invasive observations [44].
SECTION III.
Methodology: Our Investigation
This paper reports on the second phase of a two-phased socio-technical study in which a facial emotional recognition protype was developed (Phase One [1]), after which the prototype was subject to a social implications assessment in Phase Two, with the purpose of presenting policy recommendations relevant to the design and implementation of FER systems in future of work scenarios.
With respect to Phase One, an interactive FER technical prototype was developed using an existing model (https://github.com/atulapra) of emotional recognition to demonstrate the feasibility of deploying such a system in workplaces of the future, where employers might use it to monitor the emotions of their employees, possibly for health and wellbeing in addition to other purposes [1]. The goal of the technical prototype was to identify at least seven emotions of the subject (the employee) based on the pre-trained dataset used in the original project. The model was trained on the FER-2013 dataset downloaded from Kaggle (https://kaggle.com/deadskull7/fer2013). The pipeline was built using OpenCV and TensorFlow while being hosted on a single computer, with the dominant emotion displayed on the front end. It is noteworthy that the aggregate data collected from this emotion recognition system could be further analyzed to determine the emotional state of the subject. The scope of the current system was limited to a single subject, which was enough to test the feasibility of the idea. This system could be further scaled based on the requirements of the employer. Further information regarding the first phase of the project is available [1].
Regarding Phase Two, this paper offers a social implications assessment by examining the technical prototype developed in the initial phase in conjunction with academic literature, to explore the future of work implications from a socio-technical perspective. This is followed by policy recommendations, building on the suggestions and work published in [1], in order to provide initial insights into the workplace related challenges that may ensue and the requirement for a suitable policy or regulatory response.
SECTION IV.
Implications for Workforce Wellbeing After Emotional Surveillance Technologies
The social implications assessment of the interactive FER prototype demonstrated three dominant themes that warrant further exploration. These include the emotional literacy implications of FER systems deployed in a workplace setting; the coding biases that are likely to emerge and have been demonstrated in comparable contexts such as general facial recognition; and the implications from the perspective of diverse working groups, such as women, racial minorities, undocumented immigrants and refugee workers, and employees living with a disability [5], [30].
A. Emotional Literacy Implications
1) Ability to Communicate and Relate:
Relationship therapist Esther Perel warns of emotional intelligence implications that can result from the use of emotion reading technologies. She refers specifically to emotional literacy, the knowledge and consciousness of self-emotions, and the ability to recognize them in others, as follows: “So many people would rather send messages than talk. They would rather text than talk. Why would I use a technology that will help you understand the emotions of the person who’s writing to you, instead of actually walking five meters and going to talk to your neighbor? And what you need to learn is to re-engage with people and deal with the discomfort, which would be actually less effort. The more we insulate ourselves from the physical, messy, interactive, iterative processes of relating to others, the more we will need to develop technologies that help us deal with the discomforts. But we have created those very discomforts or amplified those very discomforts by minimizing friction, by minimizing all the situations in which people used to deal with others” [31].
Relevant to the FER system presented in Phase One of the study and its potential workplace application more specifically, there is the risk of undermining existing workplace communication dynamics and relational aspects within the work environment should a system of this nature be implemented. The risk extends to overreliance on technology, minimization of necessary friction and less interpersonal communication resulting in a siloed workplace environment, the social implications of which are yet to be articulated but likely to have far-reaching consequences in view of employer-to-employee, and employee-to-employee relationships.
2) Emotional Coded Bias:
The language of emotions has changed with time, and it varies across cultures, and with political and scientific knowledge. Psychologist Robert Plutchik developed one of the most popular emotion wheels, known as the Plutchik wheel (Fig. 2). He suggested that people experience eight core emotions, which he arranged in opposite pairs on the wheel [32]. This model of emotions included the following labels: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger and Anticipation. In a study carried out by Berkeley psychologists through self-reporting after watching videos, there were 27 distinct varieties of reported emotional experience [33]. Other scholars of emotional research have claimed there are around 100 emotions [34]. The machine’s classifications and interpretations when labelling an emotion will be limited to the emotional data it is fed, which itself is limited by the classifications and interpretations of the coders that create the algorithms. For example, certain emotions like anger are socially penalized and therefore their public expression will likely be hidden [35], [36]. Expression of anger may also be penalized differently across identities, for example amplified attention on women’s anger may limit self-expression. The extent to which emotions, like anger, are tolerated are cultural and context-dependent to local practices. Care is required when implementing universal applications of westernized standards of emotion, as it may confuse a friendly response in one culture, misinterpreting it as anger.
Fig. 2. Robert Plutchik’s Wheel of Emotions. https://en.wikipedia.org/wiki/Emotion_classification#/media/File:Plutchik-wheel.svg Machine Elf 1735 (2011).
More than contending or agreeing on a number, this debate in psychology research illuminates an important discussion that needs to occur before emotion recognition technologies are designed. How many emotions will be included? By which criteria will these be selected and by whom? The emotions that the machine will be able to read are contingent on the emotions selected by whoever designs and codes the algorithms of these systems.
Through the assessment of the phase one FER prototype, it is evident that the identification of seven emotions of the subject (the employee) is insufficient to accurately capture the complexity of human emotion, resulting in design bias. This further reinforces the need for interdisciplinary engagement in the design and development of FER systems in future of work scenarios (and in general) to ensure that design-related biases are addressed in the most suitable fashion, drawing on relevant disciplinary expertise. Another requirement is to appropriately communicate to workplaces/employers wishing to design and implement such systems of the potential biases that exist, and the inability to definitively capture the vast range of human emotions. This implies that care must be taken in making managerial and other decisions that are informed by these systems [37].
3) Coding Biases to Explore in Face-Emotional Reading:
One way to assess the social implications of emotional recognition systems in a workplace setting could be to reference the history, trajectory, and processes of past similar technologies. One that could provide a good starting point is facial recognition systems in a general sense. While facial recognition accuracy has improved in recent years, gross inaccuracies in the identification of some individuals remains.
From this starting point, and in assessing the Phase One FER prototype, we can question what emotional coding would be like and what would be the scenarios for recognition of users or employees that do not fall into predefined categories. For example, we need to explore whether a machine is capable of reading the emotions across diverse groups of individuals, including in situations that present unique scenarios, or in cases where individuals are wearing protective equipment, glasses and face masks, among many other examples. The implications of inaccurate readings based on a range of physical, cultural, and other interpretability-related factors requires additional consideration.
B. Implication for Different Working Groups
The themes mentioned above and the assessment of the Phase One prototype prompt other considerations regarding the manner in which different working groups could experience these FER technologies and systems. The following list provides examples of the possible implications for a range of groups, noting that this list is not exhaustive and does not seek to be limiting but rather demonstrative of the implications. The purpose here is to inspire engineers, designers and developers of FER systems to be aware of the importance of collaboration and interdisciplinarity.
Women: Historically, women have been more likely to express concerns about privacy and are much less likely than men to approve of the use of cameras for facial recognition purposes in the workplace [38]. The placement of the camera can be abused by the operator to stalk or harass [39].
Racial minorities: People who live in black bodies have a historic and systemic experience of being surveilled in every aspect of their lives, from going out for a walk, to shopping in store. Adding another site where these groups would be surveilled could have drastic consequences on an individuals’ mental health.
Undocumented immigrants and refugee workers: Surveillance raises fears of being deported.
Employees living with a disability/disabilities: Workers living with mental health disabilities might experience more stress and anxiety over being surveilled. According to Burr et al., “digital technologies to monitor employees is related to self-reported impacts on anxiety, stress, and depression… the use of such systems impacts negatively an employee’s mental health, and may lead to a loss of commitment, professional identity, and self-confidence” [42]. Workers with physical disabilities might have challenges navigating the technology. And in other cases, human resources may discover a worker living with a disability they were unaware of themselves.
SECTION V.
Policy Recommendations
Implementing any FER system in a workplace or future of work context, such as the interactive technical prototype developed in Phase One of this study is a decision that should entail a great deal of consideration and scrutiny. Additionally, it requires a thorough assessment of the social implications of such systems prior to design and development, some of which were identified and explored in the preceding section of this paper. These include implications relevant to monitoring and surveillance; design and other biases; and the fundamental social, relational, and other challenges emerging in the workplace setting. The challenges and implications are further exacerbated in situations where there is full automation.
It is evident that in addition to an assessment of the social implications of FER, workplace monitoring using surveillance technologies in future of work scenarios, needs to be accompanied with policy responses to counter the potential challenges and minimize the corresponding risks. A series of recommendations, framed as a workplace or employer checklist, is presented below in this regard. While we offer this list of considerations in the context of workplace policy, governmental regulations may also aid as guidance for implementation and privacy aspects of FER. Whether governments or corporations should lead decision-making on emerging technology remains a contentious topic of debate. Broader regulatory options include soft-law approaches [48], mixed-approaches, and privacy enhancing regulation through comprehensive laws or reviewing and extending civil and human rights at the federal level [43].
A. Decide if This Technology Should be Implemented in Your Workplace
Before pursuing new workplace systems that utilize FER capabilities that may potentially result in enhanced monitoring and surveillance, there should be a candid discussion of the social implications resulting from the respective system. As described previously in this paper, there are a range of implications resulting in trust-related and other challenges. Before considering a corporate policy based on a technological fix to address employee health, wellbeing, productivity or any other facet of the current and future of work environments, the following questions require consideration.
1) Have Non-Technical Solutions Been Sought?:
Many of the technological fixes brought on by workplace surveillance can be addressed through training, leadership and other mechanisms and organizational programs. Investing in training and capacity building for supervisors can help boost productivity, encourage candid conversations about employee wellbeing, and help maintain a safe work environment. Technology cannot fix a broken work culture, nor will it solve complex, social or relational related organizational challenges.
2) Who Is the Workforce?:
Studies by McStay [40] showed the majority of older people (55+) were “not ok” with emotion detecting technology including voice, facial coding, and body measurement data. In contrast, younger people in the study (18-54) were more likely to accept the technology as long as there was an “opt-in” process and control over how the data was being used. These and other workforce related insights aid in understanding that a workforce and employees within cannot be treated as a homogenous group with respect to their willingness to consent to the use of FER technologies and systems. There is therefore the need for employers and organizations to consult with their employees in developing employee monitoring programs and policies, in order to gauge perspectives regarding willingness to consent to FER systems.
3) Where Is the Technology Being Implemented?:
With the massive increase in remote working situations across many occupations, workplace monitoring, surveillance and use of FER technologies is becoming more ubiquitous outside of the physical workplace. Some surveillance technologies have long been used by employers to track employees for safety, productivity, and a variety of other reasons. While there is literature on how these technologies affect workers in the workplace, there is little research on the implications beyond the physical workplace setting, such as the home, or in remote work scenarios. Organizational policies should encompass multiple locations and work configurations considering current and hybrid workplace arrangements, much of which has escalated and been driven as a result of the COVID-19 pandemic.
B. Ensure Employees Understand the System
What is being built and why? New technologies can be used to increase productivity, build safer and more secure workplaces, promote employee health and wellbeing, or enhance communication. However, there are also challenges and social implications as articulated in this paper, and these need to be afforded equal weight and considerations as the benefits and opportunities. Furthermore, employees require transparency with respect to the details and communications concerning any new FER system. The first step in this regard should be a complete analysis and clarification of the reasons the new technology is needed and the motivations, objectives, benefits, and challenges of its introduction into the current or future work environment. This point is especially important when procuring or developing a facial recognition system due to the privacy and personal implications involved, and the social implications and themes presented in this paper.
If the decision is taken to design and or develop an FER system, employees should be engaged prior to implementation. They should be equal participants in steering the configuration and implementation and have an active voice in the governance and usage of the system once implemented. Organizational policies should furthermore reflect these consultations and discussions. Multiple studies on workplace surveillance systems have shown that they can lead to decreased job satisfaction, increased turnover, and even active retaliation and resistance from employees. To help avoid these negative effects and generate a healthy and just workplace, employers should center the preferences and rights of their workforce. Creating a positive work culture around the adoption of facial recognition systems, can only happen when employees have been consulted about the prospects of the AI-based business transformation, and agree to embrace it prior to its operation.
This is essential, as facial recognition technologies have the capability to collect vast amounts of data that can be highly personal and private. Pinpointing exactly what the system will be observing, collecting and processing is extremely important as it helps to shape how the system works and what kind of information it is gathering.
C. Ensure the System Is Explainable and the Data Gathered Is Transparent
To further build employee trust in the system and decrease the likelihood of adverse implications, it is important that the system is explainable, and the data being gathered is transparent to the employee. Employees should be aware of the data points, both intended and unintended, that are being gathered by the system. Enabling processes for employees to request or audit their data will help them build trust in the system and know what is being collected and how it is being used. Organizational policies need to clearly communicate details of the system and request and audit considerations. This is especially important for facial recognition systems due to the private nature of the data gathered and the possibility of misuse. Discussions around worker’s privacy and property rights to their data and limitations around use are also recommended.
D. Allow Employees to Opt-Out
The ability to opt-out, without any repercussions, is a fundamental requirement of any workplace FER system. Organizational policies need to set out details of opting out and stipulate any equity and rights-based considerations relevant to this point. This is an area for future research and requires much thought in subsequent work.
E. Ensure the System Benefits Employees
To avoid employees opting-out of the system entirely, there needs to be clear and engaging reasons to opt-in. If employees are sharing their data, their faces, and their trust — they should get something in return. Blackman [41] suggests that employers should ensure that the data gathered can be used to help employee development, increase job satisfaction, or help an employee remain healthy. Only implementing a system to monitor or surveil the employee without benefits can lead to negative effects and overall negative feelings from the perspective of the employee [15], [16], [42]. Take for example, Sadowski [15] who identifies that some warehouses have paramedics stationed permanently outside a premise as they expect some employees to “routinely collapse from dehydration and heat stress” [15]. Blackman [41] suggests using high level findings from data to engage with employees on shared challenges and achievements. Additionally, the consideration that behavior and emotional expression is masked or performed for the machine by the worker diminish the system’s efficiency. The emotional performative masking can have cognitive and health consequences for the worker [42].
F. Specific Policy Items to Include Such as Values
TABLE I Human-Centered Values
When crafting policies related to workplace surveillance technology, human-centered values should serve as a foundation. Table I provides a sample list of considerations (Dos and Don’ts) relative to human values that should inform policy development and should be central to engaging with employees when developing and implementing the system. These values were developed in reflection of Phase I of this study, as recommendations for best-practice approaches to development of FER.
SECTION VI.
Conclusion
This article presented the outcomes of Phase Two of a two-phased socio-technical FER study intended to explore the social implications and policy recommendations of FER in a future of work scenario and context. A technical prototype was developed (Phase One, [1]) and the social implications presented (Phase Two) in order to generate organizational policy recommendations to inform future research and practice related to facial emotion recognition. It is suggested that future research builds on the outcomes of this project to consult with relevant stakeholders with respect to their perspectives and expectations relevant to FER in the workplace, in order to ensure the responsible, ethical and considered design of emerging biometrics and facial emotion recognition systems.
ACKNOWLEDGMENT
The authors also acknowledge members of the research team at large who built the FER prototype inclusive of: Nickolas Dodd, Haowen Fan, Parth Khopkar, Francis Mendoza and Riley Tallman from the School of Computing and Augmented Intelligence, in addition to Edgard Musafiri Mimo and Yatiraj Shetty from the Polytechnic School. They extend their gratitude to Jason M. Burris, the Academic Program Manager of INTEL’s Internet of Things Group, for their support in seminar deliveries and sharing DevCloud for the Edge and OpenVinoTM training materials.
References
N. Dodd, “Facial emotion recognition and the future of work,” in Proc. IEEE Int. Symp. Technology and Society (ISTAS), Nov. 2022, pp. 1–9. doi: 10.1109/ISTAS55053.2022.10227134.
N. Deshmukh, A. Ahire, S. H. Bhandari, A. Mali, and K. Warkari, “Vision-based lip reading system using deep learning,” in Proc. Int. Conf. Computing, Communication and Green Engineering (CCGE), 2021, pp. 1–6. doi: 10.1109/CCGE50943.2021.9776430.
M. Bowen, “‘Robocop’ patrolling the streets of Dubai,” Intelligent CIO, May 25, 2017. [Online]. Available: https://www.intelligentcio.com/me/2017/05/25/robocop-patrolling-the-streets-of-dubai/
D. Braue, “ChatGPT gets a voice—and feelings,” Information Age (Australian Computer Society), May 16, 2024. [Online]. Available: https://ia.acs.org.au/article/2024/chatgpt-gets-a-voice-/-/-and-feelings.html
K. Michael, R. Abbas, P. Jayashree, R. J. Bandara, and A. Aloudat, “Biometrics and AI bias,” IEEE Transactions on Technology and Society, vol. 3, no. 1, pp. 2–8, Mar. 2022.
Y. Lutchyn, P. Johns, A. Roseway, and M. Czerwinski, “MoodTracker: Monitoring collective emotions in the workplace,” in Proc. Int. Conf. Affective Computing and Intelligent Interaction (ACII), Sep. 2015, pp. 295–301. doi: 10.1109/ACII.2015.7344586.
J. Cheesman, “If you use Slack, you can monitor company morale,” TLNT, Apr. 25, 2017. [Online]. Available: https://www.tlnt.com/if-you-use-slack-you-can-monitor-company-morale/
P. Ekman, “Universals and cultural differences in the judgments of facial expressions of emotion,” Journal of Personality and Social Psychology, vol. 53, no. 4, pp. 712–717, 1987. doi: 10.1037/0022-3514.53.4.712.
C. BasuMallick, “Can employee mood tracking improve performance?” HRTechnologist, 2020. [Online]. Available: https://www.hrtechnologist.com/articles/performance-management-hcm/can-employee-mood-tracking-improve-performance/
M. Riganelli, V. Franzoni, O. Gervasi, and S. Tasso, “EmEx: A tool for automated emotive face recognition using convolutional neural networks,” in Proc. Int. Conf. Computational Science and Its Applications (ICCSA), 2017, pp. 692–704. doi: 10.1007/978-3-319-62398-6_49.
A. R. Aguiñaga, A. Realyvazquez, M. A. Ramirez-Lopez, and A. Quezada, “Cognitive ergonomics evaluation assisted by an intelligent emotion recognition technique,” Applied Sciences, vol. 10, no. 5, Art. no. 1736, 2020. doi: 10.3390/app10051736.
A. Hern, “Microsoft productivity score feature criticized as workplace surveillance,” The Guardian, Nov. 26, 2020. [Online]. Available: https://www.theguardian.com/technology/2020/nov/26/microsoft-productivity-score-feature-criticised-workplace-surveillance
K. Michael, J. R. Schoenherr, and K. M. Vogel, “Failures in the loop: Human leadership in AI-based decision-making,” IEEE Transactions on Technology and Society, vol. 5, no. 1, pp. 2–13, Mar. 2024. doi: 10.1109/TTS.2024.3378587.
E. Ongweso Jr., “An AI paper published in a major journal dabbles in phrenology,” VICE, 2020. [Online]. Available: https://www.vice.com/en/article/g5pawq/an-ai-paper-published-in-a-major-journal-dabbles-in-phrenology
J. Sadowski, Too Smart: How Digital Capitalism Is Extracting Data, Controlling Our Lives, and Taking Over the World. Boston, MA, USA: MIT Press, 2020.
K. Rogers, “What constant surveillance does to your brain,” VICE, Nov. 14, 2018. [Online]. Available: https://www.vice.com/en/article/pa5d9g/what-constant-surveillance-does-to-your-brain
S. Tuncer, “The effects of video recording on office workers’ conduct, and the validity of video data for the study of naturally occurring interactions,” Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, vol. 17, no. 3, 2016. doi: 10.17169/fqs-17.3.2604.
P. W. Cardon, H. Ma, A. C. Fleischmann, and J. Aritz, “Recorded work meetings and algorithmic tools: Anticipated boundary turbulence,” in Proc. Hawaii Int. Conf. System Sciences (HICSS), 2021, pp. 1–10.
P. W. Cardon, H. Ma, and A. C. Fleischmann, “Recorded business meetings and AI algorithmic tools: Negotiating privacy concerns, psychological safety, and control,” International Journal of Business Communication, vol. 60, no. 4, pp. 1095–1122, 2023.
G. Orwell, Nineteen Eighty-Four. London, U.K.: Harvill Secker, 1949.
M. M. McLaughlin and S. Vaupel, “Constitutional right of privacy and investigative consumer reports: Little brother is watching you,” Hastings Constitutional Law Quarterly, vol. 2, no. 3, 1975. [Online]. Available: https://repository.uchastings.edu/hastings_constitutional_law_quaterly/vol2/iss3/5
C. De Pressigny, “The creepy AI-driven surveillance that may be infiltrating your workplace,” Business Insider, Nov. 20, 2023. [Online]. Available: https://www.businessinsider.com/ai-surveillance-detects-emotion-at-work-gets-you-fired-2023-11
M. Polachek, “5 companies that want to track your emotions,” Fortune, Aug. 22, 2020. [Online]. Available: https://fortune.com/2020/08/22/emotion-sensing-tracking-technology-apps/
K. Roemmich, F. Schaub, and N. Andalibi, “Emotion AI at work: Implications for workplace surveillance, emotional labor, and emotional privacy,” in Proc. CHI Conf. Human Factors in Computing Systems, Apr. 2023, pp. 1–20. doi: 10.1145/3544548.3580950.
M. A. Ali, J. P. Nachumow, J. A. Srigley, C. D. Furness, S. Mann, and M. Gardam, “Measuring the effect of sousveillance in increasing socially desirable behavior,” in Proc. IEEE Int. Symp. Technology and Society (ISTAS), Toronto, ON, Canada, 2013, pp. 266–267. doi: 10.1109/ISTAS.2013.6613128.
D. Palmer, I. Warren, and P. Miller, ID Scanners in the Night-Time Economy: Social Sorting or Social Order? (Trends & Issues in Crime and Criminal Justice). Canberra, ACT, Australia: Australian Institute of Criminology, 2013.
E. L. Piza, J. M. Caplan, and L. W. Kennedy, “Analyzing the influence of micro-level factors on CCTV camera effect,” Journal of Quantitative Criminology, vol. 30, no. 2, pp. 237–264, 2014. doi: 10.1007/s10940-013-9202-3.
D. J. Solove, “Privacy and power: Computer databases and metaphors for information privacy,” Stanford Law Review, vol. 53, no. 6, pp. 1393–1462, Jul. 2001.
D. Clifford, Data Protection Law and Emotion. Oxford, U.K.: Oxford University Press, 2024.
K. Michael, “Racial and genetic discrimination in automated face analysis,” in Proc. 5th Workshop on Demographic Variability in Performance of Biometric Algorithms and Related Technologies (DVPBA), Waikoloa, HI, USA, Jan. 2024. [Online]. Available: https://sites.google.com/msu.edu/dvbpa2024/home
E. Perel, “Does this tech take away our free will?” Should This Exist?, 2020. [Online]. Available: https://shouldthisexist.com/
R. Plutchik, “The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice,” American Scientist, vol. 89, no. 4, pp. 344–350, 2001.
A. S. Cowen and D. Keltner, “Self-report captures 27 distinct categories of emotion bridged by continuous gradients,” Proceedings of the National Academy of Sciences of the United States of America, vol. 114, no. 38, pp. E7900–E7909, 2017. doi: 10.1073/pnas.1702247114.
L. Nummenmaa, R. Hari, J. K. Hietanen, and E. Glerean, “Maps of subjective feelings,” Proceedings of the National Academy of Sciences, vol. 115, no. 37, pp. 9198–9203, 2018. doi: 10.1073/pnas.1807390115.
A. Lorde, “The uses of anger,” Women's Studies Quarterly, vol. 9, no. 3, pp. 7–10, 1981.
L. R. Owens, Love and Rage: The Path of Liberation Through Anger. Berkeley, CA, USA: North Atlantic Books, 2020.
K. Michael, “Emotion detection systems and the future of work,” in Proc. Biometrics Institute Asia-Pacific Conference, 2024.
L. Stark, A. Stanhaus, and D. L. Anthony, “‘I don’t want someone to watch me while I’m working’: Gendered views of facial recognition technology in workplace surveillance,” Journal of the Association for Information Science and Technology, vol. 71, no. 9, pp. 1074–1088, 2020. doi: 10.1002/asi.24342.
G. J. D. Smith, “Exploring relations between watchers and watched in control(led) systems: Strategies and tactics,” Surveillance & Society, vol. 4, no. 4, pp. 280–313.
A. McStay, “Emotional AI, soft biometrics and the surveillance of emotional life: An unusual consensus on privacy,” Big Data & Society, vol. 7, no. 1, Art. no. 2053951720904386, 2020. doi: 10.1177/2053951720904386.
R. Blackman, “How to monitor your employees—While respecting their privacy,” Harvard Business Review, May 28, 2020. [Online]. Available: https://hbr.org/2020/05/how-to-monitor-your-employees-while-respecting-their-privacy
C. Burr, J. Morley, M. Taddeo, and L. Floridi, “Digital psychiatry: Risks and opportunities for public health and wellbeing,” IEEE Transactions on Technology and Society, vol. 1, no. 1, pp. 21–33, Mar. 2020. doi: 10.1109/TTS.2020.2977059.
E. Hajric, “AI and data rights considerations for U.S. policy,” IEEE Technology and Society Magazine, vol. 40, no. 3, pp. 93–98, Sep. 2021. doi: 10.1109/MTS.2021.3101924.
B. Latour, Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford, U.K.: Oxford University Press, 2005.
P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System (FACS): A Technique for the Measurement of Facial Action. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
D. J. Solove and W. Hartzog, “Kafka in the age of AI and the futility of privacy as control,” Boston University Law Review, vol. 104, p. 1021, 2024. doi: 10.2139/ssrn.4685553.
Y. Apolo and K. Michael, “Beyond a reasonable doubt? Audiovisual evidence, AI manipulation, deepfakes, and the law,” IEEE Transactions on Technology and Society, vol. 5, no. 2, pp. 156–168, Jun. 2024. doi: 10.1109/TTS.2024.3427816.
C. I. Gutierrez, G. E. Marchant, and K. Michael, “Effective and trustworthy implementation of AI soft law governance,” IEEE Transactions on Technology and Society, vol. 2, no. 4, pp. 168–170, Dec. 2021.
Authors
School for the Future of Innovation in Society, Arizona State University, Tempe, AZ, USA
Elma Hajric received the Bachelor of Arts degree in international affairs and German from Northern Arizona University, Flagstaff, AZ, USA, in 2012, the master’s degree in Science and Technology Policy from the School for the Future of Innovation in Society, Arizona State University, Tempe, AZ, USA, in 2019, and the Ph.D. degree in human and social dimensions of science and technology from the College of Global Futures, Arizona State University in 2024.
She has held a fellowship in the National Science Foundation National Research Traineeship in Citizen-Centered Smart Cities and Smart Living, and was a Humanities, Arts, Science and Technology Alliance and Collaboratory Scholar. Previously, she interned with the Government Accountability Office, Washington, DC, USA. Her work on ‘AI and Data Rights Considerations for U.S. Policy’ has been published in IEEE. Her research is in the fields of critical data studies and science and technology studies, using a feminist technoscience lens to examine data narratives and imaginaries at the intersections of surveillance and smart urbanism. Her research aims are in the interest of data justice and intersectionally inclusive futures.
School for the Future of Innovation in Society, Arizona State University, Tempe, AZ, USA
Farah Najar Arevalo received the the B.A. degree in international affairs and the M.Sc. degree in global technology and development. She is currently pursuing the Doctoral degree with School for the Future of Innovation in Society (SFIS), Arizona State University (ASU).
Her research explores smart city technologies at the intersection of development studies, science and technology studies, and metropolitan studies. She is specially interested in the technology assessment of urban mobility technologies, with a special focus on their impact on women and marginalized groups. She brings a rich background in the nonprofit sector, local government, and academia, with her experience spanning across the USA, Mexico, Brazil, and the Middle East.
She currently serves at ASU’s GenAI Committee since 2023, a Ph.D. student-led forum addressing AI-related student concerns, experiences, and inquiries. Her previous service roles at ASU include serving as the Assembly President of the Graduate and Professional Student Association from 2022 to 2023. Looking ahead, She aspires to pioneer research that aids global cities in making informed and equitable technology decisions. Her goal is to create impactful research that guides urban centers worldwide toward more informed, purposeful and just applications of technology.
Tribal DataWorks, Phoenix, AZ, USA
Leonard Bruce was born in AZ, USA. He received the B.S. degree in sociology and the M.A. degree in Science and Technology Policy from Arizona State University, Tempe, AZ, USA.
He is a Community Technologist with Tribal DataWorks. His research interests focus on the labor and economics of indigenous communities, particularly those on the urban periphery. His work centers around his ancestral homelands at Gila River, exploring topics, such as decolonizing employment, increasing social and economic mobility, and building social and economic resilience for indigenous nations. He is passionate about leveraging technology to address community needs and ensuring indigenous voices are at the forefront of discussions on emerging technologies. A significant part of his work involves digital preservation and cultural memory projects to bridge traditional knowledge with modern technology, creating innovative tools for cultural preservation and intergenerational learning within his community. His professional experience includes various projects and writings related to the intersection of technology and indigenous history. His work revolves around using technology to address community needs while centering indigenous voices in the discussion of emerging technologies. He is dedicated to fostering technological advancements that benefit indigenous communities while preserving their cultural heritage.
Mr. Bruce is a member of the Gila River Indian Community.
Georgia Environmental Protection Division, Albany, GA, USA
Fritz Antony Smith was born in Decatur, GA, USA. He received the bachelor’s degree in theology from Lee University, Cleveland, TN, USA, and the Master of Science degree in technology policy from Arizona State University, Tempe, AZ, USA.
He currently works as an Environmental Compliance Specialist for the Georgia Environmental Protection Division and serves in the Georgia National Guard, where he is training to become a Military Officer. His professional and research interests focus on the intersection of environmental policy, ethics, political philosophy, and sustainability. He is also pursuing advanced studies in philosophy, with a particular emphasis on ethical frameworks and their implications for addressing future societal challenges.
School for the Future of Innovation in Society and the School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA
Katina Michael (Senior Member, IEEE) received the B.S. degree in information technology from the School of Mathematical and Computing Science, University of Technology, Sydney, NSW, Australia, in 1996, the Doctor of Philosophy degree in information and communication technology from the Faculty of Informatics, University of Wollongong, Wollongong, NSW, Australia, in 2003, and the Master of Transnational Crime Prevention degree (with Distinction) from the Faculty of Law, University of Wollongong in 2009.
She researches the social, legal, and ethical implications of emerging technologies. She has a joint professorial appointment with the School for the Future of Innovation in Society and the School of Computing and Augmented Intelligence, Arizona State University, where she is the Director of the Society Policy Engineering Collective. She has been funded by national research councils in Australia, USA, and Canada. She is also an Honorary Professor with the School of Business, University of Wollongong, where she was previously the Associate Dean International of the Faculty of Engineering and Information Sciences. She is the Founding Editor-in-Chief of the IEEE Transactions on Technology and Society.
Citation: E. Hajric, F. N. Arevalo, L. Bruce, F. A. Smith and K. Michael, "Facial Emotion Recognition in the Future of Work: Social Implications and Policy Recommendations," in IEEE Transactions on Technology and Society, vol. 6, no. 3, pp. 295-304, Sept. 2025, doi: 10.1109/TTS.2024.3477512.