Facial Emotion Recognition and the Future of Work

IEEEISTAS22International Symposium on Technology and Societyconferencepeer-reviewed

28 Aug

Citation: N. Dodd et al., "Facial Emotion Recognition and the Future of Work," 2022 IEEE International Symposium on Technology and Society (ISTAS), Hong Kong, Hong Kong, 2022, pp. 1-9, doi: 10.1109/ISTAS55053.2022.10227134.

Abstract

This article provides an overview of the emergent area of facial emotion recognition (FER). Increasingly, videoconferencing applications are being used to bridge the geographic divide, as telecommuters have opted to work remotely (e.g. from home) during the COVID-19 pandemic. organizations have established business continuity programs that want to ensure that their remote workers are maintaining their health and well-being. As a result, some videoconferencing software can now perform FER, providing feedback to supervisors about how well their workforce is coping during these unprecedented times. What might such technologies mean for how people present in front of the camera? And how do these systems work? The significance of this paper is in presenting an end-to-end technological description of FER systems in the context of a “future of work” use case scenario presenting a working prototype developed by the authors and available on GitHub. The concepts of fairness and algorithmic bias in context is considered, before proceeding to address some of the human and social dimensions of these systems.

SECTION I.

Introduction

**Figure 1.** French artist Charles Le Brun in his famous 1668 lecture titled: “Conférence sur l’expression générale et particulière” created a set of pictures to demonstrate that the passions of the soul affect the muscles of the face **[7]**. Source: https://commons.wikimedia.org/wiki/File:Sixteen_faces_exp ressing_the_human_passions._Wellcome_L0068375.jpg Sixteen faces expressing the human passions. Colored engraving by J. Pass, 1821, after C. Le Brun. Credit: Wellcome Images.

The global COVID-19 pandemic has driven many workplaces to seek and implement new technologies to improve efficiency, maintain worker safety, and enhance team cohesion [1]. There has been an increase in use of video conferencing technologies in many occupations to sustain team cohesion when physical meetings are unsafe. These same video conferencing technologies are also being used to track worker movement and activity while they are away from their physical place of work [2], [3]. Literature has shown that workplace surveillance in many aspects has the danger of creating distrust between employers and employees, but little research has been conducted on whether these feelings will be exacerbated when the audiovisual-based surveillance systems invade people’s homes. This paper will present one such technology that could be used on employees at home or in the workplace, known as “emotional detection The act of recognizing a person’s emotional state without human intervention is known as facial emotion recognition (FER). In 1978 [4] the facial action coding system (FACS) was founded by Ekman and Friesen to detect changes in specific facial muscles that were known as action units. These momentary changes in someone’s face, are said to depict individual human emotions [5]. There are seven basic emotions that FER can detect: happiness, surprise, anger, sadness, fear, disgust, and neutral (see Figure 1). In 2014, Tao and Martinez [6], put forward 22 emotions inclusive of what is known as compound emotions (a combination of two basic emotions), plus an additional three emotions including appall, hate, and awe.

This paper begins with a brief introduction to facial emotion recognition and corresponding concerns when attempting to automate the interpretation of affect. The paper is wholly centered on a Future of Work use-case scenario and presents a systems design of one such FER built to the prototype stage for remote worker health and well-being. The paper describes the importance of deep convolutional neural networks (CNN) and the role of cloud computing in CNN. An interactive demonstration is then presented showing one user’s emotion being detected, and thereafter a discussion on accuracy of FER, and various forms of known biases stemming, for example, from data sets, design, algorithms and more. Fairness and equity are then addressed before a closing discussion on the human and social dimensions of FER in the context of the future of work.

SECTION II.

Automating Emotion Recognition: Concerns

A. Interpreting Human Emotions

There is a great variability in how we perceive emotions [8]. In fact, when talking about emotion recognition, we must first define the criteria for what the true emotion is. One possibility is that we can predict one’s emotion based on what most people would agree with. For instance, if someone puts on a big smile, most people will think that person is very happy. However, the person themselves may pretend to be happy and actually feel very sad. The knowledge-based approach was derived from this criteria, where the domain knowledge and characteristics of emotions are used to detect and predict emotions [e.g., 9]. Since this approach pertaining to emotion recognition and classification is established in the literature, the approach is considered to be efficient and economical. However, it also has some limitations in terms of dealing with complex emotions and handling subtle differences in emotions.

An alternate approach is to ask someone about what they truly feel, create an algorithm to learn from a huge number of “accurate” examples, and then predict the unknown emotions. This approach is effective only when someone has a good estimate of his internal feelings, is willing to share with developers, and is capable of describing their emotions in a detailed manner. However, if people have alexithymia [10] defined as the inability to recognize or describe one’s own emotions, this will serve as a false input to the system and brings uncertainty in the learning process and accordingly to the predicted results. When building facial emotion recognition systems, we cannot eliminate those circumstances, but we can decrease their possible influence by sufficiently increasing the number of participants as input. This introduces the statistical approach in emotion recognition, where different kinds of machine learning algorithms are utilized. There are three types of machine learning algorithms: supervised learning, unsupervised learning and reinforcement learning. Support Vector Machines, Naive Bayes and Maximum Entropy are the most commonly employed supervised machine learning algorithms in emotion recognition [11]. There are also widely used unsupervised machine learning algorithms, also called deep learning algorithms, being adopted in emotion recognition [12]. Different architectures of artificial neural networks, such as convolutional neural networks [13], long short-term memory, and extreme learning machines are quite popular because of their success in emotion recognition related application areas: natural language processing, speech recognition and computer vision.

B. How Accurate is Emotion Detection?

Expressions of emotions are varied, complex and situational. An emotional category does not capture the essence of an emotion, rather it represents a category that applies to highly variable individuals. Let’s think of how we predict emotions in real life. When we see people in person, we have much more information about the context of their emotions than simple facial analysis. We can not only see their actions, and continuous body language, but also hear their voices. So will it be sufficient to use a visual algorithm to detect emotions?

According to a recent review [14], while tech companies have already sold their emotion detection algorithms based on a series of facial movements, some psychologists have pointed out that these emotion detection algorithms are far from accurate [15]. You can use an algorithm to detect a scowl and straight eyeballs, but that’s not the same thing as detecting anger. In fact, emotions are expressed in a huge variety of ways, and it is nearly impossible to reliably infer how someone feels simply from a set of facial movements. For example, according to a study by psychologists, when people scowl, they are actually angry less than 30 percent of the time [16], they may even be feeling afraid instead.

AI excels at finding the hidden connections in data. That is why we often see machine learning being utilized to make decisions, not because it is reliable, but because it can be easily measured. You can judge your algorithm on statistical data such as accuracy, precision and recall. When creating the labeled emotion dataset for machine learning, it is often methodologically flawed. For example, when test subjects are asked to label emotions, they are often provided with exaggerated faces and a limited selection of emotions. This encourages test subjects to come to a similar conclusion, i.e., we want it to be the emotion it most “looks” like, but people can smile when they are in distress, though it seems illogical [17]. If there is thus significant variability in how people interpret emotions and label them, even between genders, then what might that mean when robots are placed in charge of interpreting emotions [18]?

Our review of the failings of emotion detection algorithms leads us to a question: Will we accept the common deployment of emotion recognition algorithms and change our behavior to accommodate its failings? We have seen changes in how people act online when they know it will be interpreted by algorithms. To avoid interrupting ads, people tend to not click “like” on certain images on Instagram [19]. We may end up putting on exaggerated faces because we know that’s how our emotions are recognized by the machines: for comparison, imagine students are forced to look straight at the teacher’s face because that is the way you are considered attentive, or they are expected to sit up straight and gaze into the camera above the blackboard [20].

SECTION III.

Convolutional Neural Networks (CNN)

Image processing in computer science has a long history, but the current trend is to use deep learning. Deep learning has proven to be more effective than traditional image processing [21]. It is certainly not more computationally efficient, but it has more flexibility and produces better results. A convolutional neural network (CNN), is a variant of a neural network that is specifically designed for image inference. Regular feed-forward neural networks are immutable to the order in which the inputs are fed. For example, an automotive insurance company is predicting the overall risk of a potential client, and they have some driving data about the client they can use. They have information about the number of accidents, number of speeding tickets, number of years with a license, and so on. They can use a neural network for this problem because the data that they have is not in any particular order. A neural network will just look at the data at face value without regard to the order. Image data is stored as a collection of pixels and has a particular order to it, and so a feed-forward neural network is not a good choice.

A convolutional neural network is mathematically designed to recognize the order of the data in an image. It uses many matrices called kernels to traverse the image and output an intermediate image that contains more useful information. These are known as the convolutional layers and are responsible for extracting features. In a paper written by Krizhevsky, Sutskever and Hin ton in 2012 [22] that was cited over 105,000 times, Figure 3 depicts a visualization of deep convolutional neural networks, specifically showing there are 96 11x11x3 kernels from the first layer with an input image of 224x224x3. In brief, a collection of these kernels is learned by the AI. Each square is a kernel, or a filter, that is passed over an image in a sliding window fashion. Basically, the filters detect general features like edges and comers and encode that information into the image. Each convolutional layer does this, which eventually reaches the final layers with relevant information. These layers are the decision making layers and are usually just simple neural networks.

Just like feed-forward neural networks, CNNs are characterized by how many layers and neurons each layer has. CNNs have more details, such as the dimensions of the kernels, stride length, and number of filters, but the efficiency calculation of the model remains the same: it depends on the number of learnable parameters [23]. In short, the learnable parameters are an aggregate of the variables that the AI changes in order to satisfy its error metric. These variables are small numbers that take time to learn and so there is a positive correlation between training time and the number of parameters in a neural network. This correlation applies to inference as well.

SECTION IV.

The Role of Cloud Computing in CNN

Edge computing is simply computing on mobile devices such as smartphones, microcontrollers, laptops, and autonomous vehicles. Generally speaking, edge computation is done in close proximity to the source of data and users. Cloud computing is the opposite: computing is done farther away from the source of data where there are more powerful computational resources available.

With a large scale application, such as the one depicted in this article pertaining to the “future of work there will be many images from many users to analyze, even in a single transnational organization. The computation required to run the CNN would be detrimental to the employee’s computer. As a result, and to avoid this issue, the researchers that created the facial emotion recognition system prototype described in this article, have implemented the CNN in the cloud. The process is as follows:

Images are taken on the edge devices.
The image data is converted into a base 64 string for optimized Internet transportation.
The base 64 string is sent to a cloud computer.
The cloud computer analyzes all images as they arrive.
(optional) The cloud computer sends back the analysis results.

Another benefit to cloud computing in this project is that Graphics Processing Units (GPUs) can be used, which speed up inference many times over compared to the CPU. This is a well-known fact about Neural Networks — they can be parallelized on a GPU, which greatly speeds up processing [24]. GPU computing would be extremely difficult to implement on every employee’s computer in a corporate environment because many laptops and computers do not have standalone GPUs. GPUs are useful for creative work such as video and photo editing as well as gaming, but they are not very useful for administrative work like Microsoft Office products, email, and web-browsing. For this reason, GPUs are often not included in corporate computers in order to save money.

SECTION V.

Other Hardware and Systems Requirements

The “Future of Work” application developed in this research project, requires the end-user to have a webcam attached to a computer. This is the only hardware requirement on the end-user side. This does not need to be a high-end camera, as the images fed into the CNN are cropped and shrunk in size. In overview, the system works on a server (implemented using Flask, a Python module), which runs the model continuously for each still image that it receives from the client’s webcam. It uses the Haar cascade technique to first find the face in the image and then passes it to the model to determine the emotion classification. Once this is determined, the server continuously displays the image on the website with a bounding box displayed for the face that is detected, along with the determined emotion (see Fig. 2). One of the requirements for server performance is that the inference process is fast so that there is minimal lag between when the server receives a still image and when it sends the image back after inference. We also used ngrok (https://ngrok.com/) to expose a connection to the server on the Internet so that we can allow a broad audience set to access the demo [25].

**Figure 2.** Interactive demonstration of Facial Expression Recognition (FER) technology

A potential incremental innovation to the current setup is to explore the feasibility of running the model in the client’s browser using TensorFlow.js. This would remove the requirement of having a server running a model in the backend part of the system.

SECTION VI.

Systems Design

A. Future of Work Use Case Scenario

Consider the following hypothetical use-case based on the future of work context being explored in this article. The employer wants to assess the stress level of the employee (e.g. software engineer) during work hours to promote well-being. The employer needs consent from the employee and time of observation needs to be authenticated. Following consent, the webcam on the work-computer is used to track the stress level of the employee. If the employer determines that the employee is under undue stress, they might consider additional support on a given project, for example. It would also enable the employer to be more empathetic towards the needs of their employees.

If such a hypothetical system is deployed on scale for the entire department, the employer would have a good idea of the number of personnel needed for a project, the composition of the teams (ratio of senior engineers to junior engineers) and assignment of timelines. This could be used to promote employee well-being as part of existing project management tools used in office environments.

B. High Level Systems Design

**Figure 3:** High level block diagram of Future of Work system: Demonstration case

The goal of systems design is to understand the architectural load of individual components and to verify its fidelity at scale. Figure 3 gives a high-level understanding of how the components of this architecture have been organized in our Future of Work system on a single host computer. The goal is to assess the emotional state of the subject and send the analysis to the employer for further action based on the use-case.

1) Attributes of the design

Any systems design needs to have a clear set of functional requirements against which the architecture needs to be tested. The following are few functional requirements within the context of our demo case:

The pipeline should be simple to implement to adhere to the proposed deadline.
The webcam should be able to track the presence of the subject’s face.
The system should be able to identify and display the emotions of the subject in real-time [26].
The system should be low cost and user friendly.
The system should not store any data but rather process the image snapshot and reveal the dominant emotion out of the selected seven emotions.

2) Limitations of the design

The following are the limitations of this design:

Due to lack of an accurate use-case, the exact load that each component is under cannot be determined.
As this is a demonstration technology, the component selection has not been optimized, i.e., the rationale behind the choice has not been justified.
In is uncertain whether the exact algorithm used for emotion recognition can be scaled-up for multiple users.
One of the main objectives is to illustrate the overall process behind the emotion recognition algorithm and what enables it.
The current system detects the emotion of the user; detection of mood would be more useful to the employer as a measure of workplace quality [3].
User consultation was entirely absent from the design process.

C. Low Level Systems Design

The model used in this project is based on publication in the International Conference on Machine Learning (ICML)[27]. The general steps involved are as follows (figure 4):

Face detection
Feature extraction
Feature selection
Emotion classification

A simple 4 layer convolutional neural network (CNN) was implemented to estimate the probability of six basic emotions (happiness, sadness, anger, fear, disgust and surprise) and a neutral state. The advantage of mapping facial expressions to the aforementioned basic emotions is that these basic emotions are arguably discernible by the general population, which has been factored into the usage of our dataset.

The machine learning emotion recognition python script used for the emotion recognition implementation takes in a video capture/feed from a video source such as a webcam and captures an image snapshot of the user. It then draws a box around the detected face on the captured image using the “haar cascade”, which aids in drawing the boundary box around the face. The haar cascade classifier, using open CV2 then helps create an instance of the identified capture face that is then converted to a grayscale version of the image. Different color intensities of grayscale are then detected with a scale factor representing vectors depicting the captured face.

The facial recognition is done by OpenCV, which provides a box around the identified and detected user’s face in the captured video feed, and around the collected snapshot instances or frames that are constantly collected in the viewing process.

As more image frames data are collected, the incorrectly captured frames are skipped and rejected as they do not associate to any predefined emotion label. The processed and revised snapshots are cropped and resized to 48x48 pixels [27] and then eventually processed and transformed to grayscale. The captured face is depicted as in four dimensions represented by the four vectors or coordinates (x,y,w,h) of the frame that needs to be processed.

**Figure 4:** Block diagram of the system

Next each captured vector coordinate is processed and cropped to match the expanded dimension to match what the model prediction is pertaining to the overall emotion matching density from the seven considered emotions. At this point, the emotion algorithm uses a NumPy function block to determine the dominant emotion index. The index is then interpreted in view of the emotion dictionary items, where the seven specified emotions are indexed and displayed on the user interface depicting the dominant emotion sentiment as trained by the dataset.

D. Typical Practical Applications

A typical application would involve a corporation associating the video feed data that is sampled with an IP address of an employees’ device, or with an employee based on the associated employee’s ID or email, to the computer or the device that is collecting the video frames. These frames could be collected throughout the day at a specific time interval and the overall result of the dominant predicted emotion could be stored for each of the snapshots taken throughout the day.

Afterwards, a cumulative count of an emotion’s instances from the previously captured and processed snapshots can be used to identify and predict the overall employees’ engagement and productivity which can eventually lead to predicting employee’s satisfaction and productivity depending on the variation of dominant employees’ emotions. The cumulative count of predicted emotions would show a trend of emotion variation throughout the day for the user or employee, which can then be analyzed and mapped to reflect attributes that might have impacted or influenced the employee’s performance and mood.

These applications can range from identifying whether employees are affected by the work they are doing for the company or other outside factors to detecting times and assigned work that tend to shift employee’s mood should they repeatedly show the same emotions while doing some work or during a certain period [28].

Nevertheless, the emotion recognition machine learning algorithm proposed in this article serves only to demonstrate the applicability of technology using machine learning and artificial intelligence that is becoming more prevalent due to the advancement of computing power. All the analysis is completed in real time with no intention to save any data from the user in any way, even though for practical corporation use more data would need to be stored and evaluated cumulatively to better reflect employees’ sentiments and thus match the dominant emotion index [29]. Importantly such an initiative would require an external technology assessment to be conducted at the very least, in addition to a privacy impact assessment to identify and mitigate the risks to both the employer and the employees.

SECTION VII.

Fairness in Facial Expression Recognition

A. Deep Learning and Uncertainty

Automated decision-making systems that employ machine learning algorithms have a major influence on human experience in modern society. With the proliferation of embedded data collecting devices throughout our shared environments, deep learning methods for smart cities are now both feasible and promising. While these methods offer powerful new approaches to data-informed city governance as well as the potential to transform the workplace, data driven systems have inherent limitations that must be understood in detail before being deployed at scale. While deep learning has been central to the success of machine learning in recent years, most modern deep learning models cannot easily or reliably quantify their own model uncertainty. These systems are often blindly assumed to learn and model accurate and fair representations of reality, and are with increasing frequency being employed to solve problems that have potentially serious social consequences. But this is often being done with little or no regulation, or with little or no oversight with regards to the impact on underrepresented groups within society. This is a grave problem concerning the future of machine learning, as currently most models fail to generalize to out of distribution samples (classes of data which were not present during training), and can face serious challenges when deployed in dynamically changing and uncertain environments.

B. Addressing Societal Dangers

While researchers outside of the ICT research community have been raising awareness of the potential societal dangers these systems can cause for many years, only recently, within the last 5-6 years, has there been a movement within the machine learning research community to address these issues. There are at least three important concepts currently used in the field that have the potential to partially mitigate these concerns within the workplace. First, Domain Adaptation will be a useful tool for adapting technologies to better serve employees from the group down to the individual level. Advancements in the realm of Open Set Domain Adaptation will be critical, since the successful deployment of these systems will require that they be able to distinguish between data that is anomalous or significantly different from that used in training, and to provide robustness against adversarial inputs and attacks. Additionally, Domain Adaptation, along with Active Learning, will prove invaluable in order to best take advantage of the immense wealth of unlabeled data that is collected by most modem corporations [30]. Secondly, Deep Reinforcement Learning is a promising technical framework from which to attack the problem of dynamic adaptation in the environment. Finally, Bayesian Deep Learning is a promising research thrust that has only recently been applied to the problem of fairness in machine learning. Bayesian Deep Learning is a probabilistic approach to deep learning that, among other things, enables model uncertainty to be naturally taken into account.

C. Types of Bias

**TABLE I** Types of Facial Expression Recognition Bias

In the context of Facial Expression Recognition (FER), two major problems that may contribute to unfair and discriminatory algorithms can be broadly categorized as biasing effects within the data and model overfitting. Although the FER problem has been well studied for several years now, almost no datasets for this task have been collected with a fair distribution of the human population in mind. This is concerning especially when considering that the related task of facial recognition has been thoroughly proven to have widespread problems with both racial and gender bias, with accuracies for females with darker skin tones reporting more than 30% lower than those for white males across the most popular facial recognition systems. The literature on causal factors underlying biases in these systems is vast [31]. Some important examples of such bias can be found in Table 1:

In addition, there are many definitions of fairness, most of which utilize some form of statistical test or measure, such as equalized odds, equal opportunity, demographic parity, and treatment equality. However, there is no agreed upon definition of fairness. In fact, such a universal measure may never exist given the difference in definitions of equity and fairness across varying cultures and social norms [32].

D. Assessing Fairness and Equity

In regards to FER, most datasets that currently exist that are exclusively designed for this task do not contain protected attributes such as race, gender, and age, making analysis of fairness difficult. However, recently in [33], a first ever study of its kind was conducted which attempts to assess bias and fairness for the FER task. This work sheds light on the problems in existing systems and provides some paths forwards for mitigation. The authors use three different models in their analysis. As a baseline model, the popular and widely used ResNet-18, which directly learns to predict an emotion category by minimizing a straightforward cross-entropy loss given the input image and not using any sensitive attributes. In the second model, sensitive attributes are mapped into an encoded representation via a linear layer. The representation vector from ResNet-18 for the image, along with this sensitive attributes representation vector, are then fed into a final classification layer to make the emotion predictions. The intuition for this approach is to analyze whether making the model explicitly aware of an individual’s sensitive attributes will have an effect on its ability to mitigate bias. Lastly, in the third model, a disentangled representation learning approach is utilized. To accomplish this, first a ResNet-18 model is used to obtain a feature representation vector from the image. From this point the model splits into multiple branches, with one primary branch for emotion prediction, and the other branches used to try to enforce fair ResNet feature learning by introducing a confusion loss that intuitively ensures that each of the sensitive attributes cannot be predicted using the learned ResNet feature vector.

Experiments for the study [33] using these three approaches were conducted on RAF-DB and CelebA datasets, which are some of the only datasets for this task that include gender, age, and ethnicity labels. RAF-DB was applied to the Future of Work prototype presented in this article, since it had the same label set (categories of emotions) as the FER model proposed to be used in our framework. Data augmentation was used to attempt to address the potential problem of model overfitting by randomly cropping, rotating, and mirroring training samples as well as increasing the global contrast of the images via histogram equalization. These steps serve to increase confidence that the following bias analysis and results in the study is not confounded with model Overfitting. Accuracy and equal opportunity (a particular measure of fairness) are used as evaluation metrics. Equal opportunity ensures that equal true positive rates are maintained across the various subpopulations in the data.

Results for the three models detailing the accuracy breakdowns by various sensitive attributes are depicted in [33, Table 3]. We can see that there is a fair amount of variation in performance on the baseline models, which are the most popularly used approaches for these tasks used in the field. However, using a disentangled representation learning approach shows promise in being able to reduce variance across subpopulation mean accuracies. This being said, for several groups, while variation across subpopulation mean accuracy is achieved by disentangled representation learning, accuracy for a given group may decrease. Furthermore, we see that in general, data augmentation leads to the best accuracies, but significantly increases variation in results across subpopulations within the baseline model. This is concerning as data augmentation is a standard method used for these tasks to improve accuracy and attempt to mitigate overfitting, and is thus widely used in practice.

Fairness metrics for the RAF-DB dataset are presented in [33, Table 4]. We see that in general, an employee’s age and joint Gender-Race group is more likely to affect results they receive from the FER model in this case. However again, for every category, the disentangled representation learning approach proposed increases fairness as measured by the equal opportunity metric used in the study.

When applying the results of the study in [32] to the Future of Work system, it becomes apparent that great care must be taken before using FER systems in the workplace, if we want to be able to guarantee that employees are not being discriminated against because of their race, gender, age, or other protected attributes. This is especially important if the predictions of the model are used in ways that affect an employer’s view of the employee’s job performance. Thorough audits should be conducted prior to deployment to characterize the fairness of the system with regards to various different statistical measures of fairness, if they are used.

SECTION VIII.

Human & Social Dimensions of Emotion Detection

The Future of Work project raises some fundamental human and social dimensions that require further exploration beyond the scope of this paper. These are listed below in summary only.

Corporate power and employee powerlessness- The subject is an employee-usually in a corporate structure. Employees do not have decision making power, hence exploring the policies and ethics applied to the FER context.
Governance issues- The legal ramifications of using FER for workplace surveillance, even when employees are remotely working from their own home, are significant. This leads to the following questions: how much oversight does, or should, the government have in such scenarios, if any? Do governments have the will to enact regulations to protect the rights of the citizens?
Data privacy issues- If our FER system is deployed, data privacy and security issues for both the individual and the corporation needs to be considered. As of the time of the demonstration system, there are no associated privacy risks as users are required to choose to experiment with the emotion recognition technology, and none of their data is stored on our database. Thus, by choosing to experiment on the demo platform, users have consented to have their image captured, processed and analyzed as needed for the sake of the use case scenario and addressing issues pertaining to user experience.
Social implications- What will be the social implications of such technologies on different demographics? Technological bias of the algorithms is a well-researched field. This leads to the following questions: Does deploying such a technology aggravate inequities in the workplace? Does this technology actually solve the problem, or were there better techniques to obtain the same results?

SECTION IX.

Conclusion

This article explored the emerging domain of facial emotion recognition (FER) as applied to a future of work use-case scenario. COVID-19 and additional forces have meant that more employees than ever before are working remotely, in addition to many workplaces selling off capital, maintaining that telecommuting is a long-term trend that is here to stay. So what might employers do to ensure their employee’s mental health and well-being? One suggestion is to invest in visual systems that can track and detect patterns based on one’s facial expressions. This would be to be proactive when employees present with possible negative emotions, before patterns of behavior present where employees develop a condition. In this article we have presented important elements that must be considered before full-blown deployment of a FER system, inclusive of the potential for inaccurate emotion readings, bias in the design of these systems and respective data sets, and how to overcome these fairness and equity issues. An interactive prototype of the FOW system is available for end-users to test the capability online. Human and social dimensions are prevalent in emerging technologies like these, but there are ways to address or overcome limitations and shortcomings of new technologies. One important aspect that maintain s user privacy in this prototype is that no images of employees are stored on hardware but that does not mean that all employers will do the same. Such systems will no doubt require government policy responses to ensure that regulation is able to introduce worker safeguards. While today’s technology may lead us to say that we can “read” emotions, from a psychologically point of view and in relation to health, the area is fraught with danger, with the potential for inaccuracies, and discrimination. Greater research must be conducted in the human and social dimensions, if the technology is really going to thrive.