When in the 1990s computer scientist Rosalind Picard startet to envision the field of affective computing, she reports, the problem of „too little emotion“ (x) in computers has not been adressed properly. Instead, decision making (which is integral part of computing/artificial inteligence) has been based on rationality only, which, as Picard argues empathically, is not enough to make computers „genuinley intelligent, to adapt to us, and to interact naturally with us“ (Picard x). Today, it seems, this genuinley artificial intelligence is still not on its way. What we do face, instead, is the rise of affect-sensitive technologies that started to shape and change social relations and political institutions. It is not only by companies advertising these technologies such as Affectiva (the Startup founded by Picard herself), that affect recognition is being deployed. We can find less explicit but similar technologies in social media anlytics as well as in user experience design (and: what, today, is not user experience by design?).
In this paper I argue that it is precisely because of the strategic ambiguity of affect as a concept (and not: despite of it) that this field of technological innovation has been flourishing in the last decade or so. And I will make the (short!) case for rationality (though I will not argue for conceptual precision!) when it comes to resistance and critique of these new technologies and their effects.
Affect as a topic of theoretical inquiry has a long and complex tradition, but it is only during the 1990s that it became subject of international academic discussion dubbed as „The Affective Turn“ (Clough/Halley). The writings of Brian Massumi, who draws on GillesDeleuze’s interpretations of Spinoza, have been playing a prominent role in these discussions. It is his rich and paradox notion ofaffect (and his poetic writing) that have set the cornerstone for the Affect Studies.
Massumi clearly rejects the belief that affect can be extensively captured by technological means. In his writings he argues for a notion of affect that is to be regarded as a characteristic of bodily and inter-corporeal processes, which take place primarily in a pre-individual world of becoming (aka the „virtual“). Thus, affect for Massumi is a “nonconscious, never-to-be-conscious autonomic remainder (…), as disconnected from meaningful sequencing, from narration, as it is from vital function” (25). On the other hand – and this is where Massumi is trying to give a comprehensive idea on how to access affect, on how to theorize it in the first place – he has to locate affect somehow, and sees it as “most directly manifested in the skin – at the surface of the body, at its interface with things ” (25). For Massumi, it isprecisely this tension between non-conscious residual and physical manifestation that enables a potential to think of affect as ontological power and source of resistance. In this, the body becomes a “sensor of change”, and functions as a “transducer of the virtual” (135). And, contrary to common usage, the virtual is not in digital technologies, it is not to be equated with simulation (Baudrillard), but something, „a mode of thought“ that only occurs in the analog and that digital technologies seek to capture/connect to in order to actualize.
Picard’s notion of affect, interestingly, is not that far away from Massumi’s. In her 1997 book on Affective Computing she states:
„If computers are to utilize the natural channels of emotional communication used by people, then when computers learn to recognize human emotion, they will have to rely primarily on sentic modulation, as opposed to having people explicitly tell them the names of their emotional feelings. To give computers affect recognition requires understanding the physical manifestations of emotion.“ (Picard 26)
Three things are remarkable in this statement: First, it is noticeable that Picard makes no distinction between affect and emotion. On the contrary, affect seems to be a kind of terminus technicus, if not to serve as an umbrella term here in order to encompass emotion, feelings, bodily reaction, even communicational subtleties. Second, Picard assumes that the bodies („the natural“ in the channels) and their channels of communication themselves (whatever they are) bring about a sort of modulational process that can be (technologically) sensed („sentic modulation“). In other words: she states that whatever people communicate through their bodies (may it be their voice, their mimics, or their electrodermal activity) is saying something true about people, even if they can not or do not want to say it themselves (or if they are saying something different). And third, it becomes explicit here that, according to Picard, affect or emotion can be accessed trough the physical manifestations of the bodies – but the nature of this manifestation remains open.
So, what are the possible manifestations? What manifestations are prevalent in nowadays affect-sensitive technologies? What channels exactly are there? What strategies do we find in this field of new innovations?
I think there are (at least) three main strategies (or: structures) that reside in the fields of the symbolic-dynamic, the expressive and the vital. They need to be considered when it comes to the ontological power of affect nowadays:
- The processing of symbolic input dynamics: Big parts of communication today is based on symbolic input, such as writing/typing words, clicking on links, liking pages, sharing pictures, retweeting, using emoticons, etc. This kind of input is being used in order to analyze the polarity, the vector, the emotional valence and affective patterns of users, the virality of content, the intensity of relationships and the effects of certain user experience modes („A/B-Testing“). Some of this input is also being displayed to its users, for instance when counting clicks or likes or assessing reviews with stars or maybe even providing „sentiment analysis“ for the readers of your online magazine (check out german newspaper „DieZeit Online“).
- The processing of expressive flows and movements: With ubiqutious computing came also ubiquitous camera and microphone usage usually recording non-stop. Given a certain quality of these recordings, they provide the basis for face and voice recognition. By analyzing the movement of muscles, of posture, the speed of speech or its tonality, even the rhythm of typing and clicking, extensive information on user’s reaction can be drawn from. This is not only used in marketing („Realeyes“, „Google Assistant“), but especially inpredictive policing.
- The processing of vital functions: Smartphones and wearables come with sensors detecting, among other things, activity, pulse, body temperature, even blood volume and oxygen level or skin conductivity. The more we equip ourselves, our homes and public spaces with (smart) sensors, the more this „inner“ information of bodies themselves becomes instructive for intervention, assessment and self-reflection.
Of course, these three strategies are not being used separately, they intertwine in complex and often obscure ways since this kind of processing mostly takes place in R&D-departments of big tech-companies or is being offered by startups as special insight given they are provided with the relevant data sets.
Nevertheless, what stands out, is the sort of knowledge this technolgical complex is about to produce (or at least is promising): affect-sensitive technologies are about extracting a „truth“ about subjects/users that escapes their own attention, ability or consciousness, so it becomes a more reliable „truth“ about them. This is why affect comes into play: not only is it a dynamic and relational category, it also promises insight into the body before the subject can even name it (or is willing to do so) since it works like an „autonomic remainder“ (Massumi). And, also, affect opens up the field of situative on-going: when using the body as „sensor of change“ (Massumi), many interesting conclusions can be made, may it concern bodily effects of environmental change, social effects (such as anxiety, depression or agreeableness) or political upheavals (at borders for instance, where infrared cameras are being used to detect refugees). Today, our bodies (their symbolic action, their expressive flows, their vital functions), our usage of ubiquitious and wearable digital media technology, and the growing confluence of bodies and media, is at the center of technological and capitalist interest. In this, affect-sensitive technologies produce a new, situative knowledge of the human, that not only changes what it means to be human today, but how humans can be governed, mobilized and tweaked, how they can be observed and predicted, on an individual scale as well as on a collective scale. And this is not only unprecedented, it still is pretty unclear where this will lead us (but since it seems to be powerfull, it will not necesserily lead into utopia). What becomes clear is that Massumi is probably right: The digital will never capture extensively the virtual (the analog/affect). But that is, unfortunately, the reason why there will be ever more venture capital and innovative energy flowing into technological advancements trying and promising exactly that: capturing the virtual/affect/the analog.
This brings us to the following conclusions: First, this new kind of knowledge about affects does not only play a big role in the tech-industry, where it is being capitalized and used to produce new media formats, or by the police or intelligence services. This kind of knowledge is also becoming part of our everyday life, may it be through the usage of smartphone apps that promise to help managing your feelings („Woebot“) or in social meda dynamics where knowing what others think and feel about certain content in real-time changes our societal self-awareness. So, when we say that these technologies are changing our affect cultures today, it seems, this happens through a new form of (real-time) perception of individual and societal reactions that we could call a big-data-based episteme (Foucault).
Second, we can not assume that the concept of affect (or any concept), if it is formulated only ambiguously enough and if it possibly connects us to „the virtual“ or the „pre-individual“ is already politically charged in the „right“ way and full of potential for resistance. The field of (capitalist) digital technologies is very eager to open up new markets and enter fields full of promises and openings, where imaginaries and potentials can be envisioned and, in the end, capitalized. So, what then could be a strategy for resistance?
There, for sure, is not just one answer to this. But, one, I would like to discuss, we can get from Massumi who draws on something I would like to call „poetic rationalism“. With this term I have a twofold strategy in mind: On one hand, it is obvious we need to talk about these kind of technologies, even though they mostly remain obscure or are hard to understand. Talking about it, generating (transmedial!) discourse, that could make perceivable what is going on when we use digital media today and what social/societal effects they bring about. And this, obviously, has already started. But, what needs to be the goal with this is provoking more „Einsicht“ (understanding, insight, sense) among users. It is not about bringing knowledge to the people/masses, it is about finding ways to describe and to adress these transformations in ways that it relates to people, that it affects and provokes involvement, to form „ideas“about what is going on (Spinoza). So, it is all about the mode of describing and discussing. If poetics is the art of „density“ (german: „Dichten“), coming from the greek word „poesis“(englisch: making/creating), it is about creating dense/thick descriptions that are able to escape from „meaningful sequencing, from narration“ (Massumi) but nevertheless can make you contemplate an idea of what is going on.
Baudrillard, Jean. Simulacra and Simulation, The University of Michigan Press. 1994.
Clough, Patricia T.; Halley, Jean. The Affective Turn. Theorizing the Social. Duke University Press. 2007.
Massumi, Brian. Parables for the Virtual. Movement, Affect, Sensation. Duke University Press. 2002.
Foucault, Michel. The Order of Things: An Archaeology of the Human Sciences.Vintage. 1994.
Picard, Rosalind. Affective Computing. MIT Press. 1997.
Spinoza, Baruch de. Ethics: Proved in Geometrical Order. Cambridge University Press. 2018.