Semiotics, Language, and Mind, No.2:

The Tacit Made Legible

New artifacts create novel relations and understandings of the world. New materialities bring about new modes of acting and thinking.

—Lambros Malafouris[1]

We have no power of Intuition, but every cognition is determined logically by previous cognitions.

—Charles S. Peirce[2]

What’s really striking about AI poetry, prose, visual art, and music is not so much how impressively human-like they are, but how impressed we are by it.

—Philip Ball[3]

In October 2011 “Hello World!” a piece of contemporary classical music by Iamus — an artificial system designed to compose music — for violin-clarinet-piano trio was given its premiere at a music festival in Spain. Not long after, other compositions by Iamus were performed and recorded by the London Symphony Orchestra that later received two strikingly different reviews. Tom Service from the Guardian who reviewed it knowing beforehand that it was composed by Iamus, found it grey and unmemorable — the work of a machine "slavishly manipulating pitch cells" — though early in the review he did wonder whether he was "falling victim to a perceptual bias against a faceless computer program."[4] When a musicologist was asked to assess the same work from a recording but told nothing about its origin, he commented that the music was “artistic and delightful.”[5] Their different reactions prompted one to ask: how much of our judgment is formed based on a prior knowledge and assumption of a work’s origin and authorship and why?

The emergence of AI-generated art like its musical counterpart has stirred reactions of various kinds. Those who come from more traditional backgrounds tend to scorn any nonhuman creative endeavors, while those less traditional and more technically inclined tend to be less dismissive. Our approval and refusal of AI-generated art tell us a lot about ourselves and how we relate to art and music.

The aim of this series of essays is to understand why we prefer some creative expressions over others, using the advent of artificial intelligence as a lens for such an exploration. What makes artificial intelligence a good lens is that increasingly it is being designed to perform human tasks, its successes and failures serve as reflections of our own cognition and behavior.

Click here to read the complete score

Whereas the previous essay worked through the philosophy of mind, this one approaches the same problem from two other directions. First, we treat the work as a record of the artist-at-work and the artist-becoming. We use Lambros Malafouris's Theory of Material Engagement (MET) to show how ideas emerge through physical negotiations between the body, material, and tool rather than from the mind alone. Second, we consider the work from a judging perspective — drawing on Charles S. Peirce’s semiotics, we examine how value, meaning, and origin are inferred from signs, symbols, and indices. In short, this essay examines works of art to trace their makings, then to identify the signs by which we recognize and judge them.

When an artist works with brush and paint, camera, or other digital apparatus, the tool is usually absorbed into the identity of the artist. But artificial intelligence disturbs this arrangement because even as a tool, it appears to move closer to the position of agency itself. The artist no longer seems merely aided by a medium but partially displaced by a system that generates and recombines.*  But, tools, materials, and surfaces are not neutral conduits each carry a set of affordances and resistances that shape the conceptual and material directions of the final work. The work is the result of a negotiation between the artist and his medium.

To demonstrate, Artemisia Gentileschi’s Self-Portrait as the Allegory of Painting (La Pittura), c. 1638-9 articulates this physical negotiation between the artist and her tools. For this piece, rather than painting an elaborate interior, the artist has situated herself within a shallow visual depth of field. Her choice of colors in much of the space is restrained — yellowish brown, yellow, green, dark brown.  Within this controlled color palette, her subject — herself as painter — is the only element that’s illuminated by a flesh-color in sharp contrast with the rest of the colors in shade. Unambiguously, this is a work about the act of painting — a symbolic action representing the identity of the subject.  Meanwhile two fixed visual elements also mark her as a painter: the brush in one hand, the palette and spare brushes in the other.

The interaction between hand and tool establishes her identity as a painter. The actively tilted head and half-leaned torso, which shows her casting a gaze toward her subject further ahead. Tool and body together fix her in the act of painting. Lambros Malafouris's Theory of Material Engagement develops this point further by mixing the order of priority that traditional accounts of artistic agency assume. In this instance, the artist’s mind does not operate in isolation formulating ideas through the paint and brush. As evident in the active pose that Artemisia Gentileschi’s painting presents, it is a process by which the body, brush, paint, and environmental atmosphere co-operate together.

Artemisia Gentileschi, Self-Portrait as the Allegory of Painting (La Pittura), c. 1638–39. Royal Collection Trust / © Royal Collection Enterprises Limited 2026 | Royal Collection Trust.

In other words, the painting is not just a product of a mental effort, but physical skills manifested through the handling of her brush and paint. The work is the consequence of an active engagement during which mind and consciousness are just part of the whole. On the “hypothesis of material agency,” Malafouris writes:

The argument here is that what we call agency is not a human property but rather the relational and emergent product of situated activity. There is no way that human and material agency can be disentangled. Humans and objects work together to produce agency. So, the question we should ask is not “What is an agent?” but “When is an agent?” What an entity is in itself does not really matter. Instead, what matters is what it becomes and where it stands inside a community of action and material engagement.[6]

When looking at this painting under infra-red reflectography and x-radiography, the position of her right hand fingers was found to have been changed as the artist worked to resolve the area — lengthening the index finger after its initial placement.[7]  This discovery of the pentimento leads us, centuries later, to a natural conclusion that the work is not an execution of a mental effort alone, but an enactment between mind, material, and environment.

However, an AI intervention disturbs this arrangement because it is not viewed as a tool chosen by the artist to execute his work, but a system that seems to operate on its own and perform operations once attributed to the human mind. What is overlooked is its prior program design and data training. None of that can be achieved without human intervention. Importantly, by arguing the point of AI-generated art it simultaneously reminds us of the crucial role that a tool has in completing a work of art. Likewise, when we look at a photographic piece by an established photographer, seldom do we believe that the camera took the creative lead in the work even though it was indispensable in the success of the work.

We started this essay with an anecdote of the London Symphony Orchestra’s recording of a composition by Iamus to show how the prior knowledge of its origin produced a bias in the final review of the composition. To investigate why, we look to Charles S. Peirce to understand how we attribute value and meaning to the work we experience. In “One, Two, Three: Fundamental Categories of Thought and Nature,” Peirce lays out a triad of sign types — icon, which shows a similarity to the object; index, which points to the object without naming it; symbol, which names the subject by means of convention and habit.[8]  As the previous formal analysis established, the artist’s brush and palette denote the role of a painter by a habitual convention in classical European paintings. In Peirce's sense, they are symbolic marks attributing the foreground figure as a painter. The painting itself can be correctly assumed as an index, and the more interesting indices are in the details. For example, the necklace shown as if in mid-swing, the hair falling over one side, the lines breaking across the fabric of her sleeve all suggest physical movements of a body adjusting itself to study a subject.

What is instructive about all this AI-generated art commentary is that it makes legible that any work of art is a field of attribution.

After all, "a painter at work" is what we infer, assembled according to symbol, signifier, and index, each cuing a conclusion in our consciousness so fluently that the inference feels like simply seeing, when the truth is that we do not perceive the painter, we assume. What we read as the presence of an artistic mind in Artemisia Gentileschi's self-portrait is really a case of the tacit rendered legible.

If our judgment can be reorganized by signs, indices and external factors — such as prior knowledge of origin and provenance, then judgment must involve more than the physical object alone. Henceforth,the change of judgement when the composition’s origin is revealed to have been Iamus or AI-generated occurs in the interpretation phase. In other words, the only variable is in us because the work hasn’t been altered. In Languages of Art, Nelson Goodman analyzes a similar problem that shines light on the shifting of meaning when context is changed.  It begins with a question of what happens when a fake and an original appear to have no difference between them when displayed and viewed side-by-side. If both are equally convincing, then must both carry the same artistic merit? To Goodman, the answer is no because our ability to judge aesthetic value depends on prior training and education of each individual, that very knowledge instructs the viewer to look differently.[9]

By the same token, institutions rely on labels and provenance to elevate the cultural and historical value of the items on display. As a play on this institutional practice, contemporary artist Ai Weiwei has made a series of works challenging the symbolic values of artifacts and objects deemed as national treasures. His attack is on what the objects stood for, not on the physical objects per se. Though that argument requires a separate essay.

In Dropping a Han Dynasty Urn (1995), Ai Weiwei photographed himself dropping a ceramic vessel — assuming it was a genuine article, the artifact would be about two thousand years old — presenting the act as a triptych across three photographic frames: artist holding an urn, releasing an urn, shattered pieces on the ground. The entire narrative relies on the sequence of the three photographs to represent a continuity to establish a causal relation. As separate photographs, they do not hold the same narrative. Nothing in the photographs informs us the vessel is in fact a Han period artifact, it is assumed and inferred by the work’s title and by the artist’s reputation for being a rebel against the use of history as statecraft.

Ai Weiwei, Colored Vases (2007-10) and Dropping a Han Dynasty Urn (1995/2009) at the Hirshhorn Museum.
Photograph by Cathy Carver / Courtesy Hirshhorn Museum and Sculpture Garden.

When applying Peirce’s terms, the three sign-types in this work come apart cleanly. The photograph is an index of the act: a mechanical-causal trace of an urn being dropped to the ground; the images at the same time hold an icon of the gesture — it resembles a man holding something, opening his hands, and letting go. But the severing of oneself from statecraft and commodity staged as heritage is purely symbolic. The real conceptual part of the work waits on the spectator to complete by furnishing the historical connection. The index and the icon are on the surface; the symbolic meaning is attributed. Take away the title and the artist’s name, and the same work indexes a man holding a vase, a vase in mid-air that may and may not have been dropped by the man just because his hands are shown opening apart, and lastly it shows shattered pieces of a broken vessel that may have already been there before the man walked into the frame. What changes between priceless and worthless is never anything in the object, it is the viewer’s attribution.

This is how the disclosure of a work’s origin can reorganize its judgment while nothing in the work has changed. In “Art and Authenticity,” Goodman makes this process very clear by saying: “My knowledge of the difference between the two pictures, just because it affects the relationship of the present to future lookings, informs the very character of my present looking. This knowledge instructs me to look at the two pictures differently now, even if what I see is the same.”[10]  In other words, the name and the title on the label is not external to conscious perception; they become part of the sign-system through which the work is understood.

When a label is “AI-generated” it reorganizes the looking itself. AI-generated art, of course, does not introduce the inference by which we move from sign to maker but interrupts it — unsettling the index that bears the mark belonging to a human body. What is instructive about all the AI-generated art commentary is that it makes legible that any work of art is a field of attribution. As in the Iamus anecdote at the start of this essay, when the source is unknown the quality of the piece is deemed significantly better than when its origin is revealed. So, what were people responding to in the first place before knowing it is not by human? In our recent conversation with Philip Ball, he gave us this answer: “We respond to many aspects of it, generally in terms of patterns and regularities. That’s how we perceive melodies: as organized features in sequences of notes. And we respond to those by generating expectations of what will come next, and our emotions are engaged according to whether our expectations are fulfilled or not.”

What changes between priceless and worthless is never anything in the object, it is the viewer's attribution.

To conclude, Peirce wrote: “as the thought is determined by a previous thought of the same object, it only refers to the thing through denoting this previous thought.”[11]  By this theory, the attribution of artistic merit always involves an inference — a hypothesis based on prior knowledge and training. This mental process is formed so habitually that it translates seamlessly into our conscious awareness causing us to trust our judgments as sound and correct. When AI-generated works are being dismissed by some more traditional critics and reporters, it is evident they were encountering something unfamiliar, something that does not echo their inner expectations, and therefore had no appropriate language or thoughts to properly understand it.  Obviously, the advent of AI-generated art did not introduce inference into aesthetic judgment, rather it renders more obvious that no judgement of any cultural expression is pure and untainted by one’s own personal history. This is how the tacit is rendered legible.

[1] Lambros Malafouris, People Are STRANGE: Material Engagement and the Creation of Self-Consciousness (Cambridge, MA: MIT Press, 2026), 63.

[2] Charles Sanders Peirce, Peirce on Signs: Writings on Semiotic, ed. James Hoopes (Chapel Hill: University of North Carolina Press, 1991), 57.

[3] Philip Ball, The Book of Minds: How to Understand Ourselves and Other Beings, from Animals to AI to Aliens (Chicago: University of Chicago Press, 2022), 314. Also see Axis of Culture: Conversation with Philip Ball.

[4] Tom Service, “Iamus’s Hello World! – Review,” The Guardian, July 1, 2012, https://www.theguardian.com/music/2012/jul/01/iamus-hello-world-review

[5] Sylvia Smith, “Iamus: Is This the 21st Century’s Answer to Mozart?,” BBC News, January 3, 2013, https://www.bbc.com/news/technology-20889644

[6] Malafouris, People are STRANGE, 53.

[7] Royal Collection Trust, “Self-Portrait as the Allegory of Painting (La Pittura),” accessed June 18, 2026, https://www.rct.uk/collection/405551/self-portrait-as-the-allegory-of-painting-la-pittura.

[8] Peirce, Peirce on Signs, 141.

[9] Nelson Goodman, Languages of Art: An Approach to a Theory of Symbols, 2nd ed. (Indianapolis: Hackett Publishing Company, 1976), 103–5.

[10]Ibid., 104.

[11]Peirce, Peirce on Signs, 68.

*Many objections to generative-AI concern the unauthorized mining of other artists’ works as data, and the outcome is an averaging of the artistic styles held in the database. This is a valid concern when images and works are being used without legal authorization.
This essay series does not adjudicate the aesthetic value of AI-generated art against work made with more traditional mediums by humans. The project began from a simple objective: to learn — through our varied reactions to artificial intelligence — more about ourselves. In the recent surge of artists and designers turning to AI for human tasks, I saw that studying the successes and failures of such endeavors offers a natural opportunity to examine our own cognition and behavior. Visual art is an apt subject precisely because it leans so heavily on subjective views. 
Back to Series Page
Written by Chennie Huang