Factum Foundation has initiated a collaboration with a team from the Physics, Materials Science and Engineering, and Art History and Art Departments at Case Western Reserve University in Cleveland, Ohio. The goal is to apply machine learning methods to surface profilometry images of paintings in order to attribute stylistic components of brushwork in some of the later paintings of El Greco, with the aim of exploring differences between the hand of El Greco, those of his son Jorge Manuel and his workshop, and later conservation interventions.

The team at Case Western Reserve has developed image analysis and machine learning techniques via a series of controlled experiments using a series of painted images of lilies by artists from the Cleveland Institute of Art. In order to simulate workshop methods and produce sufficient training data, the team created virtual patches in each of twelve paintings, three by each of four artists. After training and validation on two of the three paintings, testing on the remaining four paintings was carried out using machine learning methods.

They were able to correctly attribute 98% of the patches, a significant result given that the patch size was a fraction of a complete brush stroke. The methods made it possible to confidently attribute brushwork even when the virtual patches were as small as a few bristle widths. 

The implications of this project for the art world are wide-reaching and game-changing. Connoisseurs will soon have a new tool to assist with thorny attribution questions for many paintings. This project is one of several under the umbrella of art.lab.cle, all of which are transdisciplinary collaborations using 21st-century technology which will have a major impact on our understanding of art.

The project comprises a close collaboration with the Fundación Casa Ducal de Medinaceli and the Hospital de Tavera in Toledo, and with The Auckland Project in Bishop Auckland.

You can read the results of the preliminary study carried out by Case Western Reserve University here. 

[...] Artificial intelligence (AI) for image recognition and image analysis has been successful in recent years, with applications ranging from art to ecology, national security to healthcare. The same techniques for clustering and recognising images can be applied to any 2D array of image data, such as a render of surface topography. In the case of the El Greco paintings, one experiment will be to train a machine learning (ML) algorithm on regions that a specific connoisseur has attributed to El Greco (Category A) vs. regions attributed to his son (Category B). Then we will apply the trained algorithm to the unknown regions. By applying the algorithm to unknown regions, the AI will suggest with a numerically defined certainty that these regions fit best into one of the trained categories (e.g., Category A with 85% certainty).

We expect the results to illuminate the physical qualities used in determining attribution, and perhaps lend insight into the role of reconstruction and damage in this determination. Further, information encoded at different length scales within the paintings will contribute to these results. Understanding which length scales the algorithm weighs highly for its determination could result in novel analytical tests for studying paintings. Additionally, we plan to study various imaging modalities, such as panoramic photography, for the ML algorithm, and understand the robustness of the results, so that we can begin to identify the surface information (height, directionality, colour, etc.) contributing to the attribution.

Finally, clustering via AI can be applied to the measurements of the paintings. With clustering, the AI will group similar patches of the paintings with no training. By using various metrics to cluster the data spots or spot sizes, we may see groupings arise within and across the paintings. Then, meaning can be designated to the groupings using knowledge obtained by archival and art-historical research. This method has been known to lend support to conclusions and enlighten new roads of interpretation. Initial tests on the ability of machine learning to differentiate between artists using surface profile images have been carried out using a methodology developed at Case Western Reserve University in Cleveland. Working with high-resolution 3D recording and data analysis software in a scientifically controlled environment, the team was able to identify the brush-marks made by two different hands with more than 90% accuracy after initial analysis. The three paintings by El Greco contain more ‘noise’ and have undergone several undocumented interventions that will obscure some data. By use of different models and careful training using sections of each painting, it is hoped that we will be able to allocate characteristic traces to the way each hand applies the paint. It is anticipated that this will reveal the presence of several hands at work on each canvas.

The project aims to offer a unique look at these paintings by El Greco through the filter of cutting-edge technologies. It is hoped that through digital analysis we will provide new data about how works are made, who made them, and how they have been conserved, demonstrating that paintings are both complex and dynamic in terms of content, context and material evidence. The recording and ML analysis will hopefully contribute to the knowledge and appreciation of some of the most representative works of the last stage of El Greco’s career. Factum Foundation has been committed from its formation to the recording of the surface of paintings as a way to learn more about their historical and artistic significance. The link with Case Western Reserve University will result in the writing of new software to analyse the 3D information.

Adam Lowe, The hand of the artist. Graph analysis and El Greco, in 'The Aura in the Age of Digital Materiality', 2020

A team from Factum Foundation carried out the high-resolution recording of the Crucifixion by El Greco within the collection of the Cleveland Museum of Art, Ohio. Both the surface data and accurate colour were acquired using the Lucida 3D Scanner and composite photography.

The project aims to document and analyse late paintings by El Greco and his studio, in order to establish the importance of recording the surface of paintings and reveal new information about the working practice of the artist’s studio through the use of machine learning algorithms.

Another version of the Crucifixion will soon be recorded in the Spanish Gallery at Bishop Auckland. 

The Lucida 3D Scanner recording the surface of the Crucifixion © Factum Foundation

Carlos Bayod Lucini operating the Lucida 3D Scanner to record the surface of the painting © Teresa Casado for Factum Foundation

Adam Lowe was also a keynote speaker at the 47th Annual Cleveland Symposium by Case Western Reserve University, centered on the topic “Aura: Authenticity, Experience, and Art” and hosted at the Cleveland Museum of Art on November 12.

Following conversations about the extraordinary painting of the Crucifixion of St Andrew with Dean Yoder, Conservator of Paintings at the Cleveland Museum of Art where the original hangs, a copy of the painting in the Museo de Santa Cruz in Toledo was also recorded in 3D and colour. The painting in Toledo, in poor condition but of great interest, is one of three copied versions of the original.

The painting was digitised in 3D and colour, using the Lucida 3D Scanner and composite photography, over a period of four days starting on December 14. 

The Lucida 3D Scanner recording the surface of the Crucifixion © Osama Dawod for Factum Foundation

The Lucida 3D Scanner recording the surface of the Crucifixion © Osama Dawod for Factum Foundation

The Lucida 3D Scanner recording the surface of the Crucifixion © Osama Dawod for Factum Foundation