Classifying encyclopedia articles

This repository contains the code developed for classifying French encyclopedia articles as part of the GEODE project.

Overview

This git repository contains the code developed for a comparative study of supervised classification approaches applied to the automatic classification of encyclopedia articles written in French. Our dataset is composed of 17 volumes of text from the Encyclopédie by Diderot and d'Alembert (1751-72) including about 70,000 articles. We combine text vectorization (bag-of-words and word embeddings) with machine learning methods, deep learning, and transformer architectures. In addition evaluating these approaches, we review the classification predictions using a variety of quantitative and qualitative methods. The best model obtains 86% as an average F-score for 38 classes. Using network analysis we highlight the difficulty of classifying semantically close classes. We also introduce examples of opportunities for qualitative evaluation of "misclassifications" in order to understand the relationship between content and different ways of ordering knowledge.

Experiments

Our experiments compare approaches to classifying EDdA articles using vectorization and supervised classification. We test the following combinations:

1. Bag-of-words vectorization and classic ML algorithms (Naive Bayes, Logistic Regression, Random forest, SVM and SGD);

2. Vectorization using static word embeddings (Doc2Vec) and classic ML algorithms (Logistic regression, Random Forest, SVM et SGD);

3. Vectorization using static word embeddings (FastText[fr]) and deep learning algorithms (CNN and BiLSTM);

4. An end-to-end approach using pre-trained contextual language models (BERT, CamemBERT) with fine-tuning to adapt the model for our task.

Results

All the results (measured with precision, recall and F-score) are in the reports directory. Some tables and figures are listed below:

Mean F-scores for different models for the test set with a sample of a maximum of 500 articles (1), 1 500 (2) and no limit (3).

F-scores for classes on the test set obtained with SGD + TF-IDF (1), BiLSTM + FastText (2) and BERT Multilingual (3).

F-scores obtained with BERT Multilingual and CamemBERT for each class

F-scores obtained with Naive Bayes + TF-IDF on each class with three different sampling.

F-scores obtained with SGD on each class with three different vectorizers without sampling.

F-scores for classes on the test set obtained with SGD + TF-IDF, BiLSTM + FastText and BERT Multilingual.

Confusion matrix for SGD+TF-IDF

This confusion matrix presents the results for the SGD+TF-IDF model on the test set. We see that most articles in the classes Arts et métiers and Economie domestique (Domestic economy) were classified as Métiers. In the same manner Mesure (Measurement), Minéralogie (Mineralology), Pharmacie (Pharmacy) and Politique (Politics) were confused with Commerce, Histoire naturelle (Natural history), Médecine - Chirurgie (Medicine - Surgery) and Droit - Jurisprudence (Law), respectively. The semantic similarity between these classes illustrates the difficulty a model has when choosing a “best match.” The results confirm that when there is great semantic similarity, the model chooses the best represented class in the dataset, thereby privileging certain ENCCRE domains that contain more articles.

Cite our work

Moncla, L., Chabane, K., et Brenon, A. (2022). Classification automatique d’articles encyclopédiques. Conférence francophone sur l’Extraction et la Gestion des Connaissances (EGC). Blois, France. https://hal.archives-ouvertes.fr/hal-03481219v1

Ackowledgements

Data courtesy the ARTFL Encyclopédie Project, University of Chicago.

This work was supported by the ASLAN project (ANR-10-LABX-0081) of Université de Lyon, within the program « Investissements d’Avenir » operated by the French National Research Agency (ANR).