"1. Belles-lettres and philo live in a very similar but are differently oriented, so they have a low normalized scalar-product\n",
"1. Belles-lettres and philo live in a very similar space but are differently oriented, so they have a low normalized scalar-product\n",
"2. Architecture and Métiers spread on more separate subspaces, but are almost entirely concentrated on the 4 dimensions they share\n",
"2. Architecture and Métiers spread on more separate subspaces, but are almost entirely concentrated on the 4 dimensions they share\n",
"\n",
"\n",
"But does increasing the number of top ranks efficiently weakens the effect of the noise ?"
"But does increasing the number of top ranks efficiently weakens the effect of the noise ?"
...
@@ -555,7 +555,7 @@
...
@@ -555,7 +555,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": 18,
"execution_count": null,
"id": "667fde5c",
"id": "667fde5c",
"metadata": {},
"metadata": {},
"outputs": [
"outputs": [
...
@@ -605,7 +605,177 @@
...
@@ -605,7 +605,177 @@
"id": "ed4c2139",
"id": "ed4c2139",
"metadata": {},
"metadata": {},
"source": [
"source": [
"Of course, the top10 is included in the top100, so the most prominent n-grams remain unchanged. What the above shows is that the \"long-tail\" of less-frequent 2-grams isn't enough to counter-balance the main, noisy, dimensions."
"Of course, the top10 is included in the top100, so the most prominent n-grams remain unchanged. What the above shows is that the \"long-tail\" of less-frequent 2-grams isn't enough to counter-balance the main, noisy, dimensions.\n",
"\n",
"Let's try and find less noisy top n-grams in the new domains we've made available."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b7a352d7",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{('a', 'b'): 649,\n",
" ('b', 'a'): 202,\n",
" ('ligne', 'droite'): 193,\n",
" ('a', 'a'): 149,\n",
" ('1', 'degré'): 142,\n",
" ('2', 'degré'): 129,\n",
" ('b', 'b'): 110,\n",
" ('s.', 'f.'): 110,\n",
" ('angle', 'droit'): 102,\n",
" ('2', '3'): 97}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"top10_2grams(maths)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "20caa4fb",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{('a', 'b'): 370,\n",
" ('1', 'degré'): 301,\n",
" ('2', 'degré'): 298,\n",
" ('ligne', 'droite'): 235,\n",
" ('s.', 'm.'): 228,\n",
" ('3', 'degré'): 208,\n",
" ('s.', 'f.'): 197,\n",
" ('m.', 'newton'): 180,\n",
" ('quart', 'cercle'): 180,\n",
" ('grand', 'nombre'): 166}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"top10_2grams(physique)"
]
},
{
"cell_type": "markdown",
"id": "d00e8eac",
"metadata": {},
"source": [
"Except for `('s.', 'f.')` (and `('s.', 'm.')` only for physics, which in itself is quite interesting — is maths vocabulary more feminine ?), most of the top 10 2-grams seem actually related to their respective domains. Moreover, neither of these substantive-related 2-grams are the most frequent in their domains, so they don't carry virtually all the weight of the vectors like they used to in the Architecture and Métiers domains above."
"Projection ratio \t0.9178237157824269 \t0.8284026983397814 \n"
]
}
],
"source": [
"compareTop10_2gram(maths, physique)"
]
},
{
"cell_type": "markdown",
"id": "b40c55f4",
"metadata": {},
"source": [
"Here something very interesting appears: the same phenomenon as before occurs, but it seems legitimate this time. Indeed, their scalar product is slightly greater than their intersection (5 out of 10 is 50% or 0.50, so not too far below 0.64), but their intersection is almost entirely populated by relevant 2-grams. This behaviour is even more pronounced if we increase the size of the top ranking:"
"Projection ratio \t0.8567301785743959 \t0.8338569246286951 \n"
]
}
],
"source": [
"compareTop100_2gram(maths, physique)"
]
},
{
"cell_type": "markdown",
"id": "57a1ee20",
"metadata": {},
"source": [
"Here the intersection is proportionately much smaller but the scalar product has only slightly diminshed, corresponding to the situation we had previously observed between Architecture and Métiers where the cell at their intersection in the confusion matrix will be darker using the scalar product instead of simply counting the number of most frequent n-grams they have in common. However, contrary to the explanation between Architecture and Métiers, this is not due to noise but to relevant n-grams. Like Architecture and Métiers, they too share their most fundamental features, but this time they are the ones which actually define them.\n",
"\n",
"This shows how studying the colinearity of the most frequent n-grams in addition to counting the common n-grams can help refine the analysis."
]
]
}
}
],
],
...
...
%% Cell type:markdown id:3401c7f3 tags:
%% Cell type:markdown id:3401c7f3 tags:
# So what's wrong with scalar product vs. elements intersection ?
# So what's wrong with scalar product vs. elements intersection ?
A short notebook to explain the "shift" in classes comparisons obtained from [that other notebook](Confusion%20Matrix.ipynb). Looking at the matrices produced we observe that the scalar product, which can be seen intuitively as more restrictive a metrics than elements intersection (counting the number of elements in the intersections of two sets can be viewed as a scalar product of vectors where all elements in each set are associated to the integer `1`, so you'd expect the more nuanced scalar product to be systematically inferior to that all-or-nothing metrics), sometimes make classes appear more similar than computing the number of n-grams they have in common. Let's see why, starting from an illustration of the issue at stake:
A short notebook to explain the "shift" in classes comparisons obtained from [that other notebook](Confusion%20Matrix.ipynb). Looking at the matrices produced we observe that the scalar product, which can be seen intuitively as more restrictive a metrics than elements intersection (counting the number of elements in the intersections of two sets can be viewed as a scalar product of vectors where all elements in each set are associated to the integer `1`, so you'd expect the more nuanced scalar product to be systematically inferior to that all-or-nothing metrics), sometimes make classes appear more similar than computing the number of n-grams they have in common. Let's see why, starting from an illustration of the issue at stake:
Both pictures are confusion matrices comparing the top 10 2-grams of each domain in the EDdA. The one on the left merely counts the number of 2-grams common between both classes whereas the one on the right uses the scalar product computed on the vectors obtained for each domain, by associating to each one of the most frequent 2-grams found in that domain its number of occurrences.
Both pictures are confusion matrices comparing the top 10 2-grams of each domain in the EDdA. The one on the left merely counts the number of 2-grams common between both classes whereas the one on the right uses the scalar product computed on the vectors obtained for each domain, by associating to each one of the most frequent 2-grams found in that domain its number of occurrences.
The cells circled in green, at the intersection between Belles-lettres and Philosophie, presents the expected behaviour: The blue circle is darker on the left, where they elements are simply counted, and appear lighter when the matrix is computed with the metrics derived from the scalar product. The one in purple (Architecture / Métiers), however, is counter-intuitive: the scalar product method makes them appear closer than they were with the elements intersection method. To understand how this is possible, let's look at the top 10 2-grams of the involved classes.
The cells circled in green, at the intersection between Belles-lettres and Philosophie, presents the expected behaviour: The blue circle is darker on the left, where they elements are simply counted, and appear lighter when the matrix is computed with the metrics derived from the scalar product. The one in purple (Architecture / Métiers), however, is counter-intuitive: the scalar product method makes them appear closer than they were with the elements intersection method. To understand how this is possible, let's look at the top 10 2-grams of the involved classes.
We first notice the occurrences of the 2-gram ('s.', 'm.') and ('s.', 'f.') for "substantif masculin" and "substantif féminin", which occur very frequently at the begining of article. Those are of course unwanted, irrelevant bigrams but they haven't been filtered out (yet ?). Let's look at the Philosophie domain now:
We first notice the occurrences of the 2-gram ('s.', 'm.') and ('s.', 'f.') for "substantif masculin" and "substantif féminin", which occur very frequently at the begining of article. Those are of course unwanted, irrelevant bigrams but they haven't been filtered out (yet ?). Let's look at the Philosophie domain now:
Interestingly enough, the Philosophie domain seems to comprise much fewer masculine substantives, so that ('s.', 'm.') didn't even make the top 10. The ('d.', 'j.') bigram is there too (probably the signature of an author ?), as well as ('-t', '-il'), ('1', 'degré'), ('a', 'point'), ('grand', 'nombre') and ('sans', 'doute'). Quite a populated intersection which account for the relatively dark blue patch in the matrix on the left (7/10).
Interestingly enough, the Philosophie domain seems to comprise much fewer masculine substantives, so that ('s.', 'm.') didn't even make the top 10. The ('d.', 'j.') bigram is there too (probably the signature of an author ?), as well as ('-t', '-il'), ('1', 'degré'), ('a', 'point'), ('grand', 'nombre') and ('sans', 'doute'). Quite a populated intersection which account for the relatively dark blue patch in the matrix on the left (7/10).
Indeed, if we look at the cardinalities associated to each 2-gram they share, they aren't synchronized but rather in phase opposition instead:
Indeed, if we look at the cardinalities associated to each 2-gram they share, they aren't synchronized but rather in phase opposition instead:
- the most frequent in belles_lettres, ('d.', 'j.') (485) is the least frequent in philo (82)
- the most frequent in belles_lettres, ('d.', 'j.') (485) is the least frequent in philo (82)
- the most frequent in philo, ('a', 'point') (191) is rather low in belles_lettres (67)
- the most frequent in philo, ('a', 'point') (191) is rather low in belles_lettres (67)
- the trend is general to the other 2-grams, except maybe ('s.', 'f.') which scores about the same in both domains
- the trend is general to the other 2-grams, except maybe ('s.', 'f.') which scores about the same in both domains
As a result, their contributions, if they do not entirely cancel out, at least do not manage to produce a dot product value high enough to overcome their norms, and the normalized scalar product is not very high.
As a result, their contributions, if they do not entirely cancel out, at least do not manage to produce a dot product value high enough to overcome their norms, and the normalized scalar product is not very high.
%% Cell type:markdown id:e1fccadd tags:
%% Cell type:markdown id:e1fccadd tags:
Now looking at the other pair reveals a different story:
Now looking at the other pair reveals a different story:
They only have four 2-grams in commons, but now three out of the four are the top 3 of each domain. That is to say, the "coordinates" that contribute the most to their norms are (almost all) the ones they have in common. Moreover, even the distribution in that top 3 is similar from one domain to the other:
They only have four 2-grams in commons, but now three out of the four are the top 3 of each domain. That is to say, the "coordinates" that contribute the most to their norms are (almost all) the ones they have in common. Moreover, even the distribution in that top 3 is similar from one domain to the other:
- they are in the same order: ('d.', 'j.'), ('s.', 'm.'), then ('s.', 'f.')
- they are in the same order: ('d.', 'j.'), ('s.', 'm.'), then ('s.', 'f.')
- their frequencies are almost proportional, with a ratio close to 0.5
- their frequencies are almost proportional, with a ratio close to 0.5
Of course, looking at these top 3 2-grams themselves is also very telling: they are not specific at all, there are the two substantive one we've previously noticed and the possible signature ('d.', 'j.'), so in fact, in addition to being common between Architecture and Métier, they are also shared by Belles-lettres, and two of them are found in Philosophie. They are rather like noise, found in apparently many domains.
Of course, looking at these top 3 2-grams themselves is also very telling: they are not specific at all, there are the two substantive one we've previously noticed and the possible signature ('d.', 'j.'), so in fact, in addition to being common between Architecture and Métier, they are also shared by Belles-lettres, and two of them are found in Philosophie. They are rather like noise, found in apparently many domains.
In any case, we have an explanation of how this counter-intuitive situation can occur: in terms of algebra, they appear to live in very different (10-D) spaces, but they actually have very low components on the 7 dimensions they don't share, on which they hardly depart from their main, 3-D components, which they share. In that 3-D space where they almost live, they have in addition a very similar direction, and thus their scalar product is very high.
In any case, we have an explanation of how this counter-intuitive situation can occur: in terms of algebra, they appear to live in very different (10-D) spaces, but they actually have very low components on the 7 dimensions they don't share, on which they hardly depart from their main, 3-D components, which they share. In that 3-D space where they almost live, they have in addition a very similar direction, and thus their scalar product is very high.
Put more simply, it means that both vectors have very little in common, but what they have in common outweights the rest, so that their most important feature is what they share. Let's generalize this approach by defining a generic routine to perform the comparison, evaluating the number of n-grams two domains share, their scalar product, and the contribution of each common n-gram to the norm of the vector.
Put more simply, it means that both vectors have very little in common, but what they have in common outweights the rest, so that their most important feature is what they share. Let's generalize this approach by defining a generic routine to perform the comparison, evaluating the number of n-grams two domains share, their scalar product, and the contribution of each common n-gram to the norm of the vector.
We'll first need to be able to project a vector on a given space (in practice, we'll use it to project on the n-grams it shares with another vector, that is, their "common subspace").
We'll first need to be able to project a vector on a given space (in practice, we'll use it to project on the n-grams it shares with another vector, that is, their "common subspace").
Projection ratio 0.9508045212368091 0.8633367067102022
Projection ratio 0.9508045212368091 0.8633367067102022
%% Cell type:markdown id:ed4c2139 tags:
%% Cell type:markdown id:ed4c2139 tags:
Of course, the top10 is included in the top100, so the most prominent n-grams remain unchanged. What the above shows is that the "long-tail" of less-frequent 2-grams isn't enough to counter-balance the main, noisy, dimensions.
Of course, the top10 is included in the top100, so the most prominent n-grams remain unchanged. What the above shows is that the "long-tail" of less-frequent 2-grams isn't enough to counter-balance the main, noisy, dimensions.
Let's try and find less noisy top n-grams in the new domains we've made available.
Except for `('s.', 'f.')` (and `('s.', 'm.')` only for physics, which in itself is quite interesting — is maths vocabulary more feminine ?), most of the top 10 2-grams seem actually related to their respective domains. Moreover, neither of these substantive-related 2-grams are the most frequent in their domains, so they don't carry virtually all the weight of the vectors like they used to in the Architecture and Métiers domains above.
Projection ratio 0.9178237157824269 0.8284026983397814
%% Cell type:markdown id:b40c55f4 tags:
Here something very interesting appears: the same phenomenon as before occurs, but it seems legitimate this time. Indeed, their scalar product is slightly greater than their intersection (5 out of 10 is 50% or 0.50, so not too far below 0.64), but their intersection is almost entirely populated by relevant 2-grams. This behaviour is even more pronounced if we increase the size of the top ranking:
Projection ratio 0.8567301785743959 0.8338569246286951
%% Cell type:markdown id:57a1ee20 tags:
Here the intersection is proportionately much smaller but the scalar product has only slightly diminshed, corresponding to the situation we had previously observed between Architecture and Métiers where the cell at their intersection in the confusion matrix will be darker using the scalar product instead of simply counting the number of most frequent n-grams they have in common. However, contrary to the explanation between Architecture and Métiers, this is not due to noise but to relevant n-grams. Like Architecture and Métiers, they too share their most fundamental features, but this time they are the ones which actually define them.
This shows how studying the colinearity of the most frequent n-grams in addition to counting the common n-grams can help refine the analysis.
