TF-IDF Geometry: Angle, Length, and What Similar Means

Six short documents, one weighting switch: watch cosine similarity separate topic from document length.

Ch 7 7.2

Explorer

Unsupervised Learning & Text

16 mindifficulty 2/5

Controls

Term weighting

Sublinear tf

1 + log(count) instead of the count.

L2 normalize rows

Project every document onto the unit sphere.

Drop stop words

Remove a 20-word English stop list.

Inspect document

The corpus

D1 · clinical · 22 tokens
D2 · clinical · 20 tokens
D3 · sports · 19 tokens
D4 · sports · 18 tokens
D5 · sports injury · 19 tokens
D6 · long clinical · 45 tokens

Cosine similarity

Angle between documents. Darker means more similar.

doc	D1	D2	D3	D4	D5	D6
D1 clinical	1	0.87	0.80	0.78	0.87	0.93
D2 clinical	0.87	1	0.77	0.81	0.90	0.86
D3 sports	0.80	0.77	1	0.93	0.90	0.78
D4 sports	0.78	0.81	0.93	1	0.87	0.80
D5 sports injury	0.87	0.90	0.90	0.87	1	0.84
D6 long clinical	0.93	0.86	0.78	0.80	0.84	1

Euclidean distance

Length-sensitive: the same topic at twice the length looks far away.

D1	0	4.2	5.2	5.5	4.4	7.9
D2	4.2	0	5	4.5	3.3	9.7
D3	5.2	5	0	2.6	3.2	10.8
D4	5.5	4.5	2.6	0	3.6	10.7
D5	4.4	3.3	3.2	3.6	0	10.2
D6	7.9	9.7	10.8	10.7	10.2	0

Top terms in D1

Weight under the current scheme, relative to the largest.

1. the df=6
2. patient df=3
3. and df=6
4. in df=5
5. is df=6
6. clinic df=4
7. doctor df=4
8. knee df=4
9. pain df=2
10. reports df=2

cos(D1, D6)

0.935

same topic, D6 nearly three times longer

‖D1 − D6‖

7.94

Euclidean, same pair

cos(D1, D3)

0.804

clinical versus sports

Signal margin

0.131

same-topic minus cross-topic cosine

Vocabulary size

idf of “the”

1.0 when a term is in every document

Why the margin is the number to watch

A retrieval system is only useful if same-topic pairs score higher than cross-topic pairs. Turn on TF-IDF and read the signal margin: down-weighting terms that appear everywhere widens the gap without touching the documents.

Predict before you look

Two notes cover the same topic, but one is three times longer. Under raw word counts, which is true?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A six-document corpus small enough to read in full, vectorized live under four switches: raw counts or TF-IDF, sublinear term frequency, L2 normalization, and stop-word removal. Both a cosine matrix and a Euclidean matrix are shown, so the effect of each choice on “similar” is immediate.

The vector space model

Each document becomes a vector over the vocabulary. Raw counts make the vector's length grow with the document, so Euclidean distance between a short and a long note on the same topic is large even when their term proportions agree.

Cosine similarity divides out the norms and measures only direction, which is why it is the default in text retrieval. L2 normalization makes this explicit: after normalizing, the dot product is the cosine and Euclidean distance is a monotone function of it.

IDF down-weights terms that occur in many documents. With idf = log(N/df) + 1, a term in every document gets weight 1 while a term in one of six gets about 2.8, so discriminative words dominate the direction of the vector. Sublinear term frequency additionally caps the effect of a word repeated many times in one document.

tf-idf(t, d) = tf(t, d) · (log(N / df(t)) + 1)

cos(d₁, d₂) = ⟨v₁, v₂⟩ / (‖v₁‖ ‖v₂‖)

after L2: ‖v₁ − v₂‖² = 2(1 − cos(d₁, d₂))

Try this

With raw counts, read cos(D1, D6) and ‖D1 − D6‖. Same topic — why do the two measures disagree?
Switch to TF-IDF and watch the signal margin (same-topic minus cross-topic cosine).
Inspect D6's top terms before and after TF-IDF: which word loses its dominance?
Turn on stop-word removal with raw counts. Does it recover the same margin TF-IDF gives?
Look at D5, a sports note about a clinic visit. Under which settings does it sit closer to the sports pair?

Common misconception

“Documents are points in space, so Euclidean distance tells me which are alike.”

With bag-of-words counts, vector length encodes document length rather than content. A 60-token clinical note and a 20-token clinical note differ mostly in norm, so Euclidean distance ranks them far apart while cosine — which only reads the angle — ranks them together. Text retrieval therefore compares directions, using L2 normalization or cosine, and re-weights terms by IDF so that words appearing everywhere stop dominating the direction.

DSPA3 Chapter 7 — Text mining and NLP