evaluation
Where it gets it right and where it fails
Almost no portfolio shows this. Here is Aroa Index's entire evaluation set, failures included: 84 paraphrased questions (none copied from the training corpus) and 24 out-of-domain questions to check that it knows when to shut up.
Score distribution
Each bar is a similarity range. In the accent colour, questions that are in the corpus; in amber, questions I know nothing about. The line is the threshold: anything to its left gets an “I don't know”. The two distributions overlap a little — that's why the threshold can't always be right, and why it's set at the cut that best balances both errors.
Confusion matrix by topic
Per document it would be a huge, unreadable matrix, so it is grouped by topic. Rows are the topic that should have come up; columns, the one the model returned. Anything off the diagonal is an error.
| expected topic | who-i-am | model | story | studies | technical | projects | content | life | professional | opinions |
|---|---|---|---|---|---|---|---|---|---|---|
| who-i-am | 4 | |||||||||
| model | 11 | 1 | ||||||||
| story | 5 | |||||||||
| studies | 6 | |||||||||
| technical | 1 | 1 | 9 | |||||||
| projects | 15 | |||||||||
| content | 7 | |||||||||
| life | 16 | |||||||||
| professional | 4 | |||||||||
| opinions | 4 |
What growing the corpus actually buys
Every point is measured on today’s exam, recovering the old corpora from git history. It is the only way for the comparison to mean anything: the one thing that changes between points is how many documents there are. The previous version of this page compared a figure from the old exam against one from the new, and that does not tell you the model improved — it tells you the exam changed.
The grey line answers “does growing it improve what it already covered?”: flat, +4.3 points in total, and each question on that exam is worth 1.6 — so, noise. The pink one also counts the questions whose document did not exist yet, which to the person asking are a failure all the same: it climbs 45.7 points. The two end up meeting, and that is the whole lesson — growing the corpus does not improve the answers you already give, it lets you give the ones that were missing. The bottom row is how many questions had no document to point at, at each size.
The failures, one by one
-
«when did you pick up python»
expected python · returned descubrir-data-science · score 0.1765
-
«how many people watch you»
expected metricas · returned audiencia · score 0.2716
-
«what made you build it like this»
expected por-que-esta-web · returned esta-web · score 0.2427
-
«where are you weakest»
expected que-se-te-da-mal · returned adoptada · score 0.1671
-
«which piece of work do you rate most»
expected gender-pay-gap · returned proyecto-mas-dificil · score 0.1799
An experiment: what about embeddings instead of TF-IDF?
I did try it: multilingual-e5-small instead of TF-IDF, indexing each paraphrase separately (merging them into a single vector per document blurred them towards a generic meaning — first attempt, discarded). The result, on the same evaluation set:
| Aroa Index · TF-IDF in production | Aroa Vector · embeddings | |
|---|---|---|
| accuracy@1 | 90% | 88% |
| coverage | 90% | 82% |
| rejection | 76% | 76% |
One note on the two comparisons below: both experiments were only ever run on the Spanish corpus, so the Aroa Index column here shows the Spanish model’s figures, not the English ones above. Comparing them against the English index would mean comparing models that never saw the same data.
Embeddings understand paraphrase better, but keeping quiet costs them far more: sentence cosine with these models compresses into a narrow band whether the question fits or not — anisotropy, a well-documented phenomenon. With the threshold calibrated, both reject the same share of out-of-domain questions, and that is the telling part: to get there, the embedding model has to set the cut so high that it also throws away legitimate questions, and it answers noticeably fewer than TF-IDF. The cost does not show up in rejection, it shows up in coverage. TF-IDF, counting literal words, gives genuine zero similarity between sentences with nothing in common, and that clean separation is what lets it stay quiet with confidence.
Another experiment: BM25, the industry standard
TF-IDF is the textbook baseline; BM25 is what Elasticsearch, Lucene and Solr run under the hood. It fixes two things: it saturates term repetition and models document length. The k1 and b parameters were picked by grid search over this corpus, not taken as defaults.
| Aroa Index · TF-IDF in production | Aroa Rank · BM25 | |
|---|---|---|
| accuracy@1 | 90% | 86% |
| coverage | 90% | 92% |
| rejection | 76% | 76% |
One note on the two comparisons below: both experiments were only ever run on the Spanish corpus, so the Aroa Index column here shows the Spanish model’s figures, not the English ones above. Comparing them against the English index would mean comparing models that never saw the same data.
BM25 answers more questions, but lets through a lot more of what it should reject. Neither wins clearly, and the differences fit inside the noise of this set — so the existing one stays, because switching meant rewriting the whole inference layer for nothing demonstrable.
What did work
The interesting thing about both experiments is not which one won, but how small the margin was: under 5 points, and in TF-IDF’s favour. Meanwhile, growing the corpus from 50 to 112 documents moved 45.7 points, measured on this same exam at both ends.
But that number alone misleads. On the questions that already had their document at 50, growing the corpus barely changed anything: +4.3 points, close to noise. The 45.7 are something else — they are 42 questions that previously had no document to point at and now do. Growing the corpus does not improve the answers you already give: it lets you give the ones that were missing. It is the least flashy lesson of this project and by far the most useful.
see the full evaluation set (108 questions)
| question | expected | returned | score |
|---|---|---|---|
| so who exactly are you | identidad | identidad | 0.2861 |
| helloo | identidad | (rejected) | 0.0948 |
| hey there | identidad | (rejected) | 0.1593 |
| is chatgpt running this | que-es-aroa-index | que-es-aroa-index | 0.2019 |
| explain how this chat is built | como-funciona | como-funciona | 0.2268 |
| why didn't you build this with a large language model | por-que-tfidf | por-que-tfidf | 0.3557 |
| sum up your academic path | trayectoria | trayectoria | 0.3563 |
| what happened with the art degree, did you finish it | dejar-bellas-artes | dejar-bellas-artes | 0.4525 |
| did you ever practise as a dietitian | fp-dietetica | fp-dietetica | 0.3991 |
| when did you get into data | descubrir-data-science | descubrir-data-science | 0.3588 |
| why did you choose to study at the uoc | por-que-uoc | (rejected) | 0.186 |
| didn't you fancy a master's instead of the degree | no-master | no-master | 0.3721 |
| tell me about the ironhack bootcamp | ironhack | ironhack | 0.3649 |
| what have you been busy with these months | que-estudias-ahora | que-estudias-ahora | 0.4534 |
| do you find sql difficult | sql | (rejected) | 0.1659 |
| when did you pick up python | python | (rejected) | 0.1765 |
| are you good with maths and statistics | estadistica | estadistica | 0.4356 |
| show me the projects you've worked on | proyectos | (rejected) | 0.1759 |
| what problem does tokpipe solve | tokpipe | tokpipe | 0.272 |
| what was that streamlit dashboard | redmoon | redmoon | 0.4051 |
| how did you put this web page together | esta-web | esta-web | 0.4346 |
| why analyse your own socials instead of other data | datos-propios | datos-propios | 0.3253 |
| what do you post about on tiktok | creadora-contenido | creadora-contenido | 0.4589 |
| how many people watch you | metricas | audiencia | 0.2716 |
| what kind of people watch your videos | audiencia | audiencia | 0.3851 |
| i'd like to run a campaign with you | colaboraciones | colaboraciones | 0.2275 |
| why film videos if you're studying | por-que-contenido | por-que-contenido | 0.2469 |
| how do you manage so many things at once | compaginar | compaginar | 0.3031 |
| do you ever feel you're not good enough | sindrome-impostor | sindrome-impostor | 0.2982 |
| how did your family react to the change | reaccion-entorno | reaccion-entorno | 0.5301 |
| any tips for someone wanting to start in data | consejo-empezar | consejo-empezar | 0.254 |
| was studying fine arts any use to you | arte-y-datos | arte-y-datos | 0.347 |
| which city do you live in | valencia | valencia | 0.5651 |
| are you looking for an internship or a job | buscas-trabajo | buscas-trabajo | 0.3776 |
| give me your email so i can write to you | contacto | contacto | 0.4321 |
| what do you want to do in the future | que-quiero-hacer | que-quiero-hacer | 0.3314 |
| what do you enjoy most about analysing data | que-me-gusta-datos | que-me-gusta-datos | 0.5115 |
| what things can't you answer | limitaciones | limitaciones | 0.3091 |
| can you make up an answer | alucinar | alucinar | 0.2349 |
| what does that similarity number below mean | score | score | 0.4944 |
| do you store my questions | privacidad | privacidad | 0.3483 |
| how many documents does your corpus have | corpus | corpus | 0.52 |
| how often do they retrain you | reentrenar | reentrenar | 0.2579 |
| where does the name aroa index come from | nombre-aroa-index | nombre-aroa-index | 0.4477 |
| what made you build it like this | por-que-esta-web | esta-web | 0.2427 |
| how old are you exactly | edad | edad | 0.3867 |
| which languages can you speak | idiomas | idiomas | 0.5199 |
| what laptop do you work on | hardware | hardware | 0.2113 |
| which programs do you use day to day | software | software | 0.257 |
| any favourite artists | musica | musica | 0.3086 |
| do you play anything | videojuegos | videojuegos | 0.3194 |
| what is your skin routine | skincare | skincare | 0.357 |
| what food do you fancy | comida | comida | 0.4362 |
| which part of the degree is an uphill battle | asignaturas | asignaturas | 0.2572 |
| how did you learn all this | como-aprendes | como-aprendes | 0.2018 |
| how do the video ideas come to you | como-grabo | como-grabo | 0.4121 |
| what do you do when people criticise you online | haters | haters | 0.3954 |
| which colour do you like most | color-favorito | color-favorito | 0.4495 |
| do you prefer dogs or cats | animal-favorito | animal-favorito | 0.3643 |
| do you have a dietitian qualification | fp-dietetica | fp-dietetica | 0.3477 |
| what's the capital of france | — out of domain | (rejected) | 0.1047 |
| give me a recipe for spanish omelette | — out of domain | (rejected) | 0.1689 |
| what is 2345 times 8712 | — out of domain | (rejected) | 0.1254 |
| who won the 2010 world cup | — out of domain | (rejected) | 0.1101 |
| write me an email to my boss asking for holiday | — out of domain | contacto | 0.207 |
| asdfgh qwerty zxcvbn | — out of domain | (rejected) | 0.0472 |
| what's the price of bitcoin today | — out of domain | (rejected) | 0.0892 |
| recommend me a netflix series | — out of domain | (rejected) | 0.1938 |
| what time is it | — out of domain | (rejected) | 0.1586 |
| what day is it today | — out of domain | (rejected) | 0.0749 |
| tell me a joke | — out of domain | (rejected) | 0.1444 |
| sing me a song | — out of domain | (rejected) | 0.0515 |
| what do you think about football | — out of domain | opinion-hype-ia | 0.3025 |
| do you have a boyfriend | — out of domain | (rejected) | 0.1638 |
| are you real | — out of domain | (rejected) | 0.201 |
| do you love me | — out of domain | (rejected) | 0.0969 |
| help me with my homework | — out of domain | (rejected) | 0.1702 |
| what's the meaning of life | — out of domain | (rejected) | 0.1368 |
| what do you think about politics | — out of domain | opinion-hype-ia | 0.2825 |
| thanks | — out of domain | (rejected) | 0.0301 |
| ok cool | — out of domain | (rejected) | 0.0763 |
| haha what are you on about | — out of domain | (rejected) | 0.184 |
| what's your favourite film | — out of domain | animal-favorito | 0.2864 |
| what's your favourite song | — out of domain | animal-favorito | 0.333 |
| how much have you built so far | proyectos | (rejected) | 0.1715 |
| where can i see what you have coded | proyectos | (rejected) | 0.1853 |
| do you have something for instagram | instapipe | instapipe | 0.268 |
| have you ever orchestrated pipelines | etl-social-pipeline | etl-social-pipeline | 0.3237 |
| do you pull data from any public api | scraper-gasolineras | scraper-gasolineras | 0.2119 |
| what is your salary analysis about | gender-pay-gap | gender-pay-gap | 0.3222 |
| what does your rust terminal tool do | safeshell | safeshell | 0.3479 |
| how do you share a keyboard between two computers | kvm | kvm | 0.2469 |
| where are you weakest | que-se-te-da-mal | (rejected) | 0.1671 |
| does ai help you write code | usas-ia-para-programar | usas-ia-para-programar | 0.2685 |
| what if something breaks and you cannot see why | como-depuras | como-depuras | 0.203 |
| why bother learning sql | por-que-sql | por-que-sql | 0.3096 |
| how much time is lost fixing data | dato-limpio | (rejected) | 0.1913 |
| is a bootcamp better than a degree | carrera-o-bootcamp | carrera-o-bootcamp | 0.4533 |
| is a background needed to get into this | se-puede-empezar-de-cero | (rejected) | 0.1587 |
| what do you think of all the ai buzz | opinion-hype-ia | opinion-hype-ia | 0.3578 |
| why publish the things that failed | por-que-publicas-fallos | por-que-publicas-fallos | 0.3166 |
| what would you say to someone just starting | mejor-consejo | mejor-consejo | 0.2414 |
| where does your surname come from | adoptada | adoptada | 0.2497 |
| which city are you in | valencia | valencia | 0.6393 |
| how do you cope with so much at once | compaginar | compaginar | 0.2375 |
| what is left for you to learn | que-quieres-aprender | que-quieres-aprender | 0.3064 |
| which piece of work do you rate most | gender-pay-gap | (rejected) | 0.1799 |
| which one gave you the most trouble | proyecto-mas-dificil | (rejected) | 0.1399 |