[Feature] UMAP with various clustering algorithms

[HahaBill]

[HahaBill]

This issue is being tackled here: #14

[Harsh16gupta]

These are all the test results. After some more experimentation, I will update my findings

Notes categorised by HBDSCAN--5-ms2

[ai-categorise] Best: hdbscan-5-ms2 (silhouette = 0.7504)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise] ===================================
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise] 
Cluster assignments (hdbscan-5-ms2):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 0 (11 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 2. Importing and exporting notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 1. Welcome to Joplin!
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - GSoC 2026: Joplin Note Categorization Plugin
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Attention mechanism — intuition
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - CLI tool idea — "note linker"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Embeddings — practical notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Loss functions — quick reference
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Transformer architecture — notes from Karpathy video
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Running a k-means experiment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline — Complete Report
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 1 (10 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: Detailed Architectural Roadmap (Path B & Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: 14-Day Day-by-Day Implementation Plan (Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise Product Release Plan (V1 to V3)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Budget spreadsheet v2 ideas
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Tax saving checklist (India)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Passive income ideas to actually research
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - "if it compiles it ships" — heard this from someone at a conf, need to find the 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Feature wishlist — personal finance app
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - +91 983888445 — Rohan, met at PyCon, does ML infra at a startup in Pune, follow 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Compound interest calculator — build this
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 2 (10 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Array deduplication in JS
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Atomic Habits — main takeaways
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - async/await error handling pattern
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - TypeScript discriminated unions
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Meditations by Marcus Aurelius
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Debounce vs throttle
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Design of Everyday Things — notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Remains of the Day — Ishiguro
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - useReducer over useState when...
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Poor Charlie's Almanack — mental models
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Noise/Outliers (12 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Butter chicken — weeknight version
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Lisbon trip notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 50/30/20 rule isn't for everyone
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Morning run + mobility routine
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Solo Japan itinerary sketch
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Shampoo, razor heads, floss, olive oil, tahini, lemons
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - App idea — "Streak social"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Prague + Vienna long weekend
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Protein sources — vegetarian tracker
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Schengen visa tips
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Fine-tuning vs RAG
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Mediterranean pasta — quick
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 3 (5 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Overnight oats — base recipe
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Sleep hygiene — what actually helps
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Green smoothie that actually tastes good
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - HIIT workout — 20 min no equipment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Hydration notes
plugin_com.harsh16gupta.notecategorization.js:2 

Notes categorised by HBDSCAN-3

[ai-categorise] 
Cluster assignments (hdbscan-3):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 0 (11 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 2. Importing and exporting notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 1. Welcome to Joplin!
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - GSoC 2026: Joplin Note Categorization Plugin
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Attention mechanism — intuition
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - CLI tool idea — "note linker"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Embeddings — practical notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Loss functions — quick reference
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Transformer architecture — notes from Karpathy video
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Running a k-means experiment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline — Complete Report
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 1 (5 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: Detailed Architectural Roadmap (Path B & Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Budget spreadsheet v2 ideas
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Passive income ideas to actually research
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - "if it compiles it ships" — heard this from someone at a conf, need to find the 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Compound interest calculator — build this
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Noise/Outliers (14 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: 14-Day Day-by-Day Implementation Plan (Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise Product Release Plan (V1 to V3)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Morning run + mobility routine
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Atomic Habits — main takeaways
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - async/await error handling pattern
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Solo Japan itinerary sketch
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Tax saving checklist (India)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - App idea — "Streak social"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - HIIT workout — 20 min no equipment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Feature wishlist — personal finance app
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Schengen visa tips
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Fine-tuning vs RAG
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Remains of the Day — Ishiguro
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - +91 983888445 — Rohan, met at PyCon, does ML infra at a startup in Pune, follow 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 3 (4 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Array deduplication in JS
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - TypeScript discriminated unions
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Debounce vs throttle
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - useReducer over useState when...
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 5 (8 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Butter chicken — weeknight version
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Overnight oats — base recipe
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Sleep hygiene — what actually helps
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Shampoo, razor heads, floss, olive oil, tahini, lemons
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Green smoothie that actually tastes good
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Protein sources — vegetarian tracker
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Mediterranean pasta — quick
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Hydration notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 2 (3 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Lisbon trip notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 50/30/20 rule isn't for everyone
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Prague + Vienna long weekend
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 4 (3 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Meditations by Marcus Aurelius
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Design of Everyday Things — notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Poor Charlie's Almanack — mental models

Note categorised by Kmedoids-5

plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise] 
Cluster assignments (kmedoids-5):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 0 (12 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 2. Importing and exporting notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 1. Welcome to Joplin!
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - GSoC 2026: Joplin Note Categorization Plugin
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Attention mechanism — intuition
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - CLI tool idea — "note linker"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Embeddings — practical notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Loss functions — quick reference
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Transformer architecture — notes from Karpathy video
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Fine-tuning vs RAG
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Running a k-means experiment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline — Complete Report
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 2 (10 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: Detailed Architectural Roadmap (Path B & Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: 14-Day Day-by-Day Implementation Plan (Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise Product Release Plan (V1 to V3)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Budget spreadsheet v2 ideas
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Tax saving checklist (India)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Passive income ideas to actually research
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - "if it compiles it ships" — heard this from someone at a conf, need to find the 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Feature wishlist — personal finance app
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - +91 983888445 — Rohan, met at PyCon, does ML infra at a startup in Pune, follow 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Compound interest calculator — build this
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 3 (11 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Array deduplication in JS
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Atomic Habits — main takeaways
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - async/await error handling pattern
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - TypeScript discriminated unions
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Meditations by Marcus Aurelius
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - App idea — "Streak social"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Debounce vs throttle
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Design of Everyday Things — notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Remains of the Day — Ishiguro
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - useReducer over useState when...
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Poor Charlie's Almanack — mental models
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 1 (10 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Butter chicken — weeknight version
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Morning run + mobility routine
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Overnight oats — base recipe
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Sleep hygiene — what actually helps
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Shampoo, razor heads, floss, olive oil, tahini, lemons
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Green smoothie that actually tastes good
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - HIIT workout — 20 min no equipment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Protein sources — vegetarian tracker
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Mediterranean pasta — quick
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Hydration notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 4 (5 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Lisbon trip notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 50/30/20 rule isn't for everyone
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Solo Japan itinerary sketch
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Prague + Vienna long weekend
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Schengen visa tips

Note categorised by hdbscan-3-ms2

plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise] 
Cluster assignments (hdbscan-3-ms2):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 0 (11 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 2. Importing and exporting notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 1. Welcome to Joplin!
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - GSoC 2026: Joplin Note Categorization Plugin
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Attention mechanism — intuition
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - CLI tool idea — "note linker"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Embeddings — practical notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Loss functions — quick reference
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Transformer architecture — notes from Karpathy video
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Running a k-means experiment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - UMAP Projection & Clustering Benchmark Pipeline — Complete Report
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 1 (10 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: Detailed Architectural Roadmap (Path B & Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise: 14-Day Day-by-Day Implementation Plan (Path C)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - NivWise Product Release Plan (V1 to V3)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Budget spreadsheet v2 ideas
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Tax saving checklist (India)
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Passive income ideas to actually research
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - "if it compiles it ships" — heard this from someone at a conf, need to find the 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Feature wishlist — personal finance app
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - +91 983888445 — Rohan, met at PyCon, does ML infra at a startup in Pune, follow 
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Compound interest calculator — build this
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 3 (5 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Array deduplication in JS
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - async/await error handling pattern
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - TypeScript discriminated unions
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Debounce vs throttle
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - useReducer over useState when...
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Noise/Outliers (11 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Butter chicken — weeknight version
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - 50/30/20 rule isn't for everyone
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Morning run + mobility routine
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Atomic Habits — main takeaways
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Solo Japan itinerary sketch
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Shampoo, razor heads, floss, olive oil, tahini, lemons
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - App idea — "Streak social"
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Protein sources — vegetarian tracker
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Fine-tuning vs RAG
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Remains of the Day — Ishiguro
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Mediterranean pasta — quick
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 2 (3 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Lisbon trip notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Prague + Vienna long weekend
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Schengen visa tips
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 5 (5 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Overnight oats — base recipe
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Sleep hygiene — what actually helps
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Green smoothie that actually tastes good
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - HIIT workout — 20 min no equipment
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Hydration notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]   Cluster 4 (3 notes):
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Meditations by Marcus Aurelius
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - The Design of Everyday Things — notes
plugin_com.harsh16gupta.notecategorization.js:2 [ai-categorise]     - Poor Charlie's Almanack — mental models

[Harsh16gupta]

Hey Bill, based on the result, I am thinking of removing xmeans-auto and kmeans-8 completely because they form very bad clusters.
And also, I was thinking that we can give users multiple (2-3) clustering options to use. Like I have already implemented them and let's keep them for testing as the corpus grows.
By any chance if it becomes too confusing, then we can easily remove it in the coming weeks. wdyt?

[HahaBill]

Hey Bill, based on the result, I am thinking of removing xmeans-auto and kmeans-8 completely because they form very bad clusters. And also, I was thinking that we can give users multiple (2-3) clustering options to use. Like I have already implemented them and let's keep them for testing as the corpus grows. By any chance if it becomes too confusing, then we can easily remove it in the coming weeks. wdyt?

Makes sense for me to remove the approaches that are underperforming. You can always add them back from reverting the commit id.

Regarding adding more clustering options: I think it's good idea to do it for R&D purposes. For the user experience, we can circle back to that later since it's a premature optimization problem. We should prioritize finishing the pipeline with testing and observing the results.

Let me know when you're ready for the PR to be reviewed.

[Harsh16gupta]

I have completely removed xmeans-auto and kmeans-8 from the code.
I kept K-Means, K-Medoids, and HDBSCAN in the benchmark config for R&D testing so we can observe how they perform as we test the pipeline. I also did some refactoring for some duplicate code.

The PR is ready for review when you have time.

[HahaBill]

@Harsh16gupta Could you maybe add some visualizations for us to understand more? You could get the outputs and then use scientific Python libraries to visualize them and do experiments.

[Harsh16gupta]

@Harsh16gupta Could you maybe add some visualizations for us to understand more? You could get the outputs and then use scientific Python libraries to visualize them and do experiments.

I can definitely do that. I just have one question: do you want this visualization just for our own understanding, or is it for the users as well? I am asking this because if we want it just for our own testing, we can use Python libraries (which we will remove later), but if we want to build it in the plugin for users as well, I will need to find a JS alternative.

[HahaBill]

@Harsh16gupta Could you maybe add some visualizations for us to understand more? You could get the outputs and then use scientific Python libraries to visualize them and do experiments.

I can definitely do that. I just have one question: do you want this visualization just for our own understanding, or is it for the users as well? I am asking this because if we want it just for our own testing, we can use Python libraries (which we will remove later), but if we want to build it in the plugin for users as well, I will need to find a JS alternative.

Just for our own understanding for now

[Harsh16gupta]

Here are the visualization results of running the different strategies on the note collection, projected to 2D using UMAP: