AdvancedComing soon

Build Elasticsearch From Scratch

MiniSearch, in Java, with no libraries

Build a single-node search engine that works like Elasticsearch: text analysis, an inverted index, BM25 ranking, a JSON query DSL over HTTP, durable segments on disk and aggregations. Java 17 and nothing but the JDK, with a runnable milestone at the end of every episode.

39 episodes planned · Java

This course is being made. Here is the full plan: what each episode will teach and what you will have built by the end of it.

Module 0 · Orientation

2 episodes
  1. 01 What Elasticsearch really is Documents, indices, the inverted index, the REST API; a tour of the finished MiniSearch Build: project skeleton, build script planned
  2. 02 Why LIKE '%word%' is slow A full scan is O(total text); an index jumps straight to the matches Build: NaiveSearch + a benchmark on 200k documents planned

Module 1 · Text analysis

5 episodes
  1. 03 Tokenization Splitting text into terms; punctuation, digits, Unicode letters Build: StandardTokenizer planned
  2. 04 Normalization Lowercasing, accent folding (java.text.Normalizer), stop words Build: LowercaseFilter, AsciiFoldingFilter, StopFilter planned
  3. 05 Stemming "running", "runs", "ran" → "run"; a simplified Porter stemmer; over- and under-stemming Build: PorterStemFilter planned
  4. 06 The analyzer pipeline Char filters → tokenizer → token filters; the standard, simple, whitespace and keyword analyzers Build: Analyzer interface + _analyze demo planned
  5. 07 Review quiz: analysis Build: questions.json + interactive page planned

Module 2 · The inverted index

6 episodes
  1. 08 Your first inverted index Term → sorted list of doc ids, built with a Map Build: InvertedIndex planned
  2. 09 Frequencies and positions A posting stores how often, and where, a term appears Build: Posting, PostingsList planned
  3. 10 A JSON parser in 200 lines Recursive descent: objects, arrays, strings, numbers Build: Json.parse / Json.write planned
  4. 11 Documents and the doc store _id vs. internal doc id, _source, versioning Build: DocStore, Index.index(doc) planned
  5. 12 Mappings and field types Text vs. keyword vs. long / double / date / boolean; dynamic mapping Build: Mapping, per-field indexes planned
  6. 13 Review quiz: the index planned

Module 3 · Queries

7 episodes
  1. 14 Term and Boolean queries AND = intersect sorted lists, OR = union, NOT = difference, with two pointers Build: Query, TermQuery, BooleanQuery planned
  2. 15 Skip pointers Jump ahead in long postings lists to intersect faster Build: skip lists in PostingsList planned
  3. 16 Phrase queries Use positions to match "quick brown fox" exactly Build: PhraseQuery planned
  4. 17 Prefix and wildcard A sorted term dictionary, binary search, TreeMap.subMap Build: PrefixQuery, WildcardQuery planned
  5. 18 Fuzzy search Levenshtein edit distance with dynamic programming; pruning the term dictionary Build: FuzzyQuery planned
  6. 19 Range queries Numbers and dates can't use the text index; a sorted numeric index Build: RangeQuery planned
  7. 20 Review quiz: queries planned

Module 4 · Relevance and ranking

4 episodes
  1. 21 TF-IDF Why rare words matter more, and long documents shouldn't win by default Build: TfIdfSimilarity planned
  2. 22 BM25 The formula Elasticsearch uses: term saturation (k1) and length normalization (b) Build: BM25Similarity planned
  3. 23 Top-K with a heap Return the best 10 of a million without sorting, using a PriorityQueue Build: TopKCollector planned
  4. 24 Boosts, multi-field search and explain Weighting fields, match / multi_match, showing why a document scored what it did Build: explain output planned

Module 5 · The API

4 episodes
  1. 25 The query DSL Parse Elasticsearch-style JSON (bool, match, term, range) into a query tree Build: QueryParser planned
  2. 26 A REST server PUT /books, PUT /books/_doc/1, GET /books/_search with the JDK's HttpServer Build: RestServer; curl works planned
  3. 27 Bulk indexing and pagination _bulk NDJSON; from / size, why deep paging hurts, search_after Build: _bulk, search_after planned
  4. 28 Review quiz: ranking and the API planned

Module 6 · Storage and durability

7 episodes
  1. 29 Segments Write the index to disk once and never modify it; a file format with DataOutputStream Build: SegmentWriter, SegmentReader planned
  2. 30 Refresh vs. flush The in-memory buffer, and why search is "near real-time" Build: refresh(), flush() planned
  3. 31 The translog A write-ahead log so a crash loses nothing; recovery on startup Build: Translog + a kill-the-process demo planned
  4. 32 Deletes and updates Segments can't change, so we mark "live docs" in a BitSet; an update is a delete plus an add Build: live-docs bitset planned
  5. 33 Segment merging Why segment counts grow, a tiered merge policy, reclaiming deleted docs Build: MergePolicy planned
  6. 34 Compression Delta encoding and variable-length ints (VInt) for postings; measuring the size saved Build: VIntCodec planned
  7. 35 Review quiz: storage planned

Module 7 · Aggregations, and the finale

4 episodes
  1. 36 Doc values Aggregations need a column per field, not an inverted index Build: DocValues planned
  2. 37 Aggregations Terms, histogram, stats, and aggregations over a query's results Build: aggs in _search planned
  3. 38 Highlighting and autocomplete Marking matched terms in _source; an edge-n-gram "search as you type" Build: highlight, completion planned
  4. 39 MiniSearch vs. Elasticsearch A benchmark on the same data, and what we left out (finite-state transducers, distribution, vectors) Build: benchmark report planned