Search over your own data #

Raw markdown: /search.md

For humans #

Interlock Search indexes your application's own records and answers queries inside your host's JVM. A search is a method call over memory-mapped files: there is no cluster to run, no network hop on the query path, and no copy of your data anywhere else.

It also understands what your things are. A synonym graph — an ontology you edit, version and publish like code — lets a search for fruit find a product called green banana, without that product ever containing the word.

Do this #

One class describes your data. Two methods are required.

public final class TaskSearchConfig implements InterlockSearchConfig<Task> {

    public String index() { return "tasks"; }

    public SearchSchema schema() {
        return SearchSchema.builder()
            .primaryLocale("en")
            .field(FieldSpec.text("title").perLocale().withConcepts().boost(4))
            .field(FieldSpec.keyword("state").faceted())
            .build();
    }

    public SearchDocument document(Task task) {
        return SearchDocument.builder(String.valueOf(task.id))
            .text("title", "en", task.title)
            .keyword("state", task.done ? "done" : "open")
            .updatedAt(task.updatedAt)
            .build();
    }
}

One line wires it:

Engine engine = InterlockSDK.init(key)
        .search(new TaskSearchConfig())
        .build();

The host writes documents when its own data changes, and searches whenever it likes:

TypedIndex<Task> tasks = engine.searchFor(new TaskSearchConfig());
tasks.put(task);
SearchResult result = tasks.search("green banana");

A unit searches through the capability gateway:

const result = il.search().index('tasks')
      .query('bananas').filter('state', 'open').limit(20).run();

What comes back #

record SearchResult(List<Hit> hits, List<Tier> tiers, long totalEstimate,
                    Status status, String truncation, List<String> degradedReasons)

status is the part worth reading. OK is a complete answer. DEGRADED is a real answer with something wrong — read degradedReasons. UNAVAILABLE means the engine could not look at all.

An empty list never means "unavailable". "Nothing matched" and "I could not look" are different facts, and a search that collapses them teaches its caller to trust an empty result it should not. Check status before you conclude anything from hits.

Hits arrive in bands. perfect matched every concept in the query with nothing left over; regular is an ordinary match; related is a deliberate generalization, offered only when the exact answer was thin and always labelled so a reader can tell the difference.

Fields #

perLocale() stores one value per locale and searches the locales a request asks for. withConcepts() feeds the field to the synonym graph. boost(n) weights a field: a title usually matters more than a description.

The synonym graph #

A graph is a set of concepts, the concepts each one is a kind of, and the concepts each one must not be confused with. Attach a published graph and searches start understanding your domain:

engine.attachGraph(graph);

Two kinds of exclusion, and the difference matters. An exclusive rule always excludes: a search for biscuit must never return dog-food biscuits. An inclusive rule excludes unless the declaring concept is present too: a search for a diet must not return ordinary bread, but bread made for that diet is exactly what was wanted.

Rules are inherited. State one on a category and it holds for everything under it, including the things added next year. Asking for something by name always beats somebody else's rule about it.

Changing a graph queues work. Documents carry what they are kinds of, so a published change leaves some of them stale. The index finds them itself and reports the backlog:

int pending = engine.attachGraph(newGraph);
engine.drainGraphRewrites(500);      // chunked; returns what remains

While that backlog is non-empty, results say DEGRADED with graph-rewrite-pending. That is on purpose: a publish that is still landing is a fact worth admitting.

Repairing an index #

An index is derived data and can always be thrown away. Give your config a source() and you get the repair paths for free:

engine.rebuildAll();          // read everything again
engine.reconcileAll();        // fix only what drifted

A write never throws — an index that can fail a user's save would be worse than no search. A failed write is remembered instead, shows up in status(), and is repaired by the next reconcile.

Customizing #

Two seams, for the things a general engine cannot know about your data.

DocumentWriter decorates the write side: add a field the mapping alone cannot produce. FieldEnricher is the packaged case — a value computed off the write path (a generated summary, say), stored with the hash of the text it was computed from, and treated as absent the moment that text changes. A summary of content that no longer exists is worse than no summary.

CandidateSearcher decorates the read side: refine what was retrieved. SearchPass is the packaged case — boost an exact occurrence, require one, or drop candidates that fail a stricter test. A pass refines what was retrieved and cannot add to it, so anything that must be findable has to be reachable by the query first.

Explaining a result #

SearchRequest.builder().query("fruit").explain(true).build();

Each hit then carries what matched, through which word, and what it was taken to be a kind of.

The harder question has its own call:

index.explainAbsent(docId, request);

It answers why a document is not in the results — outranked, filtered, or excluded by a rule — because a document at rank four thousand looks exactly like one that was thrown out.

Common words #

Articles and prepositions are handled per language, and never deleted from your index. Deleting them is the usual trick and it quietly loses data: a band called The Who becomes unfindable, and no amount of query tuning brings it back, because the words are no longer there.

Instead they are discounted when you search. Asking for "the best coffee maker" is not narrowed by "the", and a document whose whole title is common words is still found by them.

Fourteen languages ship in the box (en, es, pt, fr, de, it, nl, sv, pl, ru, tr, ja, zh, ko). To add one, or to replace ours, put a properties file on your classpath before the SDK's:

# search-stopwords/da.properties
stopwords=og,i,jeg,det,at,en,den,til,er,som,pa,de,med,han,af

A language with no file gets an empty list: slightly less precision on long queries, never an error.

What this is not #

Single-JVM indexes. A write on one node does not update another node's index; the reconcile pass is the repair story. No sharding, no replication, no log analytics, no aggregation language. If you need a search cluster, use a search cluster. This is for applications that want good search over their own data without operating one.

For agents #

• One capability, always il.search(). There is no per-domain facade; what an index is for is configuration.
• Check status before reading hits. UNAVAILABLE and DEGRADED are answers.
• document() must be deterministic and cheap. Anything slow or model-backed belongs in a FieldEnricher, which runs off the write path.
• Never index a value that changes faster than the corpus — prices, stock levels, anything per-place. Index identity; join the volatile part after retrieval.
• Field names are lower camel case. Names starting _ are the engine's and are refused.
• A schema change re-versions every document; the reconcile pass rewrites them.
• Compile a graph with SearchEngine.graphTextAnalyzer(). Any other normalizer silently stops matching.