Thread by @KinglyCrow

I think a lot about search every day, at one level there's search at the scale of Google, but at the other end there's search as a feature of a product, like searching a catalog of things to buy, or looking for a movie, etc. #tweet100 #tech #business #search A 🧵 about search.

Today I sat down to write and the first thing that came to mind is how difficult it is for companies to implement search well in their products. 1/

How many times have you searched for a product on a site that you thought "surely I'll find what I'm looking for'', only to realize that they don't handle synonyms well or that you can only do an exact match to find the item you want. 2/

Why is implementing good search functionality so hard? And how can we make it easier for companies to include great search functionality as part of their product? Let's do a quick run down of the scene and see what options companies have to build their search right now. 3/

First, Lucene based search engines like Elastic and Solr have formed the bedrock of many companies' search functionality. 4/

Lucene is a highly tuned low level search engine and provides a breadth of functionality like highly optimized query composition and execution at the top of the search pipeline and intelligent result merging, re-ranking and post processing at the back. 5/

Search engines built on top of Lucene have robust transformation pipelines with every type of lemmatization and stemming you could want across most languages. 6/

More recently they have been adding things like continuous vectorization for NLP functionality such as vector similarity search. 7/

In the case of Elastic their paid and licensed offering gives you added functionality such as anomaly detection and operational benefits like cross cluster data replication for sensitive information. 8/

All of these together make a very comprehensive offering that in theory should allow a business to put together a compelling search experience. In practive its often more challenging than just having access to all of these great features. 9/

Similarly there are other open source offerings like Meilisearch which have gained in prominence in the space and are itching to compete with Elasticsearch. And while they may not match in complete functionality they are looking to make search more ergonomic. 10/

More on that later. Outside of do it entirely yourself offerings there are other incumbent players like Algolia where you index your data through their API and they offer a completely tunable search index all in their application. 11/

All you have to do is drop in their search widget, or point to a branded search page and you're off to the races after twisting some knobs to your satisfaction. 12/

What Algolia trades away in flexibility it makes up for in ease of use, but even with that benefit it's up to you to make use of their features and tune the search specifically to your use case. 13/

Not to mention relative to other offerings they charge an arm and a leg so it may not be suitable if you're a small business with a lot of search usage or a lot of data to index. 14/

Outside of the ones I've mentioned there are a myriad of other players either implemented as Wordpress plugins or Shopify plugins and a few more projects looking to capitalize on the relative newness of vector similarity based search, 15/

but by and large your three options are:
• host and build the search yourself
• build the search yourself and pay to avoid maintaining the clusters yourself
• or go entirely through someone else's API 16/

The fact that Algolia is still taking on investor money and that Meilisearch has grown in serious popularity as an ergonomic alternative to Elasticsearch tells me that the space hasn't truly been cracked and that it's still difficult to implement a good search experience. 17/

Here's the reason why. Search is a domain unto its own and it requires engineers with experience in indexing and tuning to provide the quality experience we have come to expect from an application. 18/

However, not only is it dependent on search specific skills it's also incredibly dependent on deep knowledge of the specific business domain you're implementing search for. 19/

This means engineers need to have worked in the space for a while or work very closely with a domain expert while implementing search functionality, in practice this rarely happens. 20/

What companies like Algolia and Elastic provide are incredibly compelling technical implementations of search and given that both of these businesses are valued at multiple billions of dollars clearly they are solving a real part of the problem. 21/

And they're able to cohabitate in the space largely because they operate on opposite ends of the axis of ease of use/time and flexibility/price.

Now, earlier on I briefly mentioned Meilisearch and how they're attempting to make search more ergonomic. 22/

What I meant by that is compared to Elasticsearch they're quicker to set up on a self hosted instance and you don't have to spend a lot of time tuning to get search up and going. 23/

To take care of needing prior search experience they have a range of preset configurations for the majority of common search cases such as e-commerce. This let's someone with domain experience but less search experience quickly put together a relatively good search solution. 24/

In totality they're less comprehensive than Elastic and not as set-up and forget as Algolia. 25/

I also don't think they go as far as I would like with that painless configuration feature, but I think with that feature specifically they're onto something that hasn't been tackled in the business search space in any real capacity and it's what I think a new solution in… 26/

…the space should focus on to provide something extraordinarily useful that will profoundly affect the bottom line of the businesses that need to implement a good search feature for their product. 27/

Now, It's obviously impossible to provide pre-configurations for every domain but you can create a solution where using your own deep knowledge of search you provide an experience where someone with the business domain knowledge can quickly configure a compelling search… 28/

…implementation without needing to understand the underlying coefficients they're affecting and having to rely on an engineer as an interlocutor. 29/

Ease of hosting and implementation are solved, but teams still need someone with deeper search experience to provide anything above a basic text search, it's time consuming, expensive and very error prone. 30/

Above that a bad search experience for a lot of businesses means a significant amount of lost revenue. 31/

Search is a high risk feature that delivers a significant amount of value when done correctly and I'm excited to see so many new entrants and interesting takes for the business search feature space in what was only a few years ago a relatively duopolistic space. 32/