Despite increased centralization over the past 20 years, the internet feels a lot more like the wild west to me in man ways, e.g., the Google index hasn't kept up with the size of the internet, so an increasingly large fraction of the web is undiscoverable via search.
Even 10 years ago, I could basically always find old blog posts I'd read with Google.
Now, an exact string match search with site:[URL] frequently doesn't turn up the result and I have to wget the page and grep for what I'm looking for.
Now, an exact string match search with site:[URL] frequently doesn't turn up the result and I have to wget the page and grep for what I'm looking for.
If the site's too large to wget and it doesn't have a custom index, I frequently can't find the page. Large commercial sites, like Twitter, sometimes build complete indexes, but it's a non-trivial effort to index something that's even 1/100th the size of Twitter, so most don't.
Relatedly, I wrote
Within a year, new comments on that post were "that's a dumb example, everyone knows search is hard"
https://twitter.com/danluu/status/782974159156502534because a bunch of people were talking about how someone should build a better Google, seemingly not realizing that's a difficult task.
Within a year, new comments on that post were "that's a dumb example, everyone knows search is hard"
But now there's another wave of discussions about how someone should build a better search engine.
I'm not saying that can't be done, but I haven't yet seen one discussion that addresses any of the really hard parts. The discussions are focused about how idea X could work.
I'm not saying that can't be done, but I haven't yet seen one discussion that addresses any of the really hard parts. The discussions are focused about how idea X could work.
Sure, maybe X can work, but to implement X, you still need to deal with problem mentioned above (the internet is huge and even if you wan't to index almost any of it, you still need to crawl enough to find what you want and discard the rest, and if you're competing in an area
where Google & Bing have recent results, your results will seem old if you can't update your index within minutes of a new page being created).
Unless you're extraordinarily good at picking and marketing to a small niche, there are ~5 problems like that which are table stakes.
Unless you're extraordinarily good at picking and marketing to a small niche, there are ~5 problems like that which are table stakes.
Figuring out which niche avoids these problems or actually solving these problems is the hard part, not having some clever idea about how to make search better.
Just throwing various clever ideas around feels very "cocktail party idea" to me.
Just throwing various clever ideas around feels very "cocktail party idea" to me.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
