Real-time search is a hot area of search technology right now. Technology startups are jumping into the fray to build a variety of search technologies applications that address unique and unmet needs of consumer and business segments. John Greaves at Digital Media Buzz recently covered real-time search in a variety of verticals.
The key reasons why we are seeing proliferation of so many search apps in this arena are:
- Traditional search engines are based on an extensive infrastructure and framework for processing lots of content that is crawled, filtered and indexed over days and weeks if not months; and they don’t lend themselves very well when it comes to processing a fire hose of micro information generated by network updates.
- Consumption needs of consumers and businesses alike are changing. Consumers are using social feeds to research product and deals before they make purchase decisions and businesses are scouring real-time updates to gauge consumer pulse.
- The volume of information generated and hence, noise created is so huge that it is hard to make sense of it all without instant filtering.
- Recency is a major challenge because information generated in real-time can quickly become obsolete. A key component of recency is not just putting the most recent update that matched the query at the top but also to analyze the freshness and timeliness of the content. As an example, if you are looking for a deal on a digital camera and someone just tweeted an expired coupon for the camera – that tweet would ordinarily qualify for being shown at the top of search results. But its content is not recent. Analyzing content for such kind of filtering will require further processing that will decrease the freshness of information. This is a hard problem.
- Relevance and ranking algorithms for real-time content need to calibrate a completely new notion of data in the form of followers, reputation and popularity of the source – all constantly changing.
Real-time search, when applied to verticals such as online recruiting and job search, has its own interesting challenges as well as rewards. For job-related searches, real-time search offers a challenge to application developers around how to rank results: computing ranking based on recency and relevancy versus a filter of content based on a jobseeker’s qualification and interests. Add the potential of behavioral learning and there’s even more to consider. But you can do it all – with an algorithm tuned to account for peculiarities of a vertical.
For example, when it comes to recruiters looking for talent, status updates and tweets are a recruiter’s bonanza. Real-time information helps recruiters discover the hot candidate who just became available or might become available. In fact, recruiters have been using this real-time information ever since it became available in social network updates.
These vertical idiosyncrasies are hard for horizontal real-time search engines to capture. But this where an algorithm tuned for vertical search can interpret these specific data types and add value in recruiting software apps.
And just as we saw proliferation of vertical search engines in the traditional search arena, we’re seeing a similar proliferation in the real-time arena – enhancing the user experience in a variety of consumer and business apps.