Protiviti / SharePoint Blog

SharePoint Blog

August 04
Night Crawlers and Daytime Creepers: Keys to Search Freshness

Search, search, search – if you can’t find it you can’t use it! SharePoint 2013 offers a variety of crawl services designed to keep search results fresh and system resources highly available. What services are best for you? Night crawlers or daytime creepers, let’s take a look!


The Night Crawler


full crawl, AKA the night crawler, is intended to crawl all content that has been added since the last full crawl. Typically full crawls occur over a weekend and are set to run as overnight jobs or time periods when users aren’t very active such as 2:00 AM on a Sunday morning. The full crawl is your weekly search housekeeping service, refreshing content sources and getting the house in order for the week to come. Since it runs on off hours, the overall demands on system resources are lower.


Daytime creepers


For day-to-day content source freshness there are two services available – the incremental crawl or the continuous crawl. The catch is – you have to pick one. You can run a full crawl and an incremental crawl or a full crawl and a continuous crawl – but not a full crawl, incremental and continuous crawl.


An incremental crawl is intended to pick up new content that has been added since the last crawl. Incremental crawls run on a specific schedule that you select and then run again on your predefined schedule. The catch is, if new content is added during the incremental crawl, that new content doesn’t get added until the next time the incremental crawl runs. If your content is "late" to the crawl, it has to wait until the next one.


The continuous crawl aims to keep the search index as up to date as possible. Based upon an interval you set, the continuous crawl will pick up new content. A continuous crawl crawls content that was added, changed, or deleted since the last crawl. The default interval is every 15 minutes, but you can set continuous crawls to occur at shorter intervals. Because continuous crawls occur so often, they help ensure search-index freshness, especially for SharePoint content that is frequently updated. Continuous crawls are useful for content that is quickly changing and requires high availability.


Which service is right for you?


If your user community demands highly available content, a continuous crawl is a good tool to employ because based on your set interval, users will be able to discover desired search results. If users tell you that they have created new items or added documents to SharePoint, but they can’t nearly immediately find them in search results, your SharePoint site collection may be a better candidate for the continuous crawl over an incremental crawl. This will help prevent users from creating duplicate content as when they can’t find it in search, they assume it is lost or didn’t save correctly.If your user community demands less highly available content, an incremental crawl may do the job.

Crawl services that run during the day also compete with system resources for regular user demand, search execution, and other services. It’s important that your crawl services don’t slow your user experience "to a crawl," so keep an eye on your system performance as you determine the best crawl service to meet your user community needs.


About Nora Ten Broeck: Nora is a SharePoint enthusiast and expert-in-the-making with interests in collaboration, project management and improved business processes. Follow her @NoraTenBroeck.


Quick Launch

© Protiviti 2020. All rights reserved.   |   Privacy Policy