Search & Navigation - Click Tracking

If you are implementing Search & Navigation (a.k.a. FIND) but cannot use the ‘Unified Search’, you will need to implement your own server-side tracking. There’s a number of blogs out there that touch on the subject but many are outdated and none seem to capture 100% of the requirements.

If you don’t get everything correct, it’s likely you’ll be seeing all zeros for ‘Click-through rate’ in the CMS:

The following will help anyone that needs to cater for:

• Multisite setups

• Accurate reporting on ‘Most frequent searches’

• Accurate reporting on ‘Searches without hits’

• Accurate reporting on ‘Searches without relevant hits’

• Accurate reporting ‘People who searched for 'foo' also searched for…’

Note, I understand that Search & Navigation is kinda' on the way out (in favor of Graph) but hopefully this will still be of some use.

Overview

This approach:

1. Captures and submits the search query

2. Fires a ‘track event' for the query

3. Appends the tracking info (from the track event) to each search result URL

4. When a search result is clicked, this fires an event to a tracking route – /go .

5. Tracking info is read from the query parameters

6. Tracking event is fired for the click event passing the query parameters

7. The request is redirected to the search result page

Get the Search Result

var searchResult = await ServiceLocator.Current.GetInstance<IClient>()
            .Search<T>(“en”)
            .For(“foo”)    
            .Take(20)
            .GetResultAsync();

Track the Query

var tags = _tagsHelper.GetTags(false).ToList();
var trackResult = await ServiceLocator.Current.GetInstance<IClient>().Statistics().TrackQueryAsync(query.Query.ToLower(), c =>
                {
                    c.Id = new TrackContext().Id;
                    c.Query.Hits = searchResult.TotalMatching;
                    // c.Tags = _tagsHelper.GetTags(false).ToList();   -- don't do this here!
                    c.Tags = tags;
                }));

Things to note:

• tags will capture the current site, language and categories

• Do not evaluate the tags within the command action of TrackQueryAsync. This is a problem in multi-site instances. Somehow, when you evaluate the tags from within the command action, the site id is always resolved to the website with the wildcard domain. Consider this if you have multiple sites sharing the same search code.

• c.Id = new TrackContext().Id; – this is used to set an id for the user – it’s needed to determine what similar users also searched for and what they clicked on – this can be exposed via _searchClient.Statistics()?.GetDidYouMeanAsync()

Add Tracking Info to the Result URLs

var trackedUrls = new List<string>();
var resultList = searchResult.Hits.ToList();
for (int x = 0; x < resultList.Count; x ++)
{
    var trackedUrl = $"https://www.mysite.com/" +
                     $"?query={System.Web.HttpUtility.UrlEncode(query)}" +
                     $"&trackid={trackResult.TrackId}" +
                     $"&hitid={resultList[x].Id}" +
                     $"&hittype={resultList[x].Type}" +
                     $"&trackuuid={trackResult.TrackUUId}" +
                     $"&trackhitpos={x +1}" +
                     $"&page="{resultList[x].Document.LinkURL};
    trackedUrls.Add(trackedUrl);   
}

Example:

https://www.my-site.com/search/go/?query=my+query&trackid=yN8AqDDTSJEuof_p2uWVmg==&hitid=_507f0cdd-71b6-41ac-80ee-88bd90ddbac1_en&hittype=MySite_Pages_StandardPage&trackuuid=anbJ4Es_RGS_Kab0PANDeA&trackhitpos=5&page=/blog/article-one

Things to Note:

• URL Encode the search term when adding to the query parameters

Track Clicks / Hits

  [HttpGet]
  [Route("go")]
  public async Task<RedirectResult> Go(
      [FromQuery] string query,
      [FromQuery] string trackid,
      [FromQuery] string hitid,
      [FromQuery] string hittype,
      [FromQuery] string trackuuid,
      [FromQuery] string trackhitpos,
      [FromQuery] string page)
  {
      Task.Run(async () =>
      {
          var locator = ServiceLocator.Current;
          var hitIdFormatted = $"{hittype}/{hitid}";
          var tags = locator.GetInstance<IStatisticTagsHelper>().GetTags(false).ToList();
      
          await locator.GetInstance<IClient>().Statistics().TrackHitAsync(
              queryString: query,
              hitId: hitIdFormatted,
              command =>
              {
                  command.Id = trackid;
                  command.Hit.Id = hitIdFormatted;
                  command.Tags = tags;
                  command.Hit.QueryString = System.Web.HttpUtility.UrlDecode(query ?? string.Empty);
                  command.Hit.Position = int.TryParse(trackhitpos ?? "0", out var pos) ? pos : 0;
                  command.AdditionalParameters = new AttributeDictionary() { { Uuid, trackuuid } };
              });
      });
    
      return this.Redirect(page);
  }

Things to note:

• By utilizing ‘Task.Run’ here we are implementing a ‘fire-and-forget' strategy; we do not wait for the tracking to succeed before redirecting the user – this improves the performance / user-experience. Other/better ways to tackle this issue could be via a background service or queue. Some error handling and logging would also be a good idea.

With all this in place, you should now see the correct statistics starting to roll-in.

Gotchas:

• Trim and lowercase the search term to prevent tracking different queries and clicks for effectively the same thing

• Chromium-based browsers may prerender links on your page via the Speculation Rules API. This may lead to click tracking being fired even if the user does not click a result – suggest you disable this on your search results page.

• Bots may also crawl your page and register searches without clicks. One way around this could be to only track requests where the referrer domain is your own site.

• Users who click ‘back’ via the browser will trigger a second query, and skew the result – to get around this you can cache the result for the query based on a unique key for the request and user -
  var cacheKey = new TrackContext().Id + Request.QueryString + Request.Host;