Blog posts by Repo Journal2021-10-06T09:35:31.0000000Zhttps://world.optimizely.com/blogs/repo-journal/Optimizely WorldHow to exclude pages from your search index, sitemaps and internet search engines <p>Your client wants to exclude certain pages from showiing up in your site's search results, they also want prevent these pages from being crawled by external search engines (Google, Bing etc.) and lastly they want them removed from their sitemap.xml file. All of this can be achieved and controlled with a checkbox property on the Settings tab in the editor. I'll demonstrate below. </p>
<p>Create an interface with a boolean property that inherits IContent</p>
<pre class="language-csharp"><code> public interface IDisableIndexing : IContent
{
bool DisableIndex { get; set; }
}</code></pre>
<p>Add it to your base class or any page type </p>
<pre class="language-csharp"><code> public abstract class SitePageData : PageData, IDisableIndex
{
[CultureSpecific]
[Display(Name = "Disable Indexing",
Description = "Removes the page from search index, sitemap and search engines",
GroupName = SystemTabNames.Settings,
Order = 10)]
public virtual bool DisableIndex { get; set; }
}</code></pre>
<h3>Optimizely Search & Navigation</h3>
<p>You can filter from your Optimizely Search & Navigation search index by creating a module dependency class for your search conventions (this where IContent is implemented)</p>
<pre class="language-csharp"><code> [ModuleDependency(typeof(IndexingModule))]
public class FindConventionsInitialization : IInitializableModule
{
public void Initialize(InitializationEngine context)
{
var client = SearchClient.Instance;
ContentIndexer.Instance.Conventions.ForInstancesOf<IHasDisableIndex>().ShouldIndex(x => !x.DisableIndex);
}
public void Uninitialize(InitializationEngine context) { }
}</code></pre>
<h3>External Search Engines</h3>
<p>Use the same boolean property to add instructions for search robots from the <head></head> element in your layout view</p>
<pre class="language-csharp"><code><head>
@if (Model.DisableIndex)
{
<meta name="ROBOTS" content="noindex, nofollow" />
}
</head></code></pre>
<h3>Sitemaps</h3>
<p>If you're using the Geta Sitemap generator you can extend it and filter these pages from being added to when the xml file is being generated. Create a class that inherits from the abstract base <em>SitemapXmlGenerator </em>class and interface <em>ICommerceAndStandardSitemapXmlGenerator</em>. Override the <em>AddFilteredContentElement </em>method and from there you can exlude the pages with IDisableIndex. </p>
<pre class="language-csharp"><code> public class CommerceAndStandardSitemapXmlGenerator : SitemapXmlGenerator, ICommerceAndStandardSitemapXmlGenerator
{
public CommerceAndStandardSitemapXmlGenerator(
ISitemapRepository sitemapRepository,
IContentRepository contentRepository,
UrlResolver urlResolver,
ISiteDefinitionRepository siteDefinitionRepository,
ILanguageBranchRepository languageBranchRepository,
IContentFilter contentFilter)
: base(sitemapRepository, contentRepository, urlResolver, siteDefinitionRepository, languageBranchRepository, contentFilter)
{
}
//Filter content from xml sitemap
protected override void AddFilteredContentElement(CurrentLanguageContent languageContentInfo, IList<XElement> xmlElements)
{
var sitemapContent = languageContentInfo.Content as IHasDisableIndex;
if (sitemapContent != null && sitemapContent.DisableIndex)
{
return;
}
base.AddFilteredContentElement(languageContentInfo, xmlElements);
}
}</code></pre>2021-10-06T09:35:31.0000000ZBlog postOptimizing your Optimizely Search & Navigation service for large files<p>Awhile ago I had a client with an excess of large files. I had increased their upload size limit to 2 GB and many of their documents were between 50 MB; a dozen or so files were between 1 and 2 GB. Episerver recommends not exceeding the by default 50 MB maximum request size.</p>
<p>Not surprisingly the indexing job started timing out and required immediate attention. </p>
<p>I found there were several ways to tweak the the performance by filtering these files from the indexing job. </p>
<p>I created an initialization module and changed the default batch sizes for the Find service. ContentBatchSize is used for the find index job, MediaBatchSize is for the event-driven indexing on media types. </p>
<pre class="language-csharp"><code> [InitializableModule]
[ModuleDependency(typeof(IndexingModule))]
public class FileIndexingConventions : IInitializableModule
{
public void Initialize(InitializationEngine context)
{
ContentIndexer.Instance.MediaBatchSize = 3; // Default is 5
ContentIndexer.Instance.ContentBatchSize = 50; // Default is 100
}
public void Uninitialize(InitializationEngine context)
{
throw new NotImplementedException();
}
}</code></pre>
<p>I had several ways to filter out these large files. I could filter out IContentMedia from the index entirely or do the same with a custom type for pdfs and zip extensions.</p>
<pre class="language-csharp"><code>ContentIndexer.Instance.Conventions.ForInstancesOf<MyPdfMediaType>().ShouldIndex(x => false);</code></pre>
<p>Alternatively, I could stop the binary data from being indexed by decorating the propery with the <strong>[JsonIgnore] </strong>attribute:</p>
<pre class="language-csharp"><code> public class MyPdfMediaType : MediaData
{
[JsonIgnore]
public override Blob BinaryData { get; set; }
}</code></pre>
<p>But since the client wanted to have the file content searchable, I decided only to filter the property when the filesize reached the find service limit. </p>
<pre class="language-cpp"><code>ContentIndexer.Instance.Conventions.ForInstancesOf<IContentMedia>().IndexAttachment(x => !IsFileSizeLimitReached(x));</code></pre>
<p>...and for this I used an extention method to check against filesize binary data:</p>
<pre class="language-csharp"><code> private static bool IsFileSizeLimitReached(IBinaryStorable binaryContent)
{
// Note: 37 MB max. size refers to the base64 encoded file size .
const int limitKb = 37000;
try
{
var blobByte = (binaryContent.BinaryData as AzureBlob)?.ReadAllBytes() ??
(binaryContent.BinaryData as FileBlob)?.ReadAllBytes();
if (blobByte == null)
return false;
double fileSize = blobByte.Length;
var isLimitReached = (int)(fileSize / 1024) >= limitKb;
return isLimitReached;
}
catch
{
return false;
}
}</code></pre>
<p>Once in place I was able to run the job with no exceptions, no timeouts and a happy client!</p>2021-05-01T00:43:42.0000000ZBlog postRetrieving, storing file size information during upload<p>While presenting media content available for public download, it might be helpful to provide the file size, particularly for mobile users who may not want to save large files to their phone. </p>
<p>This can be performed on any IContentMedia type by extending the content model and using an initialization module to retrieve and save the file size information. </p>
<p>For this example, I'll create a custom media type for storing PDF, Word and Excel documents with a read only property to store the file size:</p>
<pre class="language-csharp"><code> [MediaDescriptor(ExtensionString = "pdf,docx,xlsx")]
public class DocumentMediaType : MediaData, IHasFileSize
{
[Editable(false)]
[Display(Order = 10, GroupName = SystemTabNames.Settings, Name = "File Size")]
public virtual string FileSize { get; set; }
}</code></pre>
<p>I'm implementing an interface I called IHasFileSize:</p>
<pre class="language-csharp"><code> public interface IHasFileSize
{
string FileSize { get; set; }
}</code></pre>
<p>After that I can create the initialization module and attach to the save and create events to store the file size data:</p>
<pre class="language-csharp"><code> [InitializableModule]
[ModuleDependency(typeof(ServiceContainerInitialization))]
public class FileBasedEventsInitialization : IInitializableModule
{
// attach to both the CreatingContent and SavingContent events
public void Initialize(InitializationEngine context)
{
IContentEvents eventRegistry = ServiceLocator.Current.GetInstance<IContentEvents>();
eventRegistry.CreatingContent += SavingMedia;
eventRegistry.SavingContent += SavingMedia;
}
private void SavingMedia(object sender, EPiServer.ContentEventArgs e)
{
if (!(e.Content is IContentMedia))
return;
IContentMedia media = e.Content as IContentMedia;
string fileSize = GetFileSizeDisplay(media);
if (media is IHasFileSize file)
{
file.FileSize = fileSize;
}
}
private string GetFileSizeDisplay(IContentMedia media)
{
if (media?.BinaryData != null)
{
using (var stream = media.BinaryData.OpenRead())
{
return FormatBytes(stream.Length);
}
}
return string.Empty;
}
public void Uninitialize(InitializationEngine context)
{
IContentEvents eventRegistry = ServiceLocator.Current.GetInstance<IContentEvents>();
eventRegistry.CreatingContent -= SavingMedia;
eventRegistry.SavingContent -= SavingMedia;
}
}</code></pre>
<p>This method is used for formatting the file size with the correct unit of measurement:</p>
<pre class="language-csharp"><code> private static string FormatBytes(long bytes)
{
string[] uom = { "B", "KB", "MB", "GB" };
int i;
double size = bytes;
for (i = 0; i < uom.Length && bytes >= 1024; i++, bytes /= 1024)
{
size = bytes / 1024.0;
}
return $"{size:0.##} {uom[i]}";
}
</code></pre>
<p>Thanks for reading, and enjoy displaying your new custom FileSize property!</p>
<p><img src="/link/296b33776eb942b088a6678b93b59974.aspx" width="396" height="451" /><br /><br /></p>2021-04-22T00:14:04.0000000ZBlog post