There many reasons why your HTML cache might not clear, but this post deals with the following specific scenario:
- You have event queues enabled.
- Your publish:end and publish:end:remote event handlers both include the Sitecore.Publishing.HtmlCacheClearer class, configured with appropriate sites.
- The first time you publish during the application's lifetime, the HTML cache clears as expected.
- Subsequent publishes do not clear the cache, and the HtmlCacheClearer does not produce any logs.
This is weird. After wrestling with the problem for hours we were completely stumped. Luckily Sitecore support we're on hand to help us out. The solution turned out to be completely unrelated to the HTML cache or the event queues.
Sitecore maintains a legacy Lucene index called __System. In our case that index was locked so it couldn't be updated. That somehow prevented the clearance of the HTML cache. The answer was to simply delete it!
It turns out that the __System index can be removed without any problems - Sitecore just recreates it afterwards. In fact, it doesn't really need it at all, as evidenced by the following setting:
<setting name="Search.UseLegacySearchEngine" value="false" />
If you like, you can even remove (or delete patch) the __System index definition from your Sitecore instance and it shouldn't complain at all,