Today the question was raised, if TarMK running on NAS is a good idea. The short answer is: „No, it’s not a good idea“.

The long answer: The TarMK relies on the ability of the operating system to map files into memory (using the so-called memory-mapped technology, short: mmap; see this wikipedia page on it). Oak does this for the heavily used parts of the TarMK to increase performance, because then these parts don’t need to be read from filesystem again, but are rather always available in memory (which is by at least an order of magnitude faster). This works well with a local filesystem, where the operating system knows about every change happening on the filesystem, because it is the only one through which access to this filesystem happens, and it can make sure, that the content of the file on disk and in memory are in sync. I should also mention, that this memory isn’t part of the heap of the JVM, but rather the free RAM of the system is used for this purpose.

With a NAS the situation is different. A NAS is designed to be accessed by multiple systems in parallel without the need to synchronize between each other. The 2 most common filesystems for this are NFS and SMB/CIFS. On NFS one system can open a file and is not aware that a second system modifies in the same time. This is a design decision which prevents that a system can keep the content of a file on NFS and in-memory in sync. Thus mmap is not usable when you use a NAS to store your TarMK files.

And because mmap is not usable, you’ll get a huge performance impact compared to a local filesystem where mmap can be used. And then I haven’t even mentioned the limited bandwidth and higher latency of a remote storage compared to local storage.

If you migrate from CRX 2.x (till AEM 5.6.1) this problem was not as visible as it is now with Oak, because there was the BundleCache, which cached data already read from disk; this bundle cache is an in-memory, in-heap structure and you had to adjust the heap size for it. CRX 2.x did not use mmap.

But Oak does not have this in-memory cache any more, but relies on the mmap() feature of the operating system to keep the often-accessed parts of the filesystem (the TarMK) in memory. And that’s the reason why should leverage mmap as much as possible and therefor avoid a NAS for TarMK.

Servicepack 3 for AEM 6.0 is now available (releasenotes).

Here’s the complete list of sling bundles in stock AEM 6.0 and the various levels of servicepacks. Bundles which are not available with a specific version are listed as “-“, version numbers marked in red appeared first in this servicepack. Where possible I added the links to the changes in servicepack 3.

The most notable change for SP3 from a Sling perspective is the switch to the fsclassloader (see 6D’s blogpost for it) for all scripting languages. So the compiled JSPs do not longer reside inside the repository (/var/classes), but now are placed in the filesystem.

In the last articles (1,2) I covered some aspects of multi-tenancy, which are very likely to occur in AEM projects (but not restricted to such projects). I stressed that there a lot of aspects which have the potential to cause trouble on a non-technical level. But you cannot draw a clear line between the business/political aspects and the technical aspects, because they often tend to fuel each other. Implementing multi-tenancy is political decision which implies design, implementation and operational decisions, which are not for free; which in turn then heat up any business discussion about the costs of the platform. And then the call goes back to the architect not to implement the full stack, but only a reduced one, which can cause trouble again on business side … there are a lot of these stories, and it only proves, that you can hardly do a decision in one domain without impacting the other.

But let’s focus now on the technical level and how it is influenced by multi-tenancy. In any multi-tenancy system the full and clean separation of the tenants is the ultimate goal. That means: no shared resources beyone the ones which are supposed to be shared intentionally. At least the usage of the shared resources must be restricted in a way, that one tenant cannot negatively influence the other tenants; or that the influence of any single tenant on the others is marginal and always managable. On the other hand it should be cost-effective, that means, that a multi-tenancy system for N clients must be cheaper than N non-multi-tenant systems (a single system for each tenant).

(If you reach this point it might make sense to evaluate if the additional cost of making a system capable to operate multiple tenants outweighs the cost and complexity of managing more systems. If that’s the case, stop here and replicate create a single-tenant application and deploy it to multiple systems.)

The simplest approach to multi-tenancy is to host all tenants (or as much as possible) on a single system. As all these tenants now live within the boundaries of a single instance (a single JVM, a single hardware/virtual machine) they share all the hardware resources (CPU, memory, I/O), but also the software resources (threads, queues, caches, „the application“). This sharing means formost, that the maximum performance of each tenant is limited under the assumption, that other tenants need resources at the same time too.
This scenario (let’s call it „friendly tenants“) is often encountered in enterprises, where multiple brands, divisions or coutries are hosted on a single platform. But it has some implications:

1. All tenants share the same application.
2. Downtime for platform upgrades/maintenance/bugfixes affects all tenants.
3. Platform failures affects all tenants.

These limitations can be quite heavy. While the limitations 2 and 3 are accepted in most cases (given that the platform is stable and performant otherwise), the limitation of the development scope is often considered as problem. Because it enforces, that all changes a tenants demands go into the platform; thus all requirements of all tenants are prioritized from a platform perspective („which features bring the most benefit for all tenants?“), so the priorities of a single tenant don’t have that much weight.
Of course you can allow custom development for individual tenants (maybe even by multiple development parties), but then the application must be designed and implemented carefully to avoid „friendly fire“ (changes to a tenant affects other tenants as well).

This „friendly tenant“ scenario is likely to have the lowest costs, as the usage of resources is low compared to the number of tenants and the individual requirements of tenants are often considered lower priority compared to the requirements shared by a set of tenants. With AEM you can implement such a scenario quite well using ACLs. The MSM gives you a good tool when the tenants also share content.

In the September session of the Ask the expert series (passcode: “Dispatch”) I talked about problems arising out the requirement to deal with multiple sites and each site having it’s own domain, and that a sling mapping is used to map the long repository paths to shorter URLs (like mapping /content/geometrix/en/services.html to http://geometrixx.com/services.html). I already tried to deal with this question in the Q&A part of the session, but I will write it here in more depth.

In the session on AEM dispatcher setups there was a question how to deal with shared content. If you do a straight-forward configuration of the dispatcher and map a shared content path (being it assets or pages) into the site structure of a site, the content is cached at this location in the dispatcher cache, but the invalidation happens only once at the „original“ path. So the content within the mapped paths in the site structure is not invalidated at all.

This is a problem, but you can see this problem from more than one angle.

The first question is, if you really need to share this content at all. I am not a SEO expert, but from what I heard, having duplicate content on multiple domains gives you a negative score on your page rank. Also from my point of view at some point the necessity rises to customize this shared content per tenant, which leads often to copy a shared page into the site and customize it there, essentially not using the shared content anymore. If there’s the risk of having this problem, you should think of using the MSM to avoid this „copy-and-adapt“ workflow and make it manageable. In that case you have true local copies and you don’t need to map the pages into the site content structure, avoiding the caching and invalidation problem completely.

The second question is, if it makes sense to offload all this shared content into a dedicated „shared ocontent“ domain, which is used by all sites; in that case the need to duplicate is avoided as well.

These are 2 suggestions to avoid some of the problems of the „shared content“ approach. If you cannot use them, you have to go the way of duplicate content at dispatcher level, with all the implications it has, mainly:

• potential SEO problems because of duplicate content
• increased disk consumption on dispatcher level

To deal with the problem of duplicate content and invalidation you have to go the way to create a custom invalidation logic, which is aware of your special setup and which does the invalidation accordingly. See the documentation on the dispatcher regarding this topic.