Sometimes you need to disable a service or a component; a simple example for this a servlet, which is used on authoring instance, but which must not be active on publish. There are several ways to achieve this. (In this blog post whenever I mention “service”, you can implicitly assume that it also works for SCR components; technically even “component” would be right wording, but in the AEM world “component” is heavily used word with a number of different meanings.)

A very simple and smart solution for your own codebase is the use of the SCR configuration policy; when this is used on a OSGI service the SCR runtime won’t start the service if no dedicated OSGI configuration exists (even the activate() method isn’t called). And because you can create OSGI configs based on run mode it’s the perfect way to enable or disable services.

Nice examples for this can be found in ACS AEM Commons:

• Bundle Disabler
• Component Disabler

This the the recommended way to write services; the decision to run or not run it is done on deployment/configuration time and not during development. And it’s the most complete way, because with proper configuration the service will never get active at all. The only (very) small drawback is that your project-managed OSIG configurations will increase.

A different way is to use a special property „enabled“, which is then checked in the service before doing something useful. But when you use the enabled-property, the service is properly started and registered to the OSGI runtime; thus it might get registered as servlet and into other service factories. You never know what is happening or what not, so you it’s always best to have the code ready.
This approach gives you also the choice on deployment time to enable or not to enable the service. But it has the drawback, that the service is active and code of it might run before checking the „enabled“ status. So from my understanding there is never really a usecase for this “enabled” property. An if it has a different function than turning the service on or off, it shouldn’t be named “enable”.

If you need to disable services, which are not under your control, and which neither offer a „enabled“ property nor the configurationPolicy approach, the only remaining choice is the ComponentDisabler of ACS AEM Commons. That’s basically a hack and should be your last resort, because it cannot prevent the startup of the service, but in fact shuts it down after the service has been started (and might have already been working). But if you can live with this constraints, it’s the way to go.

If you are a developer, I strongly recommend to learn and use the SCR ConfigPolicy setting!

Content which is being produced and stored inside AEM is often widely used. Not only for direct publishing to the web, but nowadays also in Emails (using Adobe Campaign) or for the consumption of mobile apps. Also a very common case is the integration of AEM content in 3rd party systems, when content maintained in AEM is fetched by a portal. So the portal provides the transactional parts, but the content is fetched from AEM. This is the usecase I want to discuss in this post. When I write „portal“ it’s most often a J2EE portal, but there also other options. For this post the underlying technology stack doesn’t really matter.

In this case the portal is always the leading application, and AEM has just a supporting function (providing content). Depending on the exact case, you can embed AEM content as a portlet or use the REST approach to fetch AEM content. In this case AEM is mainly used for content authoring.

This approach has the unique benefit, that you can continue to use the existing portal solution and and provide your authors a easy-to-use solution for content authoring. The existing architecture is just extended by adding AEM.

But from my point of view this approach also has some severe problems:

• In the editmode AEM can only display the elements, which are available to AEM. If the portal displays AEM content as part of the page next to a number of other elements, you don’t have these elements available on AEM. This limits the usability of the editmode or preview mode to display content in an end-user way.
• A similar problem is, that with AEM pages you can normally edit most pieces of a page. In the case of the portal, you have to limit the possibilities of an author to a way provided by the portal. That means, if the portal does not allow you to change the page footer or add special HTML headers, you cannot change this from AEM (although it might be possible ootb). Within an AEM application I always recommend to allows authors to change every text and image of a page (including headers and footers), avoiding any hardcoded content.
• When content is created in AEM, you need to develop templates and components for it. If you want to display this content in the portal, you need to build its equivalent there as well, but with a different technology. This is doubling the cost and you build dependencies in the cycles of the AEM development and the portal application development. You need to spend development on both sides to include new components or change the parts of a page, where content should be managed from authors.
• Recent versions of AEM provide a good integration of the Adobe Marketing Cloud features, so authors can easily use them. When a portal is setup in a way, that it can fetch content from AEM, this integration normally needs additional effort and implementation work, which you don’t have when AEM is in the front.

A personal conclusion: I think, that using AEM just a content-authoring system is possible, but you ignore many of the features of it, which bring a lot of value. You increase costs by the need to develop new components and templates twice (for AEM and the portal) and decrease the time-to-market by synchronizing 2 development streams, which should basically be independent. And you cannot use many of the new Adobe Marketing Cloud integrations provided out-of-the-box.

So there are quite some arguments (especially for enduser-facing systems) not to use AEM just as a simple content-feed, but to establish AEM as frontend of your platform.

Yesterday’s posting „TarMK and NAS“ got quite some attention, and today I got several times the question „And what about SAN?“. Well, here the answer to „Do you recommend TarMK on SAN?”

A little background first: A SAN (Storage area network) is a service, which offers block devices. It is part of today’s enterprise datacenter’s infrastructure, where attaching local disks to servers is not feasible and does not scale. If you are familiar with PCs, you can consider a block device like a partition on your hard disk. You cannot use a partition by itself, but you have to format it and put a filesystem on it.
That’s basically the same with a SAN: You get it as a raw device (called volume), you put a filesystem on it (for example ext4 when you use Linux) and then you can use it just like any other local drive. The only difference to a local drive is, that the connectivity is not provided by local SATA-port, but over a network (you’ll find the terms iSCSI or Fiber Channel, but that’s too much detail here).

And that’s the huge difference: With a SAN you get a block device, with a NAS you’ll get a shared filesystem.

So the basic principle is, that you can treat SAN like any local storage. And if you format it with a filesystem like ext4, btrfs or NTFS on Windows, you have a local filesystem. And like I said in yesterday’s post: When you have a local filesystem, where only a single system is controlling access to it, you can use mmap. And mmap is all we care about here!

My recommendations for TarMK are:

• When you have the choice between SAN and NAS (sometimes you have): drop the NAS and go for SAN.
• And when you have the choice between SAN and local drives, choose „SAN“ as well. Why? Because you never need to deal with the problems of „my hard drives are full and we don’t have any empty drive bays anymore on this server!“ anymore. Just allocate some more space to your SAN volume, resize the filesystem and that’s it. When you have mmap available for your TarMK, filesystem performance shouldn’t be something to worry about.

Today the question was raised, if TarMK running on NAS is a good idea. The short answer is: „No, it’s not a good idea“.

The long answer: The TarMK relies on the ability of the operating system to map files into memory (using the so-called memory-mapped technology, short: mmap; see this wikipedia page on it). Oak does this for the heavily used parts of the TarMK to increase performance, because then these parts don’t need to be read from filesystem again, but are rather always available in memory (which is by at least an order of magnitude faster). This works well with a local filesystem, where the operating system knows about every change happening on the filesystem, because it is the only one through which access to this filesystem happens, and it can make sure, that the content of the file on disk and in memory are in sync. I should also mention, that this memory isn’t part of the heap of the JVM, but rather the free RAM of the system is used for this purpose.

With a NAS the situation is different. A NAS is designed to be accessed by multiple systems in parallel without the need to synchronize between each other. The 2 most common filesystems for this are NFS and SMB/CIFS. On NFS one system can open a file and is not aware that a second system modifies in the same time. This is a design decision which prevents that a system can keep the content of a file on NFS and in-memory in sync. Thus mmap is not usable when you use a NAS to store your TarMK files.

And because mmap is not usable, you’ll get a huge performance impact compared to a local filesystem where mmap can be used. And then I haven’t even mentioned the limited bandwidth and higher latency of a remote storage compared to local storage.

If you migrate from CRX 2.x (till AEM 5.6.1) this problem was not as visible as it is now with Oak, because there was the BundleCache, which cached data already read from disk; this bundle cache is an in-memory, in-heap structure and you had to adjust the heap size for it. CRX 2.x did not use mmap.

But Oak does not have this in-memory cache any more, but relies on the mmap() feature of the operating system to keep the often-accessed parts of the filesystem (the TarMK) in memory. And that’s the reason why should leverage mmap as much as possible and therefor avoid a NAS for TarMK.

Servicepack 3 for AEM 6.0 is now available (releasenotes).

Here’s the complete list of sling bundles in stock AEM 6.0 and the various levels of servicepacks. Bundles which are not available with a specific version are listed as “-“, version numbers marked in red appeared first in this servicepack. Where possible I added the links to the changes in servicepack 3.

The most notable change for SP3 from a Sling perspective is the switch to the fsclassloader (see 6D’s blogpost for it) for all scripting languages. So the compiled JSPs do not longer reside inside the repository (/var/classes), but now are placed in the filesystem.

In the last articles (1,2) I covered some aspects of multi-tenancy, which are very likely to occur in AEM projects (but not restricted to such projects). I stressed that there a lot of aspects which have the potential to cause trouble on a non-technical level. But you cannot draw a clear line between the business/political aspects and the technical aspects, because they often tend to fuel each other. Implementing multi-tenancy is political decision which implies design, implementation and operational decisions, which are not for free; which in turn then heat up any business discussion about the costs of the platform. And then the call goes back to the architect not to implement the full stack, but only a reduced one, which can cause trouble again on business side … there are a lot of these stories, and it only proves, that you can hardly do a decision in one domain without impacting the other.

But let’s focus now on the technical level and how it is influenced by multi-tenancy. In any multi-tenancy system the full and clean separation of the tenants is the ultimate goal. That means: no shared resources beyone the ones which are supposed to be shared intentionally. At least the usage of the shared resources must be restricted in a way, that one tenant cannot negatively influence the other tenants; or that the influence of any single tenant on the others is marginal and always managable. On the other hand it should be cost-effective, that means, that a multi-tenancy system for N clients must be cheaper than N non-multi-tenant systems (a single system for each tenant).

(If you reach this point it might make sense to evaluate if the additional cost of making a system capable to operate multiple tenants outweighs the cost and complexity of managing more systems. If that’s the case, stop here and replicate create a single-tenant application and deploy it to multiple systems.)

The simplest approach to multi-tenancy is to host all tenants (or as much as possible) on a single system. As all these tenants now live within the boundaries of a single instance (a single JVM, a single hardware/virtual machine) they share all the hardware resources (CPU, memory, I/O), but also the software resources (threads, queues, caches, „the application“). This sharing means formost, that the maximum performance of each tenant is limited under the assumption, that other tenants need resources at the same time too.
This scenario (let’s call it „friendly tenants“) is often encountered in enterprises, where multiple brands, divisions or coutries are hosted on a single platform. But it has some implications:

1. All tenants share the same application.
2. Downtime for platform upgrades/maintenance/bugfixes affects all tenants.
3. Platform failures affects all tenants.

These limitations can be quite heavy. While the limitations 2 and 3 are accepted in most cases (given that the platform is stable and performant otherwise), the limitation of the development scope is often considered as problem. Because it enforces, that all changes a tenants demands go into the platform; thus all requirements of all tenants are prioritized from a platform perspective („which features bring the most benefit for all tenants?“), so the priorities of a single tenant don’t have that much weight.
Of course you can allow custom development for individual tenants (maybe even by multiple development parties), but then the application must be designed and implemented carefully to avoid „friendly fire“ (changes to a tenant affects other tenants as well).

This „friendly tenant“ scenario is likely to have the lowest costs, as the usage of resources is low compared to the number of tenants and the individual requirements of tenants are often considered lower priority compared to the requirements shared by a set of tenants. With AEM you can implement such a scenario quite well using ACLs. The MSM gives you a good tool when the tenants also share content.

In the September session of the Ask the expert series (passcode: “Dispatch”) I talked about problems arising out the requirement to deal with multiple sites and each site having it’s own domain, and that a sling mapping is used to map the long repository paths to shorter URLs (like mapping /content/geometrix/en/services.html to http://geometrixx.com/services.html). I already tried to deal with this question in the Q&A part of the session, but I will write it here in more depth.

In the session on AEM dispatcher setups there was a question how to deal with shared content. If you do a straight-forward configuration of the dispatcher and map a shared content path (being it assets or pages) into the site structure of a site, the content is cached at this location in the dispatcher cache, but the invalidation happens only once at the „original“ path. So the content within the mapped paths in the site structure is not invalidated at all.

This is a problem, but you can see this problem from more than one angle.

The first question is, if you really need to share this content at all. I am not a SEO expert, but from what I heard, having duplicate content on multiple domains gives you a negative score on your page rank. Also from my point of view at some point the necessity rises to customize this shared content per tenant, which leads often to copy a shared page into the site and customize it there, essentially not using the shared content anymore. If there’s the risk of having this problem, you should think of using the MSM to avoid this „copy-and-adapt“ workflow and make it manageable. In that case you have true local copies and you don’t need to map the pages into the site content structure, avoiding the caching and invalidation problem completely.

The second question is, if it makes sense to offload all this shared content into a dedicated „shared ocontent“ domain, which is used by all sites; in that case the need to duplicate is avoided as well.

These are 2 suggestions to avoid some of the problems of the „shared content“ approach. If you cannot use them, you have to go the way of duplicate content at dispatcher level, with all the implications it has, mainly:

• potential SEO problems because of duplicate content
• increased disk consumption on dispatcher level

To deal with the problem of duplicate content and invalidation you have to go the way to create a custom invalidation logic, which is aware of your special setup and which does the invalidation accordingly. See the documentation on the dispatcher regarding this topic.

A recurring topic in AEM projects is multi-tenancy. Wikipedia describes multitenancy as „[…] software architecture in which a single instance of a software […] serves multiple tenants“. In the AEM projects I’ve done I encountered this pattern most when a company wants to host several brands and/or subsidiaries as independent tenants within a single AEM platform (that means: connected authoring and publishing instances). In this blog post I only cover the aspect of multi tenancy in a single company. Hosting tenants for multiple independent companies is a different story and likely even more complex.

At first sight multi-tenancy seems to be only a technical problem (separation of content/templates/components, privileges, etc.), but from what I learned, there is a much bigger problem, which you should solve first. And that’s the aspect of organization and governance.

Multitenancy is hard when different tenants (being brand organizations or subsidiaries) need to integrate into the single platform. Each tenant has its own requirements (depending on its special needs), its own timelines, and its own budget. You have larger tenants and smaller tenants on your AEM platform. But this does not necessarily reflect the power of these tenants inside the company. It may even contradict, and a smaller or less powerful organization or brand has such demands, that it will be the largest tenant on your AEM platform.

That means, that there will be conflicts, when it comes to defining scope, timeline and budget. The tenant which contributes more budget wants to have more influence on these 3 aspects than another tenant, which spends a significant smaller amount. But the smaller tenant might have needs which can overrule this, for example a tradeshow where some new features on the brand pages are absolutely required, while the other tenant (yet more powerful within the organization) has requirements, which are important in a more distant future. How are these requirements prioritized?

These questions (and conflicts) are not new, they exist for decades, even not centuries. But they have a huge impact on the platform owner. The platform owner wants to satisfy the needs of all the tenants, but is often faced with contradicting requirements; while on the technical side these can be often (more or less) solved (just by throwing people and time onto the problem), there are still things which are in the first place organizational issues, and which can only be solved on a organizational or political level. Then you have topics like:

• How can you coordinate different timelines of different tenants, so you can satisfy all their needs?
• Tenants want to have their own development teams or agencies. How can they work together and feed their results into a single platform without breaking it? Who’s responsible when the platform broke down?
• How do you do funding when tenants contribute development work to the platform and other benefits from this work as well? Invoicing the tenants which benefit from other tenant’s development work?
• What’s the role of the platform owner? Does the platform have its own budget or is the platform solely funded by the tenants? Is the platform owner able to reject feature requests from tenants and say “no”?
• How should the platform owner react with contradicting requirements? Is splitting the single platform into multiple ones (with different codebase) something which is desirable?

There are a lot of questions like these, and they are very specific to the company and the platform. They can all be solved, but the company and the organization itself has to solve them, but not the platform development team(s). Because then the organization foo will go down even to the developers (and as we all know: this kind of human being doesn’t really like that :-))

My ideal multi-tenancy project looks like this: A strong platform owner with some budget on its own. The tenants have pretty much the same size, and they fund the platform for the largest part to the same amount each. A steering committee (with participants from all tenants) deciding on all the organizational topics, and the same on the technical level if required. Requirements are consolidated on a project level and then implemented by a team, which is reporting to the platform owner.

Yeah, I have to admit, I haven’t found that customer project yet 🙂 But in such a project you as a member of the development team don’t really feel anymore the multi-tenancy aspect on an organizational level, but you only have to deal with it only on a technical level. Which is very nice.