My whishlist for progressively decoupling drupal

My whishlist for progressively decoupling drupal

In the last years the world of web development was greatly influenced by frontend libraries like angular and react which allowed us build fast and interactive client-side user interfaces. As rich client-side interactions in the browser are here to stay, the Drupal community picked up on it and started to build systems using “headless” Drupal installations - using Drupal only as a data management and backend system and building the frontend from scratch with technologies like react.

The drupal community, initialized by a blog post of Drupals founder Dries Buytaert, discussed the the idea of “progressive decoupling” - adopting a client-side framework (like ember or react) into Drupal core to allow for the community to use a common base framework for richer client-side functionality. While keeping the traditional server-side rendering and routing system various elements should become more interactive and rendered by a client-side framework while communicating with Drupal via the REST API. Dries describes the idea in detail in his blog post.

Researching various options and reading up on the discussion in the Drupal community I thought I’ll write up a wishlist for Drupal frontend architecture that would in my mind simplify Drupal as a whole, while enabling possibilities attainable with client-side frameworks. This is by no means a concrete plan and surely doesn’t take into account the many intricacies of the Drupal render and theming system.

Base UI components around web components

One purpose of a client-side framework like Angular or React is creating and composing components, so when evaluating web components (and the Polymer framework) this is often perceived as different solutions for the same problem. React and libraries like Inferno actually do more than that: They also provide a DOM-Abstraction layer to be able to modify the DOM on a page in a performant way.

True component isolation

The version 1 of the web components specification (unlike V0 which was more of a proposal by Google) was a joint effort of browser vendors and most of the relevant features are either quite well supported or in development. Googles Polymer project (which released v2 not too long ago) provides a performant (like, very performant) and small polyfill. This would get us:

  • Real isolation of styling and logic for components: With the “Shadow DOM”-Feature, components bring their own assets, logic and stylesheets and are truly encapsulated. This would allow modules to provide complex components and complex associated styling which could be overridden transparently by themes.
  • Flexibility of implementation: Components could basically use whatever libraries they choose internally. They could be written in Typescript, Purescript, using Elm or provide Web Assembly binaries. This is not necessarily an advantage for development in a community, but adds to the flexibility while taking tedious frontend discussions (which aren’t Drupals focus anyway) out of the equation.
  • We wouldn’t necessarily need a build step to prepare component assets. This could be handled on a component-level and would greatly simplify working with components in Drupal as a whole. I guess having to build the whole Drupal frontend is comparable to compiling all your linux packages from scratch.

Component re-rendering and composition

The great thing about using web components in our traditional Drupal architecture is, that we can keep almost everything in place concerning server side routing while having all the perks I described above.

Two issues in this context remain:

  • Dynamically re-rendering things inside of components: Inside of our components one could use React, Inferno or any other dom-diffing library, but multiple libraries would still be a certain performance hit, even if we use intelligent, client-side asset management (more on this later).
  • Dynamic composition of components at the base level: We would still need a dom-diffing library in core.

While this seems like a big issue, I don’t think that it’s actually a huge problem:

  • We could use something like Preact in core, that totally suits our use case with minimal filesize (~3kb gzipped).
  • As this is in core, it would be safe to say that many maintainers would opt-in to this core library automatically.
  • Even if not: I’d rather have a great component with a heavier bandwidth toll and great encapsulation and reusability than no component at all.
  • We could make “entry-point” components that register with a centralized state management (more on this later) which kind of weakens the cohesion of our dynamic render structure, but also makes everything more flexibly coupled.

Use forward-facing frontend asset management

What is the purpose of asset management in Drupal? Modules register CSS and JavaScript assets for certain elements of a page so that they can be included, sorted and possibly aggregated / sharded. This is quite similar (but much less sophisticated) to what bundlers do in a typical JavaScript-Frontend context:

  • Minimizing the amount of individual HTML requests
  • Minimizing the size of files with compression
  • Managing the load order or assets
  • Managing the version of included files and making files and libraries discoverable for reuse.

A builder in JavaScript does many similar things but also prominently handles tasks like transpiling (transforming code from one syntax - for example ES6 JavaScript - to another - for example regular old browser-supported JavaScript). I firmly believe that the browser as a platform will (and already does) handle many things we use our asset management for today. A few concepts that differ from our current strategy would be:

No more transpiling

ES6 as a syntax provides us with a format and system for defining modules in JavaScript - something that has to be solved by bundlers with transpiling and own module formats like CommonJS or AMD. In the future we can use a subset of ES / JavaScript as a common baseline and avoid transpiling.

No more aggregating

While not completely negating them, HTTP/2 provides a greatly improved mechanism to handle many individual file requirements for a web page. In combination with web components Drupal wouldn’t have to know about each individual file asset anymore. Neither is it necessary for loading all required assets, nor is it an optimization issue anymore. Since HTTP/2 is already well supported the next version of Drupal could just require it.

Unified dependency management (client-side)

Using web components, we would want that if two components use a similar dependency, the dependency would just have to be downloaded once. This is typically an aspect of dependency management, because it has to be crystal-clear which package and which version of a package a component depends on. Drupal uses composer for it’s php-based internals (which is a great step forward) and it’s own system for frontend assets. Client-side applications mostly use package managers like npm or bower, while mostly not having a solution for dynamic loading of dependencies. I am sure that in the long term, browsers will implement a common dependency management for frontend assets (which means JS Modules, HTML Templates, CSS Styles, Fonts ..), which is a heavily discussed issue in the context of the html import specification.

Today one of the best options are module loaders like SystemJS, which provide many features that we need in this context. When ES6 Modules are supported natively we should get another sharp simplification of the necessary client-side dependency management.

All of this would provide us with amazing flexibility: To use a web component on a page, drupal would just have to include a single file, all dependencies necessary would be resolved in the client and, most importantly, outside of Drupal.

Create / use a flexible solution for state management and server-requests

React was very influential on our perception of the importance of state management. In itself it enforces the concept of one-way data flow: Data flows only from “top” to “bottom”, components are defined by static properties, passed in by the parent element. The logic of one component can only affect components that are nested below. To regain flexibility while keeping the very simple reasoning behind this model, often libraries like Redux or MobX are used to have a consistent way to manage and manipulate a content state object which serves as the base data object that is passed down to components.

Having a common solution for state management would be great to enable Drupal and its components to depend on a common data handling infrastructure. Components could use this common structures to dynamically alter and expand the data that is available, much like today Drupal passes in JavaScript variables and further ajax requests are used to get additional data.

One additional advantage of this is that similar queries can be batched and “deduplicated”, state can easily be served by offline storage and generally easily manipulated (for example for debugging purposes). This is also very closely related to GraphQL which could be used as a flexible means to efficiently query data from Drupal in a dynamic fashion. One architecture that already provides many concepts of this is Facebooks' Relay.

Summary - A slightly different frontend architecture

To summarize my wishlist into a rough sketch of a slightly changed Drupal frontend architecture:

Rough sketch of drupal progressive decoupling proposal

The changes are mainly defined by following concepts:

  • Encapsulate resources with web components
  • Manage frontend assets and packages on the client
  • Provide a central solution for state management (using rehydrated state from the initial page load or intelligently query additional data with GraphQL).
  • Provide a story for dynamically updating nested components based on changes in the central state management.

Conclusion

What are the problems with the described approach?

  • We have to make JavaScript a requirement for page display.
  • Prerendering a page on the server would come with added complexity (and may not be feasible).
  • We need a solid solution for client-side dependency handling which is quite an effort, but should be something that’s future-proof and maybe even could be gradually removed with browsers implementing their own solution or standard.
  • This is in many ways legacy-unfriendly with dependencies on many emerging standards with spotty implementations in the client, and HTTP/2 while being fairly common is not always available, especially in the low-end hosting segment.

What is to be gained?

So what high-level benefits can be gained by adopting such an infrastructure:

  • We create a common infrastructure that is rooted in future web standards.
  • We allow for great flexibility in implementation of individual components.
  • We can reuse web components of other vendors with great ease.
  • We can create parts of our site as web components using traditional server-side routing or use client-side routing while retaining the functionality of the components since client-side dependencies are loaded on demand on the client anyway.
  • We avoid - as a community - having to deal with many problems people would have from adopting one single complex library like ember and an equally complex build step.
  • Performance-wise the polyfills we need are fairly small, while performance is good and should only improve with better support from the browser vendors.

For me Drupal was always about innovation and providing clean, well thought-out and general solutions to a wide array of problems. The frontend was lately dominated by client-side frameworks, which provide solutions to problems that are, at least to a degree, the problem space of the browser as an application platform.

I think that the common perception of the browser as a simple document and layout engine where you have to provide huge amounts of JavaScript to make it into an application platform is going to change. I think that big libraries are going to get more fragmented and interchangeable and that this is a great opportunity for Drupal. What would have to change is the perception of JavaScript as a simple enhancement of the document-centric model we came to work with. JavaScript as a runtime environment is fundamental and an equally important part of the browser like the DOM or rendering engine.

The great thing is that a proof-of-concept is possible today, as Drupal 8 allows us to build what we need for this in contrib. While there are a few things we lack today we will continue the explore the idea at Signal to see how a web component based architecture fits with Drupal.

I’m totally aware that I surely took a lot of conceptual shortcuts while describing this whole approach and would love to have some feedback from the community on this topic - just send me an E-Mail or write me on Twitter!