Background

LoopBack is an open-source Node.jsframework built for API developers. Its primary goal is to help create APIs asmicroservices from existing services/databases and expose them as endpoints forclient applications, such as web, mobile, and IoT. LoopBack connects the dotsbetween accepting API requests and interacting with backend resources. Byfacilitating developers to implement API logic with out of box integrationcapabilities, LoopBack establishes itself as the API composition layer todifferentiate from other frameworks,such as Express, Hapi, andSails.

loopback-composition

Up to version 3.x, LoopBack built on the popularExpress framework. In retrospect, basing LoopBack onExpress was the right decision: Standing on the shoulders of Express enabledLoopBack to focus on adding value for API creation experience withoutreinventing the wheel. LoopBack also has benefitted from the Express ecosystem,especially ready-to-use middleware modules from npm as well as valuableknowledge and support by the community.

With LoopBack, developers can create and expose APIs just like following arecipe. LoopBack introduces a set ofcore concepts thatrepresent the key aspects of API implementation. To create APIs out of existingdatabases or services, developers can simply scaffold a LoopBack application,then add necessary JSON declarations and Node.js code to get their APIs up andrunning in a few minutes.

LoopBack uses Express routing and middleware as the plumbing to arequest/response pipeline for API use cases, such as authentication,authorization, and routing. Beyond inbound HTTP processing, LoopBack providesintegration facilities such as models, datasources, and connectors to allow APIlogic to interact with various backend systems, including but not limited to,databases, REST APIs, SOAP web services and gRPC microservices. The ability toglue inbound communication and outbound integration makes LoopBack a verypowerful framework for API developers. The diagram below illustrates howLoopBack fits into a typical end-to-end API processing flow.

loopback-overview

LoopBack has grown significantly in features and users with many years ofdevelopment and multiple releases. LoopBack has been well-recieved by thedeveloper community. As an indication, the community has developedmany extensions. The core teamhas also learned a lot from what we have done as well as great feedback from thecommunity.

Why LoopBack 4?

Like many projects, LoopBack has started to experience growing pains, especiallyas:

  • The code base becomes more complicated over time with more modules and morefunctionality. We would like to have more maintainers and contributors tohelp out. But the learning curve is getting steep. One of the things toblame is JavaScript itself, which is weakly-typed and lack of constructssuch as interfaces to explicitly define contracts between code. There isquite a bit hidden knowledge that is not explicit or obvious for new comers.

  • Technical debt is accumulating, for example inconsistent designs acrossmodules and feature flags for different behaviors. Here are a few examples:

    • Various modules use different registries to manage different types ofartifacts, such as remoting metadata, models, datasources, and middleware.
    • Different flavors are used to allow custom logic to interceptrequests/responses at various layers, such as middleware, remote hooks,CRUD operation hooks, and connector hooks.
    • More feature flags were added over time to keep backward compatibilitywhile enabling users to opt-in to new behaviors.

  • It is becoming more difficult to add new features or fix bugs as some areasstart to reach the limit of the current design.

    • The loopback-datasource-juggler module is a kitchen sink for manythings, such as typing, data modeling, validation, aggregation,persistence, and service integration.
    • Models are overloaded with multiple responsibilities, such as datarepresentation, persistence, and mapping to REST. Models are tied todatasources and it’s not easy to reuse the same model definition againstdifferent datasources.

  • It’s not easy to extend the framework without requesting the core team tomake code changes in LoopBack modules. The current version of LoopBack hasad-hoc extensibility at various layers. Extension points are notconsistently defined. For example,

    • Use Express to register middleware.
    • Use remoting hooks to intercept remote method invocations.
    • Use CRUD hooks to add logic around CRUD operations.

  • More projects start to use LoopBack as the underlying platform. Such usecases require more knowledge of LoopBack internals and more flexibility anddynamicity to leverage LoopBack to manage and compose artifacts using ametadata driven approach. Some good examples are:

    • Multi-tenancy which requires artifact isolation between tenants.
    • Metadata APIs to manage/activate model definitions and datasources.
    • New interaction patterns for connectors, such as eventing or messaging.
    • Extra metadata for model definitions.Since the release of 3.x, the team has been brainstorming about how to sustainand advance LoopBack. We did a lot of homework, triaged existing GitHub issues,reached out to community members and downstream products, and evaluated relevantframeworks and technologies to answer to the following questions:

  • Who is the target audience of LoopBack? Why are they interested in LoopBack?What do they use LoopBack for and how do they use it?

  • What are the critical pain points? Can we address them incrementally withoutrebuilding a new foundation?
  • What are the most requested features? Is it possible to add such features withthe current design?
  • What are latest and greatest technologies in our space? What value will theybring in if we start to adopt them?
  • How to scale the development and maintenance of LoopBack? How do we allowlarger development teams to collaborate on creating APIs using LoopBack?

  • How to further grow the community and expand its ecosystem? What can we do tobring more users and contributors to LoopBack?LoopBack has gained traction among a spectrum of users beyond Node.jsapplication developers, including:

  • API developers - Use LoopBack to create APIs in Node.js.

  • LoopBack maintainers and contributors - Build and maintain modules by theLoopBack project .
  • Extension developers - Contribute extensions to LoopBack to augment theframework.
  • Platform developers - Leverage LoopBack as the base to build theirvalue-added offerings.loopback-ecosystem

The core team decided to make a bold move and rebuild LoopBack to meet the needsof all the above groups. The decision led to the inception of LoopBack 4, a newgeneration of API creation platform. For more information, read the blog postAnnouncing LoopBack.next, the Next Step to Make LoopBack Effortlessly Extensible.

Objectives

LoopBack 4’s goals are:

  • Catch up with latest and greatest technology advances.

  • Promote extensibility to grow the ecosystem.

  • Align with cloud native experience for microservices.

    • Adopt cloud native microservices by adopting initiatives such asCloud Native Computing Foundation.
    • Make LoopBack a first-class citizen of the microservices ecosystem.

  • Remove the complexity and inconsistency across modules.

    • Use a consistent registry and APIs to manage artifacts and theirdependencies.
    • Pay down technical debts by refactoring complex modules.

  • Separate concerns for better composability.

    • Introduce new concepts such as controllers and repositories to representdifferent responsibilities.
    • Break down the runtime as a set of services and utilize the extensionpoints/extensions pattern to manage the registration, resolution, andcomposition.

Design principles

We decided not to take a “big-bang” approach to build LoopBack 4. Instead, weare doing it incrementally in multiple stages with smaller steps. This approachallows us to better engage the community from the beginning. We are followingthe principles below to pursue architectural simplicity and extensibility:

  • Imperative first, declarative later

Everything can be done by code via APIs. The LoopBack team or communitycontributors can then create varieties of user experiences with such APIs.For example, with APIs to define models, we allow applications to declaremodels in JSON or YAML files so that they can be discovered and loaded. Anextension can parse other forms of model definition, such as JSON schemas,ES6 classes with decorators, schemas in OpenAPI spec, or even XML schemasinto LoopBack model definitions.

We can also leverage programming constructs such asdecoratorsallow developers to supply metadata in code. Furthermore, LoopBack artifactscan be declared in JSON or YAML files, which will be handy for users togenerate and manipulate them by hand or tooling.

  • Build minimum features and add more later if necessary

Apply YAGNI (You Aint’t Gonna Need It). Design and build for what is needednow, not for what you think you may need in the future. There are manydifferent perspectives in API creation and people ask for a lot of features.Starting with MVP allow us to reach the root of the issues without beingderailed by noises and build the absolutely necessary features as the corebuilding blocks.

  • Developer experience first

Always keep in mind that LoopBack is built for developers by developers. Ourfirst priority is to make API developers’ life easier. When we design APIsand user interfaces such as a CLI or GUI, we want to make sure they areintuitive to and natural to their thought process.

Implementation stages

Here are the stages we are marching through toward the final version of LoopBack4 as illustrated below.

  • Rebase and rewrite the core

    • Leverage TypeScript for better code quality and productivity.

      • Provide optional type system for JavaScript.
      • Provide planned features from future JavaScript editions to currentJavaScript engines.

    • Unify the asynchronous programming model/style.

      • 100% promise-based APIs.
      • Async/Await as first-class async programming style.

    • Implement an IoC Container for better visibility and extensibility

      • Universal registry across different modules
      • Dependency injection as a pattern to manage dependencies

    • Introduce Component as packaging model for extensions

      • Component can be a npm module or a local directory
      • Component encapsulates a list of extensions as a whole

  • Validate the core design by implementing an REST/HTTP invocation chain

    • Add top-down REST API creation which starts with OpenAPI specs.

    • Build sequence of actions for inbound http processing

      • Introduce sequence as the composition of actions
      • Implement the most critical actions to fulfill the REST API routing andinvocation
    • Introduce controllers as entry points for API-related business logic.

Models are the centerpieces of the current LoopBack applications. . Theytake multiple responsibilities:

  1. - Data modeling
  2. - Anchor for API related business logic
  3. - Persistence or service invocation
  4. - Mapping to REST HTTP/JSON endpoints
  • Authentication as a component

Implement the core functionality of authentication as a component, whichincludes:

  1. - Decorators to denote authentication requirement
  2. - <code>authenticate</code> action to handle authentication
  3. - Extension points for various authentication strategies

  • Rebuild our integration and composition capabilities

    • Introduce repositories to represent data access patterns such as CRUD orKey/Value stores
    • Provide a reference implementation of CRUD and KV flavors of repositoryinterfaces using the legacy juggler and connectors
    • Refactor/rewrite the juggler into separate modules
      • Typing system
      • Model and relation definition
      • Validation
      • Query and mutation language
      • DataSource
      • Repository interfaces and implementations for data access
      • Service interfaces and implementations for service invocations
    • Define interfaces and metadata for connectors
    • Rewrite connectors
  • Declarative metadata and bootstrapping

LoopBack manages a set of artifacts, such as models, relations, datasources,connectors, ACLs, controllers, repositories, actions, sequences, components,utility functions, and OpenAPI specs. In addition to the programmaticapproach to describe these artifacts by code (apis and decorators), we wouldlike to add declarative support so that they can be declared in JSON/YAMLfiles.

  • Define a new domain-specific language (DSL) in JSON/YAML format andcorresponding templates.
  • Define the project layout to organize project artifacts.
  • Leverage the IoC Context to manage metadata/instances of such artifactsfollowing the extension point/extension pattern.
  • Define the lifecycle and serialization/de-serialization requirements foreach type of artifact.

  • Add a boot component to discover/load/resolve/activate the artifacts. Theboot process can be tailored for both tooling and runtime.

    • Tooling (CLI & UI)

  • Add CLI and UI tools to:

    • Scaffold LoopBack 4 applications
    • Manage artifacts such as sequences, actions, controllers, repositories,services, datasources and models
    • Enable cloud native experience

  • Allow controllers to be exposed as gRPC services

  • Allow interaction with other gRPC services
  • Integration with microservices deployment infrastructure such as Dockerand Kubernetes
  • Integration with service meshThe following diagram illustrates the high-level building blocks of LoopBack 4:

loopback-stack

Please note there is a common layer below the different functional areas in thestack. Let’s examine the need to build a new core foundation for LoopBack 4.

A new core foundation

The core responsibility

LoopBack itself is already modular. For example, a typical LoopBack 3.xapplication’s dependency graph will have the following npm modules:

  • loopback
  • strong-remoting
  • loopback-datasource-juggler
  • loopback-connector-*

  • loopback-component-explorerLoopBack manages various artifacts across different modules. The following are alist of built-in types of artifacts that LoopBack 3.x supports out of box:

  • Model definitions/relations: describes data models and their relations

  • Validation: validates model instances and properties
  • Model configurations: configures models and attaches them to data sources
  • Datasources: configures connectivity to backend systems
  • Connectors: implements interactions with the underlying backend system
  • Components: wraps a module that be bootstrapped with LoopBack
  • Remoting: maps JavaScript methods to REST API operations
  • ACLs: controls access to protected resources
  • Built-in models: provides set of prebuilt models such as User, AccessToken,and Role
  • Hooks/interceptors
    • Express middleware
    • remote hooks
    • CRUD operation hooks
    • connector hooks
  • Security integration
    • Identity and token management
    • Authentication schemes
      • Passport component for various authentication strategies
  • Storage component for various local/cloud object storage systems
    • Local file system
    • Amazon S3
    • Rackspace
    • Azure
    • Google Cloud
    • OpenStack
    • IBM Cloud Object Storage
  • Push component for mobile push notifications
    • iOS
    • Android
  • Different API styles
    • REST
    • gRPC
    • GraphQLMetadata for these artifacts form the knowledge base for LoopBack to glue allthe pieces together and build capabilities to handle common API use cases.

How to represent the metadata and their relations is the key responsibility ofthe LoopBack core foundation. It needs to provide a consistent way to contributeand consume such building blocks.

Key ingredients for the core

The core foundation for LoopBack 4 is responsible for managing various artifactsindependent of the nature of such artifacts.

  • A consistent registry to provide visibility and addressability for allartifacts.

    • Visibility: Each artifact has a unique address and can be accessed via a URIor key. Artifacts can also be visible at different scopes.

    • Extensibility: LoopBack artifacts can be managed by types. New artifacttypes can be introduced. Instances for a given type can be added, removed,or replaced. Organizing artifacts in a hierarchy of extensionpoints/extensions decouples providers and consumers.

  • Ability to compose with dependency resolution.

    • Composability: It’s common that one artifact to have dependencies on otherartifacts. With dependency injection or service locator patterns, the corewill greatly simplify how multiple artifacts work together.

  • A packaging model for extensions.

    • Pluggability: Extensions can be organized and contributed as a whole. Weneed to have a packaging model so that extension developers can create theirown modules as bundles and plug into a LoopBack application.

Why Express behind the scene?

Background

LoopBack had always been built on Express so we can leverage the vast communityand middleware in the Express ecosystem BUT it presented some challenges forLoopBack. With LoopBack 4 we considered moving away from Express (and even builtthe framework without Express) but eventually circled back to Express because ofits vast ecosystem.

Some of the gaps between what Express offers and LoopBack’s needs are:

  • Lack of extensibility

Express is only extensibile via middleware. It neither exposes a registrynor provides APIs to manage artifacts such as middleware or routers.

  • Lack of composability

Express is not composable. For example, app.use() is the only way toregister a middleware. The order of middleware is determined by the order ofapp.use.

  • Lack of declarative support

In Express, everything is done by JavaScript … In contrast, LoopBack isdesigned to facilitate API creation and composition by conventions andpatterns as best practices.

Circling back to Express with a twist

The main purpose of LoopBack is to make API creation easy, interacting withdatabases, services, etc., not middleware for CORS, static file serving, etc. Wedidn’t want to reinvent the wheel by writing new middleware for LoopBack 4.

The team explored leveragingExpress or Koa (butonly for their middleware support). The final decision was to use Express in away that bridges the gap by addressing the gaps identified above as follows:

Deep dive into LoopBack 4 extensibility

There are several key pillars to make extensibility a reality for LoopBack 4.

Rebuilding LoopBack experience on top of the new core

With the extensible foundation in place, we start to rebuild the LoopBack RESTAPI experience by “eating your own dog food” with the following artifacts:

Example for application developers

Before we go further, let’s try to build a ‘hello world’ application withLoopBack 4.

Basic Hello-World

@loopback/example-hello-world

Intermediate example

@loopback/example-todo

Example for extension developers

Learn from existing ones

References