DC

Create a monorepo with Lerna

Posted on
Tags: monorepo , lerna
Originally published at medium.com

tl;dr

If you want to see the final repository, take a look at it here.

Introduction

Organizing code has become a difficult task. Monolithic applications are becoming difficult to scale, and releases are beginning to crumble under their architecture.

This is how not only the deployment process evolved, but also the organization of the code.

IMHO: Monolithic applications are not bad, and the monorepo approach is not new. The way we create and manage projects started to change somewhere as we started to use more and more micro-services. From that point of view, you can also organize your code into separate packages and have a nice way to put everything back together.

In this blog post, we’ll create a starter repository for Node.js with some tools, and you’ll understand how things work in between.

Requirements

I use Microsoft Windows for development. You should be able to run everything on a Mac or Linux.

In this Post we will use following tools:

and these npm packages:

Before you start

As a software engineer, you should first consider whether this approach is appropriate for the project. If you don’t like this type of architecture, you can split the packages you have in the monorepo and develop them in a separate repository.

I tried to make all steps easy & understandable. But, I recommend reading the whole article before starting coding.

Lerna

Monorepos are not easy to handle. There are issues as dependency management, execution of tasks throughout the project, the build and pipeline process can change over time, and so on. In these cases, you can be sure that something might go wrong or even break on the way.

To fix a lot of problems for us, we can use Lerna for:

Take a look at their repository to understand what it does in detail and how it can help you.

A note to package managers: you cannot use multiple package managers for different packages within the repository. This means that you will not be able to run npm and yarn for the entire repository.

Quick start: Create a test repository with Lerna

Before we start the example project let’s experiment a little with the Lerna CLI. Execute the following commands:

npm install --global lerna
git init test-lerna && cd test-lerna
lerna init
echo "node_modules" > .gitignore

I will explain what each line does:

  1. Install Lerna CLI globally
  2. Create a folder for the repository and move into it
  3. Initialize Lerna, it will create the required standard files for us
  4. Add a .gitignore file which ignores the node_modules folders in the repository

We will have a simple root folder with the following files and folders:

packages/
.gitignore
lerna.json
package.json

You have created your monorepo. Congratulations!

Initial files

Let’s now look in detail about the structure:

In the next parts I will explain a bit about how Lerna works and what we will use for the example so that you understand what we do.

Getting started with Lerna

There are two ways to start with Lerna:

@lerna/init

To use the lerna init command, you must first install the CLI globally or use npx. We will use the global installed CLI.

In this case, Lerna runs with its default behaviour: the same version in all packages. This is useful if you only use one project with several packages.

There are two other options you can check out in the @lerna/init readme file.

You can see the lerna.json file with the following content:

{
	"packages": ["packages/*"],
	"version": "0.0.0"
}

You can add folders other than packages/* to the configuration if you wish: src/*, resources/*, etc.

Bootstrap

The command lerna bootstrap installs all dependencies within all packages below the package folder. We will use the default behaviour and npm will be used as the package manager.

Let’s add a package to our test repository:

cd packages/ && mkdir package1 && cd package1/
npm init
lerna add lodash
npx rimraf node_modules package-lock.json
cd ../..

For a clean installation we have removed the folder and files. Now run from the root folder:

lerna bootstrap

Once the command has finished, you will see a package-lock.json file and a node_modules folder in our package.

Hoisting

There is a reason why many developers do not like the monorepo approach because of the disk space and download time. Suppose you have four packages with react as a dependency, then you will download and install it four times. Don’t get fat and move smart!

The --hoist option can help you with this:

lerna bootstrap --hoist

When you run this command, Lerna installs the dependencies from the packages into the root folder. At the same time, the package-lock.json file contains the path to the root folder for the packages, so everything is linked to a node_modules folder, as far as the .bin folder is concerned.

You can read more about the details in the official documentation.

Adding dependencies to packages

In some cases we will install additional dependencies to packages. We have the command for this:

lerna add <package/dependecy>

For more information, please see the official GitHub page for @lerna/add.

This command installs the defined <package/dependecy> in all packages and updates the package.json and package-lock.json files.

To restrict this to a number of packages, you can use the --scope option. For more information about this filter option, see @lerna/filter-options —scope

You can also use lerna add inside the package folders, just like we did before and it will install everything for you.

Running commands

Just like npm run, Lerna offers us something similar: Lerna run. This command executes a command within all packages. If the command is not available in a package, it will not run and remain silent.

Let’s test the following command:

lerna run start --stream

This will run npm run start in all packages where it is available. In some cases it will not work because there is none.

The --stream option is useful because you will see the result of the process. Without it you will not see anything.

Import or add packages to the monorepo

To import existing repositories, you should try the Lerna import command for more details.

To add a new package to the monorepo execute:

lerna create <name>

Now a new package should be available in the packages/ folder.

Additional dependencies on the root folder

There might be some cases that you want to add other useful dependencies for all packages, but then you need to check where exactly you need to install them. Using these tools might be only possible to have it in the root folder. depending on if they are running tools inside or outside the repository.

Here are some tools that should go into the root folder:

Both work at a higher level because they are executed in combination with “Git” commands. The .git folder is also located in the root directory where they should go, and should only be installed once. We will include them in our example project.

The example project

Let’s create a new Lerna repository to have a clean folder. Run these commands from a folder where you want to create the new project (settings for created packages are set to default).

Hint: If you want to use the example repository from before, skip the first two commands and start creating packages.

git init lerna-monorepo-starter && cd lerna-monorepo-starter
lerna init
lerna create package1 --yes && lerna create package2 --yes
echo "node_modules" > .gitignore
npm install

The result is:

node_modules/
packages/
- package1/
  - __tests__/
  - lib/
  - package.json
  - README.md
- package2/
  - __tests__/
  - lib/
  - package.json
  - README.md
.gitignore
lerna.json
package.json

Time to do some preparations on the root and packages package.json files. To make things short, we want to merge these command steps into one:

  1. Install dependencies in root
  2. Install dependencies in all packages with lerna bootstrap
  3. Start the packages in parallel

Consideration on point three if parallel or not: You should check if you are depending on another package and if that needs to be build before, else you can run in parallel.

Root preparation

Here is the code that we will use with the scripts in the package.json file on the root folder.

{
	"name": "lerna-monorepo-starter",
	"private": true,
	"devDependencies": {
		"lerna": "^3.22.1"
	},
	"scripts": {
		"bootstrap": "lerna bootstrap",
		"lint": "lerna run lint --parallel",
		"setup": "npm install && npm run bootstrap",
		"start": "lerna run start --parallel"
	}
}

Let’s go over the script commands step by step:

Add Husky and commitlint to the root folder

We install more packages mentioned before to root and configure them:

npm install --save-dev husky @commitlint/cli @commitlint/config-conventional @commitlint/config-lerna-scopes

Commitlint configuration

Next create two config files at the root folder.

First file is commitlint.config.js:

module.exports = {
	extends: ['@commitlint/config-conventional'],
	rules: {
		'scope-enum': [
			2,
			'always',
			[
				// workspace packages
				'package1',
				'package2',
				'eslint',
				'*',
			],
		],
	},
};

Here we use some predefined configuration and add one rule to it. For more information visit the official commitlint page and rules documentation page.

Husky configuration

As recommended on the page for commitlint you should now also configure Husky in the root folder.

Create the Husky configuration file .huskyrc.json and put in the following content:

{
	"hooks": {
		"commit-msg": "commitlint -E HUSKY_GIT_PARAMS",
		"pre-commit": "lerna run precommit --concurrency 2 --stream",
		"pre-push": "lerna run lint"
	}
}

These require some explanation now at this stage.

You must know that Husky is a Git hook helper. If you do anything with a Git command that matches them in the file, it can mount itself in front of them and execute commands.

let’s check the commands line by line:

The .editorconfig file

Now the root folder is prepared for your packages that we can add. Before we do this let’s add some useful things for IDE and editor configuration.

Here is the content of the .editorconfig file:

root = true

[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = false
insert_final_newline = true

You can learn more about this file at editorconfig.org.

Final root structure

This is how your root folder should look like:

node_modules/
packages/
- package1/
  - index.js
  - package.json
- package2/
  - index.js
  - package.json
.editorconfig
.gitignore
.huskyrc.json
commitlint.config.js
lerna.json
package.json
package-lock.json

Prepare the node packages

For our example, we have two very simple Node.js hello-world packages. Here you see the content for the files:

packages/package1/lib/package1.js:

const http = require('http');

const hostname = 'localhost';
const port = 3001;

const server = http.createServer((req, res) => {
	res.statusCode = 200;
	res.setHeader('Content-Type', 'text/plain');
	res.end('Hello World - Package 1');
});

server.listen(port, hostname, () => {
	console.log(`Server running at http://${hostname}:${port}/`);
});

In this file, we will run a simple Node.js server that is showing a simple content with: Hello World. The same goes for the second package, with a slightly different content output and port to differentiate.

The packages/package1/package.json file:

{
	"name": "package1",
	"version": "0.0.0",
	"description": "",
	"keywords": [],
	"author": "",
	"license": "ISC",
	"private": true,
	"main": "lib/package1.js",
	"directories": {
		"lib": "lib",
		"test": "__tests__"
	},
	"files": ["lib"],
	"scripts": {
		"start": "node lib/package1.js",
		"test": "echo \"Error: run tests from root\" && exit 1"
	}
}

packages/package2/lib/package2.js

const http = require('http');

const hostname = 'localhost';
const port = 3002;

const server = http.createServer((req, res) => {
	res.statusCode = 200;
	res.setHeader('Content-Type', 'text/plain');
	res.end('Hello World - Part 2');
});

server.listen(port, hostname, () => {
	console.log(`Server running at http://${hostname}:${port}/`);
});

packages/package2/package.json

{
	"name": "package2",
	"version": "0.0.0",
	"description": "",
	"keywords": [],
	"author": "",
	"license": "ISC",
	"private": true,
	"main": "lib/package2.js",
	"directories": {
		"lib": "lib",
		"test": "__tests__"
	},
	"files": ["lib"],
	"scripts": {
		"start": "node lib/package2.js",
		"test": "echo \"Error: run tests from root\" && exit 1"
	}
}

Important: The name key includes the unique name of the package, that means no other package can have the same name, even if it is included in another folder name. By that Lerna can check on the name and filter the scope on its commands.

Here is a small simulation to show you the error with hints:

lerna ERR! ENAME Package name "package1" used in multiple packages:
lerna ERR! ENAME        C:\dev\lerna-monorepo-starter\packages\package1
lerna ERR! ENAME        C:\dev\lerna-monorepo-starter\packages\package2

Let’s wrap it up a little:

Lint file changes

What we need now are two packages to get the linting to go in two ways:

  1. Lint over the files inside each package folder
  2. Lint over Git staged files to have a check on files before they are committed.

Add ESLint and lint-staged to all packages. In this case, we will go to the root folder in the terminal and run the following command:

lerna add eslint --scope=package* --dev && lerna add lint-staged --scope=package* --dev

Before you get going you might question why to install these dependecies in the package folder and not at the root folder. The explanation is quite simple: You want to make the packages independent from the outside of its folder as much as possible. In that way just put all the packages into the package folder as long as it is not depending on tools outside of the monorepo (like Git).

Let’s check the package.json files for the first and second package. You will find the updates now included and installed:

{
  ...
  "devDependencies": {
    "eslint": "^7.5.0",
    "lint-staged": "^10.2.11"
  }
  ...
}

Preparations are done. Don’t forget to save your game.

Configure ESLint

Now we configure ESLint and add one npm script to the package package.json files. Here we start with packages/package1/.eslintrc.js:

module.exports = {
	env: {
		browser: true,
		es6: true,
		node: true,
	},
	parserOptions: {
		ecmaVersion: 2019,
		sourceType: 'module',
	},
	rules: {
		indent: ['error', 2],
		'linebreak-style': ['error', 'unix'],
		quotes: ['error', 'single'],
		semi: ['error', 'always'],
	},
};

Nothing dramatic for a starter. Check out the rules from ESLint in their documentation.

Copy this file to packages/package2/ folder and add the lint script to both package.json files. Your scripts block looks like this now:

packages/package1/package.json:

{
  ...
  "scripts": {
    "lint": "eslint -f codeframe \"**/*.js\"",
    "start": "node lib/package1.js",
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
  ...
}

packages/package2/package.json:

{
  ...
  "scripts": {
    "lint": "eslint -f codeframe \"**/*.js\"",
    "start": "node lib/package2.js",
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
  ...
}

Run the lint command from the root folder now:

npm run lint

You get errors because of the indent rule. Fix the files and try it again until there is no error. If you don’t get errors, then most likely your IDE/editor has fixed it for you automatically.

Configure lint-staged

ESLint can lint all files if you run npm run lint from within the package folder at this moment, so why do we use lint-staged?

It is nice to check changed files before committing them rather than linting everything that can become time-consuming. That is why we want to hook into git commit and check the files with error before we include them into our branch. The positive effect is: the linter run is shorter, as it does not check every file.

This way we will add this functionality with a little go around for the monorepo. Let’s look at what we prepared in .huskyrc.json:

"pre-commit": "lerna run precommit --concurrency 2 --stream",

This hook command is executed before a git commit. Do you see that precommit command? We need to add this npm script to the packages now:

packages/package1/package.json

{
  ...
  "scripts": {
    "precommit": "lint-staged",
    "lint": "eslint -f codeframe \"**/*.js\"",
    "start": "node lib/package1.js",
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
  ...
}

packages/package2/package.json

{
  ...
  "scripts": {
    "precommit": "lint-staged",
    "lint": "eslint -f codeframe \"**/*.js\"",
    "start": "node lib/package2.js",
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
  ...
}

The commands are set, but the configuration for lint-staged is missing, so we add a .lintstagedrc.js file in all the packages with the following content:

module.exports = {
	'*.js': 'eslint',
};

This tells lint-staged to run on following glob rule of files: all files with extension .js with the eslint command. You can play with it later to modify it for your use case.

Done. Everything is prepared.

Check how everything is running

To see if everything is working you can try the following steps:

There is a reason why there is a check on the pre-push on all the files: In a complex codebase the linting for single files might be fine, but that does not mean the changes are fine for other files. Before that, we need to check everything automatically and in case fix the issue before we push the changes.

ESLint rules stand-alone package

Now let’s say you want to share the same configuration for ESLint in all the packages. Right now, we have the same configuration twice, but we can do better than that. Let’s extend the rules by another ruleset package from within our monorepo.

To create a new package in the monorepo, run this from the root folder:

lerna create eslint-config --yes
cd packages/eslint-config
npx rimraf __tests__ lib README.md
touch index.js

Your new package should look like this now:

packages/
- eslint-config/
  - index.js
  - package.json

Copy the content from packages/package1/.eslintrc.js and paste it into the new packages/eslint-config/index.js file.

Update the packages/eslint-config/package.json file:

{
	"name": "@shared/eslint-config",
	"version": "0.0.0",
	"description": "Shareable ESLint configuration",
	"keywords": [],
	"author": "",
	"license": "ISC",
	"private": true,
	"main": "index.js",
	"scripts": {
		"test": "echo \"Error: run tests from root\" && exit 1"
	}
}

Add the package to the other packages by running lerna add from the root folder:

lerna add @shared/eslint-config --scope=package* --dev

You can see, that we are using the package name @shared/eslint-config and also scope with package*. Here we are using one package from inside the monorepo and install it under devDependecies of the packages in the scope, in our case in package1 and package2.

To test this out you need to replace now the ESLint configuration files in the other packages with following content:

packages/package1/.eslintrc.js

module.exports = {
	extends: ['@shared/eslint-config'],
};

packages/package2/.eslintrc.js

module.exports = {
	extends: ['@shared/eslint-config'],
};

You are extending your configuration with the content from the shared configuration package named by its package name.

Here you see the final structure with all files and folders from the example:

node_modules/
packages/
- eslint-config/
  - index.js
  - package.json
- packages1/
  - __tests__
  - lib
  - node_modules/
  - .eslintrc.js
  - .lintstagedrc.js
  - package.json
  - package-lock.json
  - README.md
- packages2/
  - __tests__
  - lib
  - node_modules/
  - .eslintrc.js
  - .lintstagedrc.js
  - package.json
  - package-lock.json
  - README.md
.editorconfig
.gitignore
.huskyrc.json
commitlint.config.js
lerna.json
package.json
package-lock.json

Bonus: Visual Studio Code configuration for monorepo with Multi-root Workspaces

Visual Studio Code has sometimes issues with monorepos in a way that it gets tools/extensions misconfigured. In some cases, the linting or testing will fail internally in your code editor because of the monorepo structure and that becomes a problem. This is why Multi-root Workspaces are the perfect way to handle monorepos. Here I will show you the example file for the repository.

Let’s create the workspaces: We need a .code-workspace file in the root folder:

touch lerna-monorepo-starter.code-workspace

lerna-monorepo-starter.code-workspace:

{
	"folders": [
		{
			"name": "ROOT",
			"path": "."
		},
		{
			"path": "packages/package1"
		},
		{
			"path": "packages/package2"
		},
		{
			"path": "packages/eslint-config"
		}
	],
	"settings": {
		"files.exclude": {
			"**/node_modules/**": true,
			"packages/*/*": true
		}
	}
}

Some explanation:

Sum it up

This example shows you how to create a monorepo mindfully. There are more possibilities that you can use Lerna with like publishing, versioning, cleaning, etc., that we did not cover. You can put this as a challenge for the next steps.

In the example repository, we split up ESLint configuration from all packages and made it reusable. This will also make changes easier in the future, and you are still flexible peer package to modify the rules.

From here you can start trying out more things like adding Angular, React, Svelte or Vue packages and another package that combines all of them in one together. Don’t forget to use Storybook on UI component visualization. For that, you could also create another package and use components from the other packages. The possibilities are endless.

Have a nice time coding.