Package versioning

One of the main jobs of the pub package manageris helping you work with versioning.This document explains a bit about the history of versioning and pub’sapproach to it.Consider this to be advanced information. If you want a better picture of why_pub was designed the way it was, read on. If you just want to _use pub,the other docs will serve you better.

Modern software development, especially web development, leans heavily onreusing lots and lots of existing code. That includes code you wrote in thepast, but also code from third-parties, everything from big frameworks to smallutility libraries. It’s not uncommon for an application to depend ondozens of different packages and libraries.

It’s hard to understate how powerful this is. When you see stories of small webstartups building a site in a few weeks that gets millions of users, theonly reason they can achieve this is because the open source community haslaid a feast of software at their feet.

But this doesn’t come for free: There’s a challenge to codereuse, especially reusing code you don’t maintain. When your app uses codebeing developed by other people, what happens when they change it?They don’t want to break your app, and you certainly don’t either.We solve this problem by versioning.

A name and a number

When you depend on some piece of outside code,you don’t just say “My app uses widgets.” You say, “My app useswidgets 2.0.5.” That combination of name and version number uniquelyidentifies an immutable chunk of code. The people updating widgets canmake all of the changes they want, but they promise to not touch any alreadyreleased versions. They can put out 2.0.6 or 3.0.0 and it won’t affect youone whit because the version you use is unchanged.

When you do want to get those changes, you can always point your app to anewer version of widgets and you don’t have to coordinate with thosedevelopers to do it. However, that doesn’t entirely solve the problem.

Resolving shared dependencies

Depending on specific versions works fine when your dependencygraph is really just a dependency tree. If your app depends on a bunch ofpackages, and those things in turn have their own dependencies and so on, thatall works fine as long as none of those dependencies overlap.

But consider the following example:

dependency graph

So your app uses widgets and templates, and both of those usecollection. This is called a shared dependency. Now what happens whenwidgets wants to use collection 2.3.5 and templates wantscollection 2.3.7? What if they don’t agree on a version?

Unshared libraries (the npm approach)

One option is to just let the app use bothversions of collection. It will have two copies of the library at differentversions and widgets and templates will each get the one they want.

This is what npm does for node.js. Would it work for Dart? Consider thisscenario:

  • collection defines some Dictionary class.
  • widgets gets an instance of it from its copy of collection (2.3.5).It then passes it up to my_app.
  • my_app sends the dictionary over to templates.
  • That in turn sends it down to its version of collection (2.3.7).
  • The method that takes it has a Dictionary type annotation for that object.As far as Dart is concerned, collection 2.3.5 and collection 2.3.7 areentirely unrelated libraries. If you take an instance of class Dictionaryfrom one and pass it to a method in the other, that’s a completely differentDictionary type. That means it will fail to match a Dictionary typeannotation in the receiving library. Oops.

Because of this (and because of the headaches of trying to debug an app thathas multiple versions of things with the same name), we’ve decided npm’s modelisn’t a good fit.

Version lock (the dead end approach)

Instead, when you depend on a package, your app only uses a single copy ofthat package. When you have a shared dependency, everything that depends on ithas to agree on which version to use. If they don’t, you get an error.

That doesn’t actually solve your problem though. When you do get that error,you need to be able to resolve it. So let’s say you’ve gotten yourself intothat situation in the previous example. You want to use widgets andtemplates, but they are using different versions of collection. What doyou do?

The answer is to try to upgrade one of those. templates wantscollection 2.3.7. Is there a later version of widgets that you can upgradeto that works with that version?

In many cases, the answer will be “no”. Look at it from the perspective of thepeople developing widgets. They want to put out a new version with new changesto their code, and they want as many people to be able to upgrade to it it aspossible. If they stick to their current version of collection then anyonewho is using the current version widgets will be able to drop in this new onetoo.

If they were to upgrade their dependency on collection then everyone whoupgrades widgets would have to as well, whether they want to or not.That’s painful, so you end up with a disincentive to upgrade dependencies.That’s called version lock:everyone wants to move their dependencies forward,but no one can take the first step because it forces everyone else to as well.

Version constraints (the Dart approach)

To solve version lock, we loosen the constraints that packages place on theirdependencies. If widgets and templates can both indicate a range ofversions for collection that they work with, then that gives us enoughwiggle room to move our dependencies forward to newer versions. As long as thereis overlap in their ranges, we can still find a single version that makes themboth happy.

This is the model that bundler follows,and is pub’s model too. When you add a dependency in your pubspec,you can specify a range of versions that you can accept.If the pubspec for widgets looked like this:

  1. dependencies:
  2.   collection: '>=2.3.5 <2.4.0'

Then we could pick version 2.3.7 for collection and then both widgetsand templates have their constraints satisfied by a single concrete version.

Semantic versions

When you add a dependency to your package, you’ll sometimes want to specify arange of versions to allow. How do you know what range to pick? You need to beforward compatible, so ideally the range encompasses future versions thathaven’t been released yet. But how do you know your package is going to workwith some new version that doesn’t even exist yet?

To solve that, you need to agree on what a version number means.Imagine that the developers of a package you depend on say,“If we make any backwards incompatible change,then we promise to increment the major version number.”If you trust them, then if you know your package works with 2.3.5 of theirs,you can rely on it working all the way up to 3.0.0.You can set your range like:

  1. dependencies:
  2.   collection: ^2.3.5

Note:This example uses caret syntax to express a range of versions.The string ^2.3.5 means “the range of all versions from 2.3.5to 3.0.0, not including 3.0.0.” For more information, seeCaret syntax.

To make this work, then, we need to come up with that set of promises.Fortunately, other smart people have done the work of figuring this all out andnamed it semantic versioning.

That describes the format of a version number, and the exact API behavioraldifferences when you increment to a later version number. Pub requires versionsto be formatted that way, and to play well with the pub community, your packageshould follow the semantics it specifies. You should assume that the packagesyou depend on also follow it. (And if you find out they don’t, let theirauthors know!)

Although semantic versioning doesn’t promise any compatibility between versionsprior to 1.0.0, the Dart community convention is to treat those versionssemantically as well. The interpretation of each number is just shifted down oneslot: going from 0.1.2 to 0.2.0 indicates a breaking change, going to0.1.3 indicates a new feature, and going to 0.1.2+1 indicates a change thatdoesn’t affect the public API.

We’ve got almost all of the pieces we need to deal with versioning and APIevolution now. Let’s see how they play together and what pub does.

Constraint solving

When you define your package, you list itsimmediate dependencies—thepackages it itself uses. For each one, you specify the range of versions itallows. Each of those dependent packages may in turn have their owndependencies (calledtransitive dependencies. Pubtraverses these and builds up the entire deep dependency graph for your app.

For each package in the graph, pub looks at everything that depends on it. Itgathers together all of their version constraints and tries to simultaneouslysolve them. (Basically, it intersects their ranges.) Then it looks at theactual versions that have been released for that package and selects the best(most recent) one that meets all of those constraints.

For example, let’s say our dependency graph contains collection, and threepackages depend on it. Their version constraints are:

  1. >=1.7.0
  2. ^1.4.0
  3. <1.9.0

The developers of collection have released these versions of it:

  1. 1.7.0
  2. 1.7.1
  3. 1.8.0
  4. 1.8.1
  5. 1.8.2
  6. 1.9.0

The highest version number that fits in all of those ranges is 1.8.2, so pubpicks that. That means your app and every package your app uses will all usecollection 1.8.2.

Constraint context

The fact that selecting a package version takes into account every packagethat depends on it has an important consequence: the specific version thatwill be selected for a package is a global property of the app using thatpackage.

The following example shows what this means. Let’s say we havetwo apps. Here are their pubspecs:

  1. name: my_app
  2. dependencies:
  3.   widgets:
  1. name: other_app
  2. dependencies:
  3.   widgets:
  4.   collection: '<1.5.0'

They both depend on widgets, whose pubspec is:

  1. name: widgets
  2. dependencies:
  3.   collection: '>=1.0.0 <2.0.0'

The other_app package depends directly on collection itself. Theinteresting part is that it happens to have a different version constraint onit than widgets does.

This means that you can’t just look at the widgets package inisolation to figure out what version of collection it will use. It dependson the context. In myapp, widgets will use collection 1.9.9. Butin other_app, widgets will get saddled with collection 1.4.9 because ofthe _other constraint that otherapp places on it.

This is why each app gets its own .packages file: The concrete versionselected for each package depends on the entire dependency graph of thecontaining app.

Constraint solving for exported dependencies

Package authors must define package constraints with care.Consider the following scenario:

dependency graph

The bookshelf package depends on widgets.The widgets package, currently at 1.2.0, exportscollection via export 'package:collection/collection.dart', and isat 2.4.0. The pubspec files are as follows:

  1. name: bookshelf
  2. dependencies:
  3.   widgets:  ^1.2.0
  1. name: widgets
  2. dependencies:
  3.   collection:  ^2.4.0

The collection package is then updated to 2.5.0.The 2.5.0 version of collection includes a new method called sortBackwards().bookshelf may call sortBackwards(),because it’s part of the API exposed by widgets,despite bookshelf having only a transitive dependency on collection.

Because widgets has an API that is not reflected in its version number,the app that uses the bookshelf package and calls sortBackwards() may crash.

Exporting an API causes that API to be treated as if it isdefined in the package itself, but it can’t increase the version number whenthe API adds features. This means that bookshelf has no way of declaringthat it needs a version of widgets that supports sortBackwards().

For this reason, when dealing with exported packages,it’s recommended that the package’s author keeps a tighterlimit on the upper and lower bounds of a dependency.In this case, the range for the widgets package should be narrowed:

  1. name: bookshelf
  2. dependencies:
  3.   widgets:  '>=1.2.0 <1.3.0'
  1. name: widgets
  2. dependencies:
  3.   collection:  '>=2.4.0 <2.5.0'

This translates to a lower bound of 1.2.0 for widgets and 2.4.0for collection.When the 2.5.0 version of collection is released,then widgets is also updated to 1.3.0 and the corresponding constraintsare also updated.

Using this convention ensures that users have the correct version ofboth packages, even if one is not a direct dependency.

Lockfiles

So once pub has solved your app’s version constraints, then what? The endresult is a complete list of every package that your app depends on eitherdirectly or indirectly and the best version of that package that will work withyour app’s constraints.

Pub takes that and writes it out to a lockfile in your app’s directorycalled pubspec.lock. When pub builds the .packages file for your app, ituses the lockfile to know what versions of each package to refer to. (And ifyou’re curious to see what versions it selected, you can read the lockfile tofind out.)

The next important thing pub does is it stops touching the lockfile. Onceyou’ve got a lockfile for your app, pub won’t touch it until you tell it to.This is important. It means you won’t spontaneously start using new versionsof random packages in your app without intending to. Once your app is locked,it stays locked until you manually tell it to update the lockfile.

If your package is for an app, you check your lockfile into your sourcecontrol system! That way, everyone on your team will be usingthe exact same versions of every dependency when they build your app. You’llalso use this when you deploy your app so you can ensure that your productionservers are using the exact same packages that you’re developing with.

When things go wrong

Of course, all of this presumes that your dependency graph is perfect andflawless. Even with version ranges and pub’s constraint solving andsemantic versioning, you can never be entirely spared from thedangers of versionitis.

You might run into one of the following problems:

You can have disjoint constraints

Lets say your app uses widgets andtemplates and both use collection. But widgets asks for a versionof it between 1.0.0 and 2.0.0 and templates wants somethingbetween 3.0.0 and 4.0.0. Those ranges don’t even overlap. There’s nopossible version that would work.

You can have ranges that don’t contain a released version

Let’s say afterputting all of the constraints on a shared dependency together, you’releft with the narrow range of >=1.2.4 <1.2.6. It’s not an empty range.If there was a version 1.2.4 of the dependency, you’d be golden. But maybethey never released that and instead when straight from 1.2.3 to 1.3.0.You’ve got a range but nothing exists inside it.

You can have an unstable graph

This is, by far, the most challenging part ofpub’s version solving process. The process was described as build up thedependency graph and then solve all of the constraints and pick versions.But it doesn’t actually work that way. How could you build up the whole_dependency graph before you’ve picked _any versions? The pubspecsthemselves are version-specific. Different versions of the same packagemay have different sets of dependencies.

As you’re selecting versions of packages, they are changing the shape ofthe dependency graph itself. As the graph changes, that may changeconstraints, which can cause you to select different versions, and then yougo right back around in a circle.

Sometimes this process never settles down into a stable solution.Gaze into the abyss:

  1. name: my_app
  2. version: 0.0.0
  3. dependencies:
  4.   yin: '>=1.0.0'
  1. name: yin
  2. version: 1.0.0
  3. dependencies:
  1. name: yin
  2. version: 2.0.0
  3. dependencies:
  4.   yang: '1.0.0'
  1. name: yang
  2. version: 1.0.0
  3. dependencies:
  4.   yin: '1.0.0'

In all of these cases, there is no set of concrete versions that will work foryour app, and when this happens pub reports an error and tells you what’sgoing on. It definitely won’t leave you in some weird state where youthink things can work but won’t.

Summary

That was a lot of information, but here are the key points:

  • Code reuse is great, but in order to let developers move quickly, packagesneed to be able to evolve independently.
  • Versioning is how you enable that. But depending on single concrete versionsis too precise and with shared dependencies leads to version lock.
  • To cope with that, you depend on ranges of versions. Pub then walksyour dependency graph and picks the best versions for you. If it can’t, ittells you.
  • Once your app has a solid set of versions for its dependencies, that getspinned down in a lockfile. That ensures that every machine your app ison is using the same versions of all of its dependencies.

If you’d like to know more about pub’s version solving algorithm,see the articlePubGrub: Next-Generation Version Solving.