The use case has been described in issue #1478 where a local application
can exist while being declared as a dependency as well. This allows, for
example, to work on a release where all applications may require to be
published independently, or to provide some form of 'vendoring' with a
local app.
The fix is done by decoupling the dependency source resolution form the
dependency parsing. The reason for this being that the discovery phase
needs to parse apps for their top-level deps, and dep installation needs
to resolve the packages with accuracy. In the current implementation,
both code paths call to the same function.
This patch splits up the precise discovery and makes it happen *only*
when installing dependencies, and only if a top-level app does not
already define the application needing resolving.
One weakness of this fix is that it necessarily breaks cycle detection
in dependencies that involve a root application depending on itself
since its own version as a dep will not be expanded. There appears to be
no possible way to prevent this, but should be rare enough to be worth
the tradeoff for the common case.
Aliasing only had a bit of ad-hoc support in rebar3, and various issues
have encountered problems related to the package names not mapping
properly with the application name. One such issue is
https://github.com/erlang/rebar3/issues/1290
The problem has been hard to find because it only impacts transitive
dependencies (not top-level ones) of other packages. The root cause for
this is that the application name was not being tracked by rebar3's internal
index, only the package name and its version were.
When a given application was a package app, the data for the application
name would be reconstructed from the lock file, but only if it were a
top-level app or a dependency of a source application where parsing the
lock file is necessary to know what comes next.
When a transitive dependency of a package dependency was fetched, we
instead read its dependencies directly from the in-memory package index
within rebar3. This caused us to only read the package name and version,
and lost all information regarding application name. This worked fine
for most cases since for the vast majority of packages, the package name
matches the app name, but failed for all aliases, which would then be
moved to directories that wouldn't match the app name.
This in turn broke some aspects of code analysis (in Dialyzer), or other
functionality relying on static paths, such as including .hrl files from
dependencies.
This patch reformats the internal storage format of dependencies to
align with the internal one used by rebar3, so that the app name can be
carried along with the package name and its version.
The fix can only work once `rebar3 update` is called so the index is
rebuilt internally, and will the file cached on disk will be
incompatible with older rebar3 versions.
Currently, the following is not covered:
- Tests
- Including the package hashes of dependencies so they may match what
is in a lock file -- they're being `undefined` instead, which may
break some lookups. The previous format did not lend itself to hashing
in the same way, and it is possible transitive deps were not being
tracked properly, or worked by respecting the current package hierarchy.
This will require further analysis
For now this commit can allow reviewing and discussion.
- the internal representation for package locks moves from `{Name, {pkg,
PkgName, Vsn}, Lvl}` to `{Name, {pkg, PkgName, Vsn, Hash}, Lvl}`
- the internal representation for packages moves from `{pkg, PkgName,
Vsn}` to `{pkg, PkgName, Vsn, Hash}`
- the hash can be `undefined`, meaning no check will be done
- no checking is done yet.
Instead fetching and resolving src deps (which could depend on pkg deps)
and then pkg deps this patch combines the two into a single set of
iterations by level. The only difference between src and pkg deps in this
new install_deps is how their deps list is found -- from the config or
lock file for src deps and from the neighbors of the vertex for pkg.
- Adding tests for package deps
- Adding conflict/override warnings for package deps
- Adding cycle detection for packages
- Adding cycle detection for mixed packages+source
- Fixing internal dependency format of package resources when converted
to rebar_app_info
- normalizing level-order prioritization to be based on lexicographical
sort of app names rather than traversal order (which is undefined for
package deps)
- Fixing tests for source deps for deep cycle detection
- Fixing bugs with source deps
- Relaxed version format checks for test utils
A lot of fixes are combined in there because I didn't want to commit
non-passing code, and many bugs were found when adding the second batch
of tests as part of the original effort.
- Adding tests
- fixing use of set fetching to find repeated deps and prevent infinite
loops
On a circular loop rebar3 now fails with `{error, no_sort}`, which is
uncaught and should be handled to consider the issue fully fixed.
- Reworked the helpers for existing suites and expanded them
- Created a mock git resource module to test for its dependency fetching
- Added a test suite for dependency resolving with first checks for
common cases (https://gist.github.com/ferd/197cc5c0b85aae370436)
Left to do would include:
- Verify warnings
- Verify failures
- Verify dependency updates resolving
Fred Hebert
Heinz N. Gies
Tristan Sloughter
CarlosEDP