We currently have a problem in the handling of locks in the `upgrade
<app>` command when a transitive dependency switches supervision trees.
If we start with:
A
/ \
B1 C
|
D
|
E
And upgrade to:
A
/ \
B2 C
| |
E D
by calling `rebar3 upgrade b`, we would expect the ugprade command to do
so directly. However we are currently left with the following lock file:
A
/ \
B2 C
|
D
|
E
This is _not_ critical since on the next run, the lock file is fixed
automatically through an install-deps sequence. However, this is a
jarring state in which to leave a project.
The reason is that we proceed the following way to handle an upgrade:
1. pick the name of the chosen app, and unlock it if it's at the level 0
2. grab a list of all its current children (and their own) and unlock
them
3. do the same for all the other unlocked dependencies
4. pass this new lock internal format to the get-deps provider in
upgrade mode
5. take the result and save it to the lock file.
What's interesting is that right now this yields invalid locks at
step 3, because we have removed the lock on B1, but not on E
(which we don't yet know is going to be a new child of the next
B version until we fetch it)
In step 4, we fetch dependencies, and in there we compare the
pre-upgrade locks and fetch structure with those expected in the current
lock state, and rebuild everything.
The fix in this commit adds a check there where if the app is in the
current lock set at a level deeper than the current one, the lock is
forced to be refreshed as if it were not there. This lock would get
refreshed on a blank run anyway since the current lock set would start
empty (rather than pre-populated by `upgrade`), and the level-order
traversal ensures that the locks remain safe in all cases.
Fixes#2030
* update to hex_core for hex-v2 repo support (#1865)
* update to hex_core for hex-v2 repo support
This patch adds only single repo hex-v2 support through hex_core.
Packages no longer filtered out by buildtool metadata and the
package index is updated per-package instead of fetched as one
large ets dump.
* tell travis to also build hex_core branch
* support list of repos for hex packages (#1866)
* support list of repos for hex packages
repos are defined under the hex key in rebar configs. They can be
defined at the top level of a project or globally, but not in
profiles and the repos configured in dependencies are also ignored.
Searching for packages involves first checking for a match in the
local repo index cache, in the order repos are defined. If not found
each repo is checked through the hex api for any known versions of
the package and the first repo with a version that fits the constraint
is used.
* add {repos, replace, []} for overriding the global & default repos
* add hex auth handling for repos (#1874)
auth token are kept in a hex.config file that is modified by the
rebar3 hex plugin.
Repo names that have a : separating a parent and child are considered
organizations. The parent repo's auth will be included with the child.
So an organization named hexpm:rebar3_test will include any hexpm
auth tokens found in the rebar3_test organization's configuration.
* move packages to top level of of hexpm cache dir (#1876)
* move packages to top level of of hexpm cache dir
* append organization name to parent's repo_url when parsing repos
* only eval config scripts and apply overrides once per app (#1879)
* only eval config scripts and apply overrides once per app
* move new resource behaviour to rebar_resource_v2 and keep v1
* cleanup use of rebar_resource module and unused functions
* cleanup error messages and unused code
* when discovering apps support mix packages as unbuilt apps (#1882)
* use hex_core tarball unpacking support in pkg resource (#1883)
* use hex_core tarball unpacking support in pkg resource
* ignore etag if package doesn't exist and delete if checksum fails
* add back tests for bad package checksums
* improve bad registry checksum error message
When fetching deps, if this is a clean repo there will be extensive
messages warning that dependencies which have already been fetched are
being skipped. For large projects being built and tested in a clean
environment this significantly increases the noise level of the build.
This modification adds an additional rebar option
(deps_warning_on_conflict) that will allow disabling these warning
messages. If deps_error_on_conflict is set, an error will still be
thrown. This will not change default behaviour of rebar.
There is a similar outstanding issue:
https://github.com/erlang/rebar3/issues/1105
However this seems to be a push for not outputting warnings when the dep
version is the same, rather than disabling warnings altogether.
This is done through 3 main change groups:
- replacing `~s` by `~ts` in format strings, so that strings that
contain unicode are properly printed rather than crashing
- adding the `unicode` argument to all function of the `re` module to
ensure transformations on strings containing unicode data are valid
instead of crashing (see issue #1302)
- replacing `ec_cnv:to_binary/1` and `ec_cnv:to_list/1` with matching
functions in `rebar_utils`.
The last point has been done, rather than modifying and updating erlware
commons, because binary and list conversions can be a contentious
subject. For example, if what is being handled is actually bytes from a
given binary stream, then forcing a byte-oriented interpretation of the
data can corrupt it. As such, it does not appear safe to modify erlware
commons' conversion functions since it may not be safe for all its
users.
Instead, rebar3 reimplements a subset of them (only converting
atoms and chardata, ignoring numbers) with the explicit purpose of
handling unicode string data.
Tests were left as unchanged as possible. This may impact the ability to
run rebar3's own suites in a unicode path, but respects a principle of
least change for such a large patch.
The use case has been described in issue #1478 where a local application
can exist while being declared as a dependency as well. This allows, for
example, to work on a release where all applications may require to be
published independently, or to provide some form of 'vendoring' with a
local app.
The fix is done by decoupling the dependency source resolution form the
dependency parsing. The reason for this being that the discovery phase
needs to parse apps for their top-level deps, and dep installation needs
to resolve the packages with accuracy. In the current implementation,
both code paths call to the same function.
This patch splits up the precise discovery and makes it happen *only*
when installing dependencies, and only if a top-level app does not
already define the application needing resolving.
One weakness of this fix is that it necessarily breaks cycle detection
in dependencies that involve a root application depending on itself
since its own version as a dep will not be expanded. There appears to be
no possible way to prevent this, but should be rare enough to be worth
the tradeoff for the common case.
Includes improvments and function documentation for all modules (in
alphabetical order) up to rebar_core, and may have included more in
other modules as I saw fit to dig and understand more of the internals.
Instead fetching and resolving src deps (which could depend on pkg deps)
and then pkg deps this patch combines the two into a single set of
iterations by level. The only difference between src and pkg deps in this
new install_deps is how their deps list is found -- from the config or
lock file for src deps and from the neighbors of the vertex for pkg.
Linking message will only be printed the first time the link or copy
actually happens.
Note:
rebar_file_utils:symlink_or_copy will now return 'exists'
instead of ok, if it did nothing because the link exists.
Nothing was checking the return value yet, so seemed reasonable.
Pierre Fenoll
Fred Hebert
Tristan Sloughter
Liam McNamara
pvmart
Richard Jones