I had misread the code in issue #2200 -- the per-app hooks are still
run, just not the umbrella-level hooks.

This commit switches the relx dependency to branch 4.0.0. It is the
first iteration of integration with relx 4.0 which adds many changes
to how relx works and how the two integrate.

rebar_relx module now creates a relx state from the relx configuration
and pass it along with all the OTP applications rebar3 knows about
which may be used in the release. This allows relx to not have
to duplicate the effort of application discovery like before.

Since this is intended to be released only when OTP-23 is released
the testing of OTP-18 has been removed.

The propagation was confusing source files and artifacts; the artifact
ordering was flipped, and the tagging non-mandatory (aside from as
edges), which made things hard to identify when plugins for compilers
are used.

This patch enables two features:
 * in rebar.config, {diazlyer, [{plt_apps, all_apps}]} makes rebar3 to
   add all project files into PLT for subsequent analysis
 * rebar3 ct --app=myapp allows single application success typing

This allows workflow suitable for large codebases:
 * "rebar3 ct dialyzer --succ-typings=false" to build PLT with all apps
 * "rebar3 ct dialyzer --app=myapp" to perform success typing of myapp

As suggested in #2223, `make_absolute_path/2` and `make_normalized_path/2`
should, instead of being exported in `rebar_dir`, be moved to
`rebar_file_utils` and renamed (decided to simply remove the `make`
from their names since in that module there is already a path related
function called `canonical_path`).

This allows to do quicker re-compile option validation by not requiring
to access the disk and check all candidate erlang files for option
changes. This also opens up the way to compilers that produce more than
one artifact per file.

- Normalize path behaviour to always be "/" across versions
- Normalize port visibility behaviour to match input string when
  appending
- restore rebar_utils function to avoid breaking random plugins that
  peek into our libs for their stuff
- Support URIOpts equivalent and protocol-based overrides; the behaviour
  there is not necessarily consistent across versions in terms of what
  ports are returned (http_uri always returned a port but uri_string
  only returns it if explicitly specified -- we choose the latter
  behaviour on newer releases), but the calls work to ensure URI parsing
  consistently works across versions

The latter judgement call is a bit of an odd one; for consistency we
could always mandate ports, but this would come at a performance penalty
when appending URL paths (i.e. when fetching packages).

The reason for this is that we standardize on the new uri_string
behaviour for path appending (if the port wasn't specified, we don't add
it), and with the http_uri rules, we need to do a kind of parsing round
that checks if the port was included or not to make it equivalent. This
is costly, and _not_ returning the port when it isn't specified lets us
do this transparently.

This allows to maintain backwards compat in older append functions, and
nobody aside from us currently uses the new rebar_uri module so we can
decide to introduce this potential inconsistency if we wish to.

(cherry picked from commit fa577bc46e)

Attempts to bridge http_uri (deprecated in OTP 23)
and uri_string (available as of OTP 21+).

(cherry picked from commit 7057cbe281)

While leaving the old one in place, prep the ground for new analysis
phases for the DAG work. The new DAG functions are added, but not hooked
in yet.

Fixes to the EPP handling have also been added due to issues in
resolving include_lib information

This commit is a transition point that makes some assumptions about new
callbacks for the compiler modules, which will likely not hold when it
comes to making epp be able to read and handle multiple applications at
once (so it can resolve parse transforms and behaviours across OTP
apps).

As such, a follow-up commit is going to be completing that one and
changing its API in incompatible ways; if you find this commit during a
bisect, it's probably not a good one in which to run a bisection.

- Vendor in hex_core at v0.6.8
 - Rename checksum to inner_checksum and deprecate in favor of new outer_checksum key/val
 - Change warn_vsn_once/0 to warn_vsn_once/1 so we can give a proper error message and
   take appropriate action.
 - Update rebar.lock to 1.2.0 format with continued support third-party
   that still may make use of inner checksum via continued persistence of
   inner checksums.
 - Changed app info tuple (package representation) to support inner and
   outer hashes for backwards compat support (see above)

(cherry picked from commit fa577bc46e)

Attempts to bridge http_uri (deprecated in OTP 23)
and uri_string (available as of OTP 21+).

(cherry picked from commit 7057cbe281)

This is a new way to orient the code and would allow better
parallelization or multi-app scan phases

This allows breaking apart the pre-hooks from the rest of the
compilation steps, as a preliminary step towards being able to do some
analysis on all project apps at once before actually compiling them.

This will allow project with larger dependencies sets to clean only the
apps they want to when testing or changing small things, rather than
forcing a rebuild of the whole dep set.

Also allows cleaning up apps, not just deps.

We never really need that and nobody would expect it either. We're
rendering normal files, not HTML.

When in umbrella mode (or through multiple profiles), users can specify
macros for EDocs based on either the {def, ...} or the {macros, ...}
arguments.

This patch replaces the prior options merging for umbrellas to use the
rebar3 tup_umerge utils to remove identical duplicates while preserving
correct ordering, and manually merges the {macros, ...} definitions
while ke eping the correct precedence rules since these appear (given
their behaviour) to be all individually extracted and passed as `{d,
...}` to  the compiler so that epp expands them. This compiler
function freaks out on any re-defined macros and explodes.

Do note that the macros with `{def, ...}` are edoc macros and do not
suffer from that issue, safely deduplicating multiple definitions.

Patch up and add tests on #2112

This adds an additional loading and merging of options for EDoc using
the values from the top-level along with those specified in the
rebar.config of an umbrella application.

The app-specific config values are prepended to the global ones; this
can likely cause some problems with manual path handling, but is
unlikely to happen in practice and the rest seems to work fine based on
order

Fixes the issue in #2114

This allows to reduce the number of noise to only checking deps' app
files when they're built, rather than on every run.

Since main apps and checkouts are still compiled every time, the linting
takes place there and then with a higher frequency.

Overrides should apply to a layer below where they are declared. This
patch makes it so if the project root is an application (i.e. it isn't
'root' and therefore not an umbrella project), we omit applying
overrides in rebar_app_discover.

This in turn required changing a bunch of tests, because all the tests
worked with the idea that all overrides applied to all apps to validate
that they get inherited properly. The changes re-structure the cases so
they are written with an umbrella app, demonstrating that the changes
stick.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>

Deps command shows an * if the local state of the dependencies do not
match the config file, highlighting the differences between the lock
file and the config file if there are any.

deps command was printing an * even when deps were updated, this commit
makes it call rebar_fetch:needs_update/2 for each of them

We currently have a problem in the handling of locks in the `upgrade
<app>` command when a transitive dependency switches supervision trees.

If we start with:

      A
     / \
    B1  C
        |
        D
        |
        E

And upgrade to:

      A
     / \
    B2  C
    |   |
    E   D

by calling `rebar3 upgrade b`, we would expect the ugprade command to do
so directly. However we are currently left with the following lock file:

      A
     / \
    B2  C
        |
        D
        |
        E

This is _not_ critical since on the next run, the lock file is fixed
automatically through an install-deps sequence. However, this is a
jarring state in which to leave a project.

The reason is that we proceed the following way to handle an upgrade:

1. pick the name of the chosen app, and unlock it if it's at the level 0
2. grab a list of all its current children (and their own) and unlock
   them
3. do the same for all the other unlocked dependencies
4. pass this new lock internal format to the get-deps provider in
   upgrade mode
5. take the result and save it to the lock file.

What's interesting is that right now this yields invalid locks at
step 3, because we have removed the lock on B1, but not on E
(which we don't yet know is going to be a new child of the next
B version until we fetch it)

In step 4, we fetch dependencies, and in there we compare the
pre-upgrade locks and fetch structure with those expected in the current
lock state, and rebuild everything.

The fix in this commit adds a check there where if the app is in the
current lock set at a level deeper than the current one, the lock is
forced to be refreshed as if it were not there. This lock would get
refreshed on a blank run anyway since the current lock set would start
empty (rather than pre-populated by `upgrade`), and the level-order
traversal ensures that the locks remain safe in all cases.

Fixes #2030

- vendor in hex_core at v0.5.0
  - Change where repo_name should be the org and not the parent
  - Changed rebar_utils:url_append_path/2 to not explicitly add a '?',
    this is returned in the Query chunk by http_uri:parse/1 (e.g., "?foo=bar")
  - update organization_merging test to expect the sub-repo as the repo_name
  - Add tests for rebar_utils:url_append_path/2
  - Stop referencing/setting "organization" in config and use new
     organization settings (api_repository and repo_organization)
  - Do not set (assume) the read key is valid for any/every repo
  - Set repo_organization and api_repository to org
  - Update tests to check for new config opts

- Modified rebar_hex_repos:auth_config/1 to abort on error except in
 the case of enoent, in which we return an empty map.
 - added auth_config_errors test to test/rebar_pkg_repos_SUITE

This change fixes cases where changes in .hrl files would not be picked
up in .erl files that are in extra source directories (such as those
defined with `extra_src_dirs` or modules in the test/ directory during a
CT or Eunit run).

The problem was due to the way the Directed Acyclic Graph (DAG) of
dependencies between files was being loaded and stored by the compiler
modules.

Prior to this fix, a single DAG would be used for all runs. On a regular
run, the prior DAG is loaded from disk, re-checked, and if changed, it
would get re-written to disk with the changes deciding what to
re-compile. However, whenever extra source directories were specified, a
second run would be done which swaps target directories around in the
compiler modules.

Bug 1: this second run was done without properly tracking the private .hrl
files (in src/), so the changes were invisible. This has been fixed by
re-adding the paths.

The problem is that the DAG handling is self-contained; just invoking it
was sufficient to get it loaded and rewritten to disk. But since runs
with extra src dirs were done on different sets, the compilation of
extra src dirs would be done with bad historical data (all the modules
in src/ are dropped, all those in test/ are re-added); this DAG was then
written to disk once again, polluting the next non-extra run.

This is bug 2, and it is fixed by adding an optional label to each run
so that a regular or extra compile round can be distinguished, each
tracking their own files in their own DAG.

A single test (and a lot of diffing) were sufficient for this.

consider A-MIB imports from B-MIB:

rebar compile MUST FAIL if mib_first_files is ["mibs/A-MIB.mib"]
rebar compile MUST SUCCEED if mib_first_files is ["mibs/B-MIB.mib"]