Rather than doing a check on each single file in a dep set (which can
grow quite large), do a dynamic rebuild of deps based on whether the DAG
mentions the files have been modified. To avoid reloading the whole DAG
more than once and then having to still map files, the compiler version
is added to rebar_compiler_erl's CritMeta for the DAG.

This comparison is only set in place to impact the
rebar_prv_install_deps call that handles pruncing and culling deps by
the compiler provider. Intra-DAG comparisons still need to be done by
rebar_compiler_erl for things to work, and the hope is that since most
built-in compiler mods end up generating erl files in deps, that will be
transparent enough to work well.

A DAG call to check the status of the dag (vsn, critmeta, existing,
etc.) has been added to do the faster check there and allowing proper
dispatch.

Plugins couldn't be updated because they don't really use the standard
dep-style facilities for things due to an early filtering.

Completes the fix presented in #2381

Uses an existing test to simply inject the regression.

fixes tsloughter/rebar3_hex#184

This commit adds new checks in the `set_paths` test case related to
runtime apps and their path handling with the `runtime` target in
`rebar_paths`. These new checks also hope to increase the coverage
of the changes done to the xref provider in this same PR (#2354).

This commit also adds some new helper functions to help deal with
future changes related to these new checks (i.e., changes related to
the runtime apps), besides fixing a problem when looking for the
first instace of an app/dep/plugin in the path while testing.

This was due to a subtle mistake in the new code in
https://github.com/erlang/rebar3/pull/2322 which optimized the search
behaviour of files not found by EPP. The patch mistakenly always kept
the top-level src/ directory for an app even if it did a lookup in
recursive ones.

This patch tracks and maintains the proper subdirectory hierarchy. To
keep performance adequate, the lists:keyfind/3 function is used, which
is also now a NIF to the same extent as lists:member/2; manual lookups
would likely end up reducing performance, particularly in deep
hierarchies.

A test case has been added to track the regression.
Reported by @elbrujohalcon

this fixes the issue where using 'rebar3 plugins upgrade P'
would result in a plugin's .app file having an empty
modules list. The code this replaces hadn't been touched
since rebar3 3.0.0-beta.3 and a lot of improvements have
gone into dep handling and the compiler since then. This
change should take advantage of those changes.

This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:

1. The src directories remain recursive. We turned it on by accident at
   some point in the past and now people rely on it, so we're stuck with
   it. However a new test ensures that the feature can be turned off
   on-demand as documented on the website.

2. The test directories are no longer recursive by default. The fix is
   done by properly fixing how rebar3.erl does its feature injection by
   mandating the default value there.
   I'm somewhat nervous that this change could negatively impact some
   users and older compiler module versions, but if users stick to the
   rebar_dir interface, everything should keep working transparently.

3. The test directories' configuration is no longer silently dropped.
   Due to how rebar3.erl injected test state without looking for what
   the user may have specified, multiple extra_src_dirs entries existed
   at once and were run; one with the recursion set to true and one with
   whatever the user specified. If the user disabled recursion of the
   "test" extra_src_dir, then the injected value still ran it once...

4. The handling of extra files in the compiler module is fixed to
   actually use the rebar_dir interface properly, and reinjects
   non-default directory recursion settings into the swapped options
   for the shimmed extra apps. Not doing this annotation step resulted
   in the write for swapped opts to actually drop the configured
   recursion value and make everything recursive all the time.

A single new test actually validates all of that behaviour and seems to
work fine.

Starting with OTP-22, Erlang started changing how volume names are handled in
windows: filename:join/1 drops drive letters, filename:split/2 and
file:absname/1 use c:/ as a form instead of c:\\

We adjust the file_utils and their tests accordingly.

Rebar3.14 has new tests that were not windows ready and those are fixed.

Relx has two failing tests that are not covered in this branch.

Tested in Windows 10 with Powershell.

All this ever checked was that the app was created in _checkouts
by the test code. Not really useful.

I had misread the code in issue #2200 -- the per-app hooks are still
run, just not the umbrella-level hooks.

This commit switches the relx dependency to branch 4.0.0. It is the
first iteration of integration with relx 4.0 which adds many changes
to how relx works and how the two integrate.

rebar_relx module now creates a relx state from the relx configuration
and pass it along with all the OTP applications rebar3 knows about
which may be used in the release. This allows relx to not have
to duplicate the effort of application discovery like before.

Since this is intended to be released only when OTP-23 is released
the testing of OTP-18 has been removed.

The propagation was confusing source files and artifacts; the artifact
ordering was flipped, and the tagging non-mandatory (aside from as
edges), which made things hard to identify when plugins for compilers
are used.

This patch enables two features:
 * in rebar.config, {diazlyer, [{plt_apps, all_apps}]} makes rebar3 to
   add all project files into PLT for subsequent analysis
 * rebar3 ct --app=myapp allows single application success typing

This allows workflow suitable for large codebases:
 * "rebar3 ct dialyzer --succ-typings=false" to build PLT with all apps
 * "rebar3 ct dialyzer --app=myapp" to perform success typing of myapp

As suggested in #2223, `make_absolute_path/2` and `make_normalized_path/2`
should, instead of being exported in `rebar_dir`, be moved to
`rebar_file_utils` and renamed (decided to simply remove the `make`
from their names since in that module there is already a path related
function called `canonical_path`).

This allows to do quicker re-compile option validation by not requiring
to access the disk and check all candidate erlang files for option
changes. This also opens up the way to compilers that produce more than
one artifact per file.

- Normalize path behaviour to always be "/" across versions
- Normalize port visibility behaviour to match input string when
  appending
- restore rebar_utils function to avoid breaking random plugins that
  peek into our libs for their stuff
- Support URIOpts equivalent and protocol-based overrides; the behaviour
  there is not necessarily consistent across versions in terms of what
  ports are returned (http_uri always returned a port but uri_string
  only returns it if explicitly specified -- we choose the latter
  behaviour on newer releases), but the calls work to ensure URI parsing
  consistently works across versions

The latter judgement call is a bit of an odd one; for consistency we
could always mandate ports, but this would come at a performance penalty
when appending URL paths (i.e. when fetching packages).

The reason for this is that we standardize on the new uri_string
behaviour for path appending (if the port wasn't specified, we don't add
it), and with the http_uri rules, we need to do a kind of parsing round
that checks if the port was included or not to make it equivalent. This
is costly, and _not_ returning the port when it isn't specified lets us
do this transparently.

This allows to maintain backwards compat in older append functions, and
nobody aside from us currently uses the new rebar_uri module so we can
decide to introduce this potential inconsistency if we wish to.

(cherry picked from commit fa577bc46e)

Attempts to bridge http_uri (deprecated in OTP 23)
and uri_string (available as of OTP 21+).

(cherry picked from commit 7057cbe281)

While leaving the old one in place, prep the ground for new analysis
phases for the DAG work. The new DAG functions are added, but not hooked
in yet.

Fixes to the EPP handling have also been added due to issues in
resolving include_lib information

This commit is a transition point that makes some assumptions about new
callbacks for the compiler modules, which will likely not hold when it
comes to making epp be able to read and handle multiple applications at
once (so it can resolve parse transforms and behaviours across OTP
apps).

As such, a follow-up commit is going to be completing that one and
changing its API in incompatible ways; if you find this commit during a
bisect, it's probably not a good one in which to run a bisection.

- Vendor in hex_core at v0.6.8
 - Rename checksum to inner_checksum and deprecate in favor of new outer_checksum key/val
 - Change warn_vsn_once/0 to warn_vsn_once/1 so we can give a proper error message and
   take appropriate action.
 - Update rebar.lock to 1.2.0 format with continued support third-party
   that still may make use of inner checksum via continued persistence of
   inner checksums.
 - Changed app info tuple (package representation) to support inner and
   outer hashes for backwards compat support (see above)

(cherry picked from commit fa577bc46e)

Attempts to bridge http_uri (deprecated in OTP 23)
and uri_string (available as of OTP 21+).

(cherry picked from commit 7057cbe281)

This is a new way to orient the code and would allow better
parallelization or multi-app scan phases

This allows breaking apart the pre-hooks from the rest of the
compilation steps, as a preliminary step towards being able to do some
analysis on all project apps at once before actually compiling them.

This will allow project with larger dependencies sets to clean only the
apps they want to when testing or changing small things, rather than
forcing a rebuild of the whole dep set.

Also allows cleaning up apps, not just deps.

We never really need that and nobody would expect it either. We're
rendering normal files, not HTML.

When in umbrella mode (or through multiple profiles), users can specify
macros for EDocs based on either the {def, ...} or the {macros, ...}
arguments.

This patch replaces the prior options merging for umbrellas to use the
rebar3 tup_umerge utils to remove identical duplicates while preserving
correct ordering, and manually merges the {macros, ...} definitions
while ke eping the correct precedence rules since these appear (given
their behaviour) to be all individually extracted and passed as `{d,
...}` to  the compiler so that epp expands them. This compiler
function freaks out on any re-defined macros and explodes.

Do note that the macros with `{def, ...}` are edoc macros and do not
suffer from that issue, safely deduplicating multiple definitions.

Patch up and add tests on #2112

This adds an additional loading and merging of options for EDoc using
the values from the top-level along with those specified in the
rebar.config of an umbrella application.

The app-specific config values are prepended to the global ones; this
can likely cause some problems with manual path handling, but is
unlikely to happen in practice and the rest seems to work fine based on
order

Fixes the issue in #2114