Rather than doing a check on each single file in a dep set (which can
grow quite large), do a dynamic rebuild of deps based on whether the DAG
mentions the files have been modified. To avoid reloading the whole DAG
more than once and then having to still map files, the compiler version
is added to rebar_compiler_erl's CritMeta for the DAG.

This comparison is only set in place to impact the
rebar_prv_install_deps call that handles pruncing and culling deps by
the compiler provider. Intra-DAG comparisons still need to be done by
rebar_compiler_erl for things to work, and the hope is that since most
built-in compiler mods end up generating erl files in deps, that will be
transparent enough to work well.

A DAG call to check the status of the dag (vsn, critmeta, existing,
etc.) has been added to do the faster check there and allowing proper
dispatch.

Plugins couldn't be updated because they don't really use the standard
dep-style facilities for things due to an early filtering.

Completes the fix presented in #2381

Uses an existing test to simply inject the regression.

This was due to a subtle mistake in the new code in
https://github.com/erlang/rebar3/pull/2322 which optimized the search
behaviour of files not found by EPP. The patch mistakenly always kept
the top-level src/ directory for an app even if it did a lookup in
recursive ones.

This patch tracks and maintains the proper subdirectory hierarchy. To
keep performance adequate, the lists:keyfind/3 function is used, which
is also now a NIF to the same extent as lists:member/2; manual lookups
would likely end up reducing performance, particularly in deep
hierarchies.

A test case has been added to track the regression.
Reported by @elbrujohalcon

This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:

1. The src directories remain recursive. We turned it on by accident at
   some point in the past and now people rely on it, so we're stuck with
   it. However a new test ensures that the feature can be turned off
   on-demand as documented on the website.

2. The test directories are no longer recursive by default. The fix is
   done by properly fixing how rebar3.erl does its feature injection by
   mandating the default value there.
   I'm somewhat nervous that this change could negatively impact some
   users and older compiler module versions, but if users stick to the
   rebar_dir interface, everything should keep working transparently.

3. The test directories' configuration is no longer silently dropped.
   Due to how rebar3.erl injected test state without looking for what
   the user may have specified, multiple extra_src_dirs entries existed
   at once and were run; one with the recursion set to true and one with
   whatever the user specified. If the user disabled recursion of the
   "test" extra_src_dir, then the injected value still ran it once...

4. The handling of extra files in the compiler module is fixed to
   actually use the rebar_dir interface properly, and reinjects
   non-default directory recursion settings into the swapped options
   for the shimmed extra apps. Not doing this annotation step resulted
   in the write for swapped opts to actually drop the configured
   recursion value and make everything recursive all the time.

A single new test actually validates all of that behaviour and seems to
work fine.

Starting with OTP-22, Erlang started changing how volume names are handled in
windows: filename:join/1 drops drive letters, filename:split/2 and
file:absname/1 use c:/ as a form instead of c:\\

We adjust the file_utils and their tests accordingly.

Rebar3.14 has new tests that were not windows ready and those are fixed.

Relx has two failing tests that are not covered in this branch.

Tested in Windows 10 with Powershell.

All this ever checked was that the app was created in _checkouts
by the test code. Not really useful.

The propagation was confusing source files and artifacts; the artifact
ordering was flipped, and the tagging non-mandatory (aside from as
edges), which made things hard to identify when plugins for compilers
are used.

This allows to do quicker re-compile option validation by not requiring
to access the disk and check all candidate erlang files for option
changes. This also opens up the way to compilers that produce more than
one artifact per file.

While leaving the old one in place, prep the ground for new analysis
phases for the DAG work. The new DAG functions are added, but not hooked
in yet.

Fixes to the EPP handling have also been added due to issues in
resolving include_lib information

This is a new way to orient the code and would allow better
parallelization or multi-app scan phases

This allows breaking apart the pre-hooks from the rest of the
compilation steps, as a preliminary step towards being able to do some
analysis on all project apps at once before actually compiling them.

This will allow project with larger dependencies sets to clean only the
apps they want to when testing or changing small things, rather than
forcing a rebuild of the whole dep set.

Also allows cleaning up apps, not just deps.

This allows to reduce the number of noise to only checking deps' app
files when they're built, rather than on every run.

Since main apps and checkouts are still compiled every time, the linting
takes place there and then with a higher frequency.

Overrides should apply to a layer below where they are declared. This
patch makes it so if the project root is an application (i.e. it isn't
'root' and therefore not an umbrella project), we omit applying
overrides in rebar_app_discover.

This in turn required changing a bunch of tests, because all the tests
worked with the idea that all overrides applied to all apps to validate
that they get inherited properly. The changes re-structure the cases so
they are written with an umbrella app, demonstrating that the changes
stick.

This change fixes cases where changes in .hrl files would not be picked
up in .erl files that are in extra source directories (such as those
defined with `extra_src_dirs` or modules in the test/ directory during a
CT or Eunit run).

The problem was due to the way the Directed Acyclic Graph (DAG) of
dependencies between files was being loaded and stored by the compiler
modules.

Prior to this fix, a single DAG would be used for all runs. On a regular
run, the prior DAG is loaded from disk, re-checked, and if changed, it
would get re-written to disk with the changes deciding what to
re-compile. However, whenever extra source directories were specified, a
second run would be done which swaps target directories around in the
compiler modules.

Bug 1: this second run was done without properly tracking the private .hrl
files (in src/), so the changes were invisible. This has been fixed by
re-adding the paths.

The problem is that the DAG handling is self-contained; just invoking it
was sufficient to get it loaded and rewritten to disk. But since runs
with extra src dirs were done on different sets, the compilation of
extra src dirs would be done with bad historical data (all the modules
in src/ are dropped, all those in test/ are re-added); this DAG was then
written to disk once again, polluting the next non-extra run.

This is bug 2, and it is fixed by adding an optional label to each run
so that a regular or extra compile round can be distinguished, each
tracking their own files in their own DAG.

A single test (and a lot of diffing) were sufficient for this.

consider A-MIB imports from B-MIB:

rebar compile MUST FAIL if mib_first_files is ["mibs/A-MIB.mib"]
rebar compile MUST SUCCEED if mib_first_files is ["mibs/B-MIB.mib"]

Move path management out of rebar_utils manual code path function
handling (which we leave there for backwards compat), and centralize
them to allow easier coordination of paths between plugins and deps.

On top of path handling, do a check of loaded modules to only purge and
reload those that actually need it done in order to prevent all kinds of
weird interaction and accidental purge kills. It also allows the
possible cohabitation of both at once, with a "in case of conflict pick
X" as a policy

Changing path handling in providers also highlighted a bunch of bugs in
some tests and appears to fix some in other providers, specifically
around plugins.

When REBAR_CONFIG was set it would not effect the top level app's
configuration because app_discover was rereading the top level
rebar.config which ignored REBAR_CONFIG. Instead this patch has
it use the existing configuration from REBAR_CONFIG.

* update to hex_core for hex-v2 repo support (#1865)

* update to hex_core for hex-v2 repo support

This patch adds only single repo hex-v2 support through hex_core.
Packages no longer filtered out by buildtool metadata and the
package index is updated per-package instead of fetched as one
large ets dump.

* tell travis to also build hex_core branch

* support list of repos for hex packages (#1866)

* support list of repos for hex packages

repos are defined under the hex key in rebar configs. They can be
defined at the top level of a project or globally, but not in
profiles and the repos configured in dependencies are also ignored.

Searching for packages involves first checking for a match in the
local repo index cache, in the order repos are defined. If not found
each repo is checked through the hex api for any known versions of
the package and the first repo with a version that fits the constraint
is used.

* add {repos, replace, []} for overriding the global & default repos

* add hex auth handling for repos (#1874)

auth token are kept in a hex.config file that is modified by the
rebar3 hex plugin.

Repo names that have a : separating a parent and child are considered
organizations. The parent repo's auth will be included with the child.
So an organization named hexpm:rebar3_test will include any hexpm
auth tokens found in the rebar3_test organization's configuration.

* move packages to top level of of hexpm cache dir (#1876)

* move packages to top level of of hexpm cache dir

* append organization name to parent's repo_url when parsing repos

* only eval config scripts and apply overrides once per app (#1879)

* only eval config scripts and apply overrides once per app

* move new resource behaviour to rebar_resource_v2 and keep v1

* cleanup use of rebar_resource module and unused functions

* cleanup error messages and unused code

* when discovering apps support mix packages as unbuilt apps (#1882)

* use hex_core tarball unpacking support in pkg resource (#1883)

* use hex_core tarball unpacking support in pkg resource

* ignore etag if package doesn't exist and delete if checksum fails

* add back tests for bad package checksums

* improve bad registry checksum error message

The compiling of OTP applications is done by first topographically
sorting them according to their dependencies, deps-first. This allows
all compilation to take place in order. In the current code, the same
logic extends to top-level applications in an umbrella project.

Unfortunately, there are cases where this is not going to be true: when
an application has extra_src_dirs entries (or additional directories or
files) to conditionally compile under some profiles, it may start
depending on another top-level application dedicated to that profile for
include files.

However, such an app will never make it to production and neither will
the compilation artifacts that create the dependency. Under that
scenario, current rebar3 is unusable.

This patch makes it so that the compilation provider instead changes the
logic for top-level apps: rather than copying their directories one by
one and compiling them in order, it:

1. copies all top-level apps to the build directory so the files are in
   their proper locations
2. adds the top-level apps to the path (after the global hooks have run,
   so the existing scope and env has not changed)
3. runs the compilation as usual.

Fixes #1651

on 19.x forward the compiler should now take into consideration the value
of the environment variable `ERL_COMPILER_OPTIONS` when deciding whether
or not to recompile a module

The option {recursive,boolean()} can now be set pr directory in
'src_dirs' and 'extra_src_dirs', and on top level in the new
'erlc_compiler' option. Example config:

{erlc_compiler,[{recursive,false}]}.
{src_dirs,[{"src",[{recursive,true}]}]}.

This will cause recursive compilation within the "src" directory, but
not in any other directoires.

When an include file is set in a private path (i.e. src/), the rebar3
compiler would not add them to the {i, Path} params -- only include/ and
the project root were being added.

This meant that when some extra source directories were added to the
compile job, such as test/ when running under the test profile, the
private include paths could not be shared with the test module.

This patch fixes the issues (and adds tests) for such a specific case by
adding all the configured include paths to the {i, Path} erl_opts
arguments, yielding successful compile runs.

there's no way to detect which files actually rely on a parse transform
passed to the compiler via the options (as opposed to `-compile(..)`
so if any parse transforms are in modules that need recompiling just
recompile the world

fixes #1328

changed include files were not properly picked up by `erlc_compiler`
in cases where they were in directories relative to the application
source and not the current working dir of rebar3

fixes #1199

partially addresses #1184

Several projects use an include path relative
to the project's root.
file:compile will look in three places for the include
files:
The current working directory
The directory where the module is being compiled
The directories given by the include option