This change fixes cases where changes in .hrl files would not be picked
up in .erl files that are in extra source directories (such as those
defined with `extra_src_dirs` or modules in the test/ directory during a
CT or Eunit run).

The problem was due to the way the Directed Acyclic Graph (DAG) of
dependencies between files was being loaded and stored by the compiler
modules.

Prior to this fix, a single DAG would be used for all runs. On a regular
run, the prior DAG is loaded from disk, re-checked, and if changed, it
would get re-written to disk with the changes deciding what to
re-compile. However, whenever extra source directories were specified, a
second run would be done which swaps target directories around in the
compiler modules.

Bug 1: this second run was done without properly tracking the private .hrl
files (in src/), so the changes were invisible. This has been fixed by
re-adding the paths.

The problem is that the DAG handling is self-contained; just invoking it
was sufficient to get it loaded and rewritten to disk. But since runs
with extra src dirs were done on different sets, the compilation of
extra src dirs would be done with bad historical data (all the modules
in src/ are dropped, all those in test/ are re-added); this DAG was then
written to disk once again, polluting the next non-extra run.

This is bug 2, and it is fixed by adding an optional label to each run
so that a regular or extra compile round can be distinguished, each
tracking their own files in their own DAG.

A single test (and a lot of diffing) were sufficient for this.

consider A-MIB imports from B-MIB:

rebar compile MUST FAIL if mib_first_files is ["mibs/A-MIB.mib"]
rebar compile MUST SUCCEED if mib_first_files is ["mibs/B-MIB.mib"]

- Added support to rebar_prv_xref for filtering out whole module. A
  module may now be added to the list of xref_ignores in rebar.config or
  via -ignore_xref() module attribute
  - Added unit test for whole module filtering support to xref CT suite

Dollar Variable expansion (`$VAR`) was inadvertently disabled for
windows variables, although %VARIABLES% already worked. This reduced the
portability of hooks in general.

Additionally, tests would fail on windows due to bad quoting of paths:
the path C:/a/b/c would fail when passed to the command
`cmd /q /c C:/a/b/c` because it would interpret /a /b and /c as 3
options. Using quotes makes the tests pass.

- Only set paths that need to be put as a priority
- Clean up paths before leaving API mode

The first point accounted for some performance cost, but the latter one
explains the 40% overhead in test runs: since rebar3 calls rebar3 a lot
with a bunch of fake apps, and that the new mechanism for path handling
by default does not _remove_ paths, it just _orders_ them, we would end
up in a situation where as the tests ran, more and more fake paths would
get added to the VM.

By the time the run was over, all path handling would take longer since
more paths needed filtering every time. By resetting paths at the end of
an API run, we prevent a given 'project' from polluting another one's
runtime and performance once the API successfully returns.

Some finishing touch to that code

Move path management out of rebar_utils manual code path function
handling (which we leave there for backwards compat), and centralize
them to allow easier coordination of paths between plugins and deps.

On top of path handling, do a check of loaded modules to only purge and
reload those that actually need it done in order to prevent all kinds of
weird interaction and accidental purge kills. It also allows the
possible cohabitation of both at once, with a "in case of conflict pick
X" as a policy

Changing path handling in providers also highlighted a bunch of bugs in
some tests and appears to fix some in other providers, specifically
around plugins.

* add compile type for dynamic project compilation

* new rebar_compiler abstraction for running multiple compilers

rebar_compiler is a new behaviour that a plugin can implement to
be called on any ues of the compile provider to compile source
files and keep track of their dependencies.

* fix check that modules in .app modules list are from src_dirs

* use project_type to find module for building projects

* allow plugins to add project builders and compilers

retired packages are now used the same as any other but
a warning will be printed when it is resolved.

prerelease versions are skipped unless explicitly given
as the version in the constraint or lock file.

When REBAR_CONFIG was set it would not effect the top level app's
configuration because app_discover was rereading the top level
rebar.config which ignored REBAR_CONFIG. Instead this patch has
it use the existing configuration from REBAR_CONFIG.

* update to hex_core for hex-v2 repo support (#1865)

* update to hex_core for hex-v2 repo support

This patch adds only single repo hex-v2 support through hex_core.
Packages no longer filtered out by buildtool metadata and the
package index is updated per-package instead of fetched as one
large ets dump.

* tell travis to also build hex_core branch

* support list of repos for hex packages (#1866)

* support list of repos for hex packages

repos are defined under the hex key in rebar configs. They can be
defined at the top level of a project or globally, but not in
profiles and the repos configured in dependencies are also ignored.

Searching for packages involves first checking for a match in the
local repo index cache, in the order repos are defined. If not found
each repo is checked through the hex api for any known versions of
the package and the first repo with a version that fits the constraint
is used.

* add {repos, replace, []} for overriding the global & default repos

* add hex auth handling for repos (#1874)

auth token are kept in a hex.config file that is modified by the
rebar3 hex plugin.

Repo names that have a : separating a parent and child are considered
organizations. The parent repo's auth will be included with the child.
So an organization named hexpm:rebar3_test will include any hexpm
auth tokens found in the rebar3_test organization's configuration.

* move packages to top level of of hexpm cache dir (#1876)

* move packages to top level of of hexpm cache dir

* append organization name to parent's repo_url when parsing repos

* only eval config scripts and apply overrides once per app (#1879)

* only eval config scripts and apply overrides once per app

* move new resource behaviour to rebar_resource_v2 and keep v1

* cleanup use of rebar_resource module and unused functions

* cleanup error messages and unused code

* when discovering apps support mix packages as unbuilt apps (#1882)

* use hex_core tarball unpacking support in pkg resource (#1883)

* use hex_core tarball unpacking support in pkg resource

* ignore etag if package doesn't exist and delete if checksum fails

* add back tests for bad package checksums

* improve bad registry checksum error message

- resolves #1645

This would cause crashes on linux and force people to have a src_dirs
config that is strictly matching what is on the file system rather than
acting as a specification of those that are valid.

To compare, if lib_dirs worked the same, then any repo that did not both
have apps/ and lib/ would crash, as the spec mentions both options as
valid.

- this is done by adding a HiPE dependency
- also in this: adding a test for directories, and fixing some indent

When adding the 'TEST' macro to the test profile, we mistakenly sourced
the erl_opts values from the base profile rather than the test profile
itself.

This means that in cases where the base profile set an option such as
'no_debug_info' and a profile overrode it with 'debug_info', the default
options would get injected within the test profile, and broke the
precedence rules, yielding incompatible values.

This patch fixes things by adding the macro to the values sourced from
the test profile itself, fixing the issue.

This reverts commit 4ad1db9733.

The way arguments are passed by the alias provider is not compatible
with all rebar providers/commands. Especially the release (relx)
provider does not like getting its arguments as a plain string. It
expects its arguments in a pre-parsed format as returned by
getopt:parse/2.

Other commands, e.g. eunit, seem to be fine with both ways of argument
passing. Therefore, this fix changes the alias provider argument passing
to the getopt format in general.

Although fairly irregular, the presence of a subdirectory in ebin/ while
running rebar3 escriptize would crash the run and generate a dump.

Since ebin subdirectories are generally not supported (adding a path for
an application only adds ebin/ and no subdirectories to the VM's path
set), we just silently ignore the directory altogether.

Fixes #1693

- Uses the code at https://github.com/tsloughter/rebar_alias and brings
  it within rebar3
- adds safety checks to prevent redefining built-in commands or obvious
  circular dependencies between commands (indirect circular deps are
  still possible)
- adds tests
- adds a systest to ensure no clash with the existing plugin

Adds an option (-m, --min_coverage, or {cover_opts, {min_coverage,X}})
to the 'cover' command, where the value is an integer between 0 and 100.

If the total coverage during the analysis is below the value received,
the command will fail with output like:

    ===> Requiring 64% coverage to pass. Only 62% obtained

If the rate is correct, the command silently passes as it currently
does.

This feature allows to enforce code coverage standards in a project if
desired.

Current rebar3 uses debug_info rules where debug_info is added by
default, and no_debug_info prevents the default from being added, and
removes any explicit debug_info, if any. The problem is that if
'no_debug_info' is anywhere in the config for a run, it cannot be
removed even with other profiles. additionally, no_debug_info ignores
special tuples like {debug_info, {Mod, Data}} and {debug_info_key, Key},
which can be used to add debug info and encrypt it (in lieu of plain
debug_info) respectively.

This patch makes it so that the following rules are in place:

- the last option seen takes priority, allowing profile overrides by
  the ordering rules the compiler expects
- the overriden options shall be explicitly deleted to avoid confusing
  the compiler
- any option related to debug info seen last cancels any no_debug_info
  that preceded it
- any no_debug_info option seen last cancels all of the other
  debug_info options
- if debug_info is seen last, it cancels out {debug_info_key, Key}
- if {debug_info_key, Key} is seen last, it cancels out debug_info
- All other options are left untouched in that context (defines can
  still be expanded and so on)

This should allow proper profile rules to be followed. Note that erl_opt
profile merging puts precedence on the *last* element of a list, to
match the compiler.

This replaces all deprecated function usage by alternative ones based on
a version switch enacted at compile time, preventing all warnings.

This will likely introduce some possible runtime errors in using a
Rebar3 compiled on OTP-20 or OTP-21 back in versions 19 and earlier, but
we can't really work around that.

A bunch of dependencies have been updated to support OTP-21 without
warnings as well.

The compiling of OTP applications is done by first topographically
sorting them according to their dependencies, deps-first. This allows
all compilation to take place in order. In the current code, the same
logic extends to top-level applications in an umbrella project.

Unfortunately, there are cases where this is not going to be true: when
an application has extra_src_dirs entries (or additional directories or
files) to conditionally compile under some profiles, it may start
depending on another top-level application dedicated to that profile for
include files.

However, such an app will never make it to production and neither will
the compilation artifacts that create the dependency. Under that
scenario, current rebar3 is unusable.

This patch makes it so that the compilation provider instead changes the
logic for top-level apps: rather than copying their directories one by
one and compiling them in order, it:

1. copies all top-level apps to the build directory so the files are in
   their proper locations
2. adds the top-level apps to the path (after the global hooks have run,
   so the existing scope and env has not changed)
3. runs the compilation as usual.

Fixes #1651

Getting real tired of that nondeterministic set of runs on OSX CI