While leaving the old one in place, prep the ground for new analysis
phases for the DAG work. The new DAG functions are added, but not hooked
in yet.

Fixes to the EPP handling have also been added due to issues in
resolving include_lib information

This commit is a transition point that makes some assumptions about new
callbacks for the compiler modules, which will likely not hold when it
comes to making epp be able to read and handle multiple applications at
once (so it can resolve parse transforms and behaviours across OTP
apps).

As such, a follow-up commit is going to be completing that one and
changing its API in incompatible ways; if you find this commit during a
bisect, it's probably not a good one in which to run a bisection.

This reverts commit 025a96b143.

Solves #2071, IPV6 support for proxy, by trying to resolve proxy
host name without specific IP address family. If it fails, it
attempts a different IP family, and in case that succeeds, it
sets up HTTP client profile to use IP family that worked.

This enables relative paths for checkouts_dir configuration variable.

It is quite normal to have git 2.17.1 on MacOS X.

The code was written at the time only one lockfile version was being
explicitly tagged (1.1.0), and all the other older ones were untagged.
In introducing the 1.2.0 format for external hashes, the current version
check still took place, but no provisions were made to explicitly check
between older or newer versions. As such, rebar3 would warn (prompting
people to upgrade) every time a lockfile on the 1.1.0 version (which is
likely over 99% of those in the wild as of today) would be encountered,
we'd prompt people to upgrade.

This patch introduces a constant for the list of supported tagged
versions for config/lock files, which allows to drop warnings when
rebar3 can transparently upgrade the lock file for the user.

It pre-supposes a potential "unsupported lock file" error could be added
in the future if a config format is not newer but in fact explicitly
deprecated. This isn't implemented but the current logic works towards
making that support possible.

- Vendor in hex_core at v0.6.8
 - Rename checksum to inner_checksum and deprecate in favor of new outer_checksum key/val
 - Change warn_vsn_once/0 to warn_vsn_once/1 so we can give a proper error message and
   take appropriate action.
 - Update rebar.lock to 1.2.0 format with continued support third-party
   that still may make use of inner checksum via continued persistence of
   inner checksums.
 - Changed app info tuple (package representation) to support inner and
   outer hashes for backwards compat support (see above)

This is another tricky commit towards replacing the current module
analysis with EPP. The compiler DAG is now being shared across multiple
applications being compiled, rather than a per-application basis, which
promises to allow better ordering, parallelism, and more thorough
invalidation of runtime dependencies when they are modified.

This however required changes:
- The compiler DAG is no longer private to `rebar_compiler`, and has
  been extracted to the `rebar_compiler_dag` module
- The compiler DAG is now started by the `rebar_prv_compile` module,
  which oversees the calls to `rebar_compiler` for each OTP application
- The compiler DAG has been refactored to use a "dirty flag" to know if
  it was modified, rather than just tracking modifications in a
  functional manner, since the scope change (going multi-app) makes it
  impossible to cleanly use the functional approach without much larger
  changes
- The DAG used to be cached within each OTP application. This is no
  longer possible since it is shared. Instead the DAG is stored in the
  state's deps_dir, which allows to cleanly split caches between regular
  apps for the user's project and plugins
- The DAG supported a "label" mode that was used to store distinct DAGs
  for extra_src_dir runs and regular modules; this label is now used
  (and extended to `rebar_prv_compile` internals) to distinguish between
  "compile runs", such as "project_apps", or just "apps" (deps). The
  label is optional (i.e. not used by plugins which have no such need)
- The extra_src_dirs for each app is now compiled using the main app's
  DAG, but the run takes place later in the compilation process. This
  may need changing to detect and prevent dependencies from src_dirs
  into extra_src_dirs, but this should not technically be a problem for
  runtime anyway.
- Reworked the support for extra_src_dirs that are at the root of an
  umbrella project (and therefore do not belong to any single app) to
  use the new structure, also as part of the project_apps DAG.

All tests keep passing, and this puts us in a better place to use EPP
with cross-app support in the near-future.

(cherry picked from commit 08eb611a3d)

(cherry picked from commit fa577bc46e)

(cherry picked from commit 1c9e91b80a)

Attempts to bridge http_uri (deprecated in OTP 23)
and uri_string (available as of OTP 21+).

(cherry picked from commit 7057cbe281)

The problem is that the shell util handlers receive a byte stream, which
we handle as a list coming from a port. However the byte stream is utf-8
encoded in most systems, which breaks encoding expectations (binaries
only) for Erlang.

This patch turns the list of characters to a binary for all shell calls,
before re-encoding it properly as unicode lists to keep the format compatible.

Fixes #2205

This is a new way to orient the code and would allow better
parallelization or multi-app scan phases

This allows breaking apart the pre-hooks from the rest of the
compilation steps, as a preliminary step towards being able to do some
analysis on all project apps at once before actually compiling them.

This warning is emitted every single time one starts a shell with applications
configured for automatic startup (e.g. `{shell, [{apps, [cowboy]}]}`).

Infortunately it does not make any sense: people who use this feature need it
to interact with their application or library during development, and are
therefore subjected to the warning every single time they start the shell even
though this is the right way to use it.

While the shell is not to be used to run applications in productions, those
who do it won't be stopped by a warning.

Ultimately, the warning annoys people who use the feature as intended, and do
nothing to stop people who abuse it. Thus I believe it should be removed.

See https://github.com/erlang/rebar3/issues/2206.

close #2206

async_do with arguments is useful for debugging a particular suite/test
with r3:break()

Example session:
Add r3:break() in recompile_when_hrl_changes test in test/rebar_compile_SUITE.erl

rebar3 shell
...
1> r3:async_do(default, ct, "--suite=test/rebar_compile_SUITE.erl --case=recompile_when_hrl_changes").
ok
Running Common Test suites...
%%% rebar_compile_SUITE:
=== BREAK ===

2> r3:resume().
ok
3> .
All 1 tests passed.

For OTP 19 and earlier compatibility.

This function was used for trimming the version from a `{file, File}` app
vsn in .app.src but caused a crash because it did not exist.

Switching to uri_string as the warning suggest
would ruin OTP 20 compatibility.

See [1] for context.

1. https://bugs.erlang.org/browse/ERL-1113

Switching to uri_string as the warning suggest
would ruin OTP 20 compatibility.

In some cases its nice to have dialyzer statistics out of box without
any additional steps.

Windows does not have the same quoting rules as Unix so
we need to use platform independent quoting. Also for
os:cmd like operations we do not expect the program to
read any input so in order for it to work on windows we
should put -noinput there.

In r3_hex_core.erl we set default hex config. For example, we set the
api_organization to be undefined. By merging the hex auth config into
the default config this way round we allow users to override the default
config.

This will allow project with larger dependencies sets to clean only the
apps they want to when testing or changing small things, rather than
forcing a rebuild of the whole dep set.

Also allows cleaning up apps, not just deps.

We never really need that and nobody would expect it either. We're
rendering normal files, not HTML.

Fixes #2159; this is done by force-reloading the handlers to match the
config.
This can create a bit of a funny effect if sys.config specifies an INFO
log level (or lower) is specified. While apps are booted for config
changes before the cth_failonly hook is enabled, supervision and other
application log messages can start being output. They will start being
suppressed once the CT run begins.

This is not a bug, it's a race in instantiation of control and enabling
of log levels. Nothing we can do about that. It might however surprise
people a good bit. If non-default stdout handlers are added, they are
similarly likely to become noisy; specifying a test-specific sys.config
file may be necessary then.

Also includes a bump of cth_readable version, which now checks for
updates to the log formatter on every test output.

See http://erlang.org/doc/man/ct_hooks.html#Module:on_tc_fail-4

To avoid a conflict the Kernel's default handler have to be disabled
before the new default handler is added

    {kernel,
     [{logger,
       [{handler, default, undefined}
       ]}
     ]},
    {my_app,
     [{logger,
       [{handler, default, my_handler, #{}}
       ]}
     ]}

To support this behavior in shell it's necessary to process the
handler-default-undefined tuple.

It's not clear what to do with the simple handler (logger_simple_h)
however.  It's added at the start and removed if a default handler was
added.  Should it be added in reread_logger_config/1 if there's no
default handler?