Diagnostic mode is intended to be used for rebar3 maintainers to debug
and understand issues. It shows all the things and all the data and
works with the assumption that people has intent to use the source code
to debug things.

Debug mode is promoted to user-oriented debugging, based on the idea
that what they care about is figuring out their interactions with the
tool. A major improvement in usability would be to directly tie the user
config to the interpreted config (ie merged profiles) to clearly hint at
seen rebar3’s behaviour.  If the debug output can show that and even tie
some behaviours to the config value that impacts things, we could get
some discoverability out of it too, something missing from declarative
tools like this.

This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:

1. The src directories remain recursive. We turned it on by accident at
   some point in the past and now people rely on it, so we're stuck with
   it. However a new test ensures that the feature can be turned off
   on-demand as documented on the website.

2. The test directories are no longer recursive by default. The fix is
   done by properly fixing how rebar3.erl does its feature injection by
   mandating the default value there.
   I'm somewhat nervous that this change could negatively impact some
   users and older compiler module versions, but if users stick to the
   rebar_dir interface, everything should keep working transparently.

3. The test directories' configuration is no longer silently dropped.
   Due to how rebar3.erl injected test state without looking for what
   the user may have specified, multiple extra_src_dirs entries existed
   at once and were run; one with the recursion set to true and one with
   whatever the user specified. If the user disabled recursion of the
   "test" extra_src_dir, then the injected value still ran it once...

4. The handling of extra files in the compiler module is fixed to
   actually use the rebar_dir interface properly, and reinjects
   non-default directory recursion settings into the swapped options
   for the shimmed extra apps. Not doing this annotation step resulted
   in the write for swapped opts to actually drop the configured
   recursion value and make everything recursive all the time.

A single new test actually validates all of that behaviour and seems to
work fine.

- Only set paths that need to be put as a priority
- Clean up paths before leaving API mode

The first point accounted for some performance cost, but the latter one
explains the 40% overhead in test runs: since rebar3 calls rebar3 a lot
with a bunch of fake apps, and that the new mechanism for path handling
by default does not _remove_ paths, it just _orders_ them, we would end
up in a situation where as the tests ran, more and more fake paths would
get added to the VM.

By the time the run was over, all path handling would take longer since
more paths needed filtering every time. By resetting paths at the end of
an API run, we prevent a given 'project' from polluting another one's
runtime and performance once the API successfully returns.

* add compile type for dynamic project compilation

* new rebar_compiler abstraction for running multiple compilers

rebar_compiler is a new behaviour that a plugin can implement to
be called on any ues of the compile provider to compile source
files and keep track of their dependencies.

* fix check that modules in .app modules list are from src_dirs

* use project_type to find module for building projects

* allow plugins to add project builders and compilers

* update to hex_core for hex-v2 repo support (#1865)

* update to hex_core for hex-v2 repo support

This patch adds only single repo hex-v2 support through hex_core.
Packages no longer filtered out by buildtool metadata and the
package index is updated per-package instead of fetched as one
large ets dump.

* tell travis to also build hex_core branch

* support list of repos for hex packages (#1866)

* support list of repos for hex packages

repos are defined under the hex key in rebar configs. They can be
defined at the top level of a project or globally, but not in
profiles and the repos configured in dependencies are also ignored.

Searching for packages involves first checking for a match in the
local repo index cache, in the order repos are defined. If not found
each repo is checked through the hex api for any known versions of
the package and the first repo with a version that fits the constraint
is used.

* add {repos, replace, []} for overriding the global & default repos

* add hex auth handling for repos (#1874)

auth token are kept in a hex.config file that is modified by the
rebar3 hex plugin.

Repo names that have a : separating a parent and child are considered
organizations. The parent repo's auth will be included with the child.
So an organization named hexpm:rebar3_test will include any hexpm
auth tokens found in the rebar3_test organization's configuration.

* move packages to top level of of hexpm cache dir (#1876)

* move packages to top level of of hexpm cache dir

* append organization name to parent's repo_url when parsing repos

* only eval config scripts and apply overrides once per app (#1879)

* only eval config scripts and apply overrides once per app

* move new resource behaviour to rebar_resource_v2 and keep v1

* cleanup use of rebar_resource module and unused functions

* cleanup error messages and unused code

* when discovering apps support mix packages as unbuilt apps (#1882)

* use hex_core tarball unpacking support in pkg resource (#1883)

* use hex_core tarball unpacking support in pkg resource

* ignore etag if package doesn't exist and delete if checksum fails

* add back tests for bad package checksums

* improve bad registry checksum error message

- added type specs for following private functions:
    - run/1
    - set_options/2
    - test_state/1
    - safe_define_test_macro/1
    - test_defined/1

fixes an issue when shell is terminated with an error "Bus error: 10",
on attempt to run rebar shell with verbose logging

Based off a macro by @okeuday at https://github.com/erlang/otp/pull/1783

When adding the 'TEST' macro to the test profile, we mistakenly sourced
the erl_opts values from the base profile rather than the test profile
itself.

This means that in cases where the base profile set an option such as
'no_debug_info' and a profile overrode it with 'debug_info', the default
options would get injected within the test profile, and broke the
precedence rules, yielding incompatible values.

This patch fixes things by adding the macro to the values sourced from
the test profile itself, fixing the issue.

When a config file exists at the root of a project, defines a given
configuration value for a given profile, and that a sub-application
(umbrella app) also has the same profile defined with the same key (but
different values), the configuration values of the sub-application's
profile would get silently dropped.

The problem being that when the function to merge profiles is applied
recursively, it is applied to each profile (so it will merge on the keys
test, prod, etc.) rather than to each of the values of each profile.

This patch reworks the profile merging so that the current behaviour is
respected overall (a profile cannot be cancelled by a subdep's
non-existant profile since its value should have been ignored), but
ensures that sub-deps' profiles are otherwise applied recursively with
the proper rules:

- dependencies favor prior values
- plugins favor new values
- erl_first_files combine the lists
- relx uses the tuple merge algorithm
- erl_opts has its own custom merge as well
- otherwise the new value takes precedence

A test has also been added.

There is a risk of breakage in some applications that may have relied on
the buggy behaviour to work, though at this time we are aware of none of
them.

This is done through 3 main change groups:

- replacing `~s` by `~ts` in format strings, so that strings that
  contain unicode are properly printed rather than crashing
- adding the `unicode` argument to all function of the `re` module to
  ensure transformations on strings containing unicode data are valid
  instead of crashing (see issue #1302)
- replacing `ec_cnv:to_binary/1` and `ec_cnv:to_list/1` with matching
  functions in `rebar_utils`.

The last point has been done, rather than modifying and updating erlware
commons, because binary and list conversions can be a contentious
subject. For example, if what is being handled is actually bytes from a
given binary stream, then forcing a byte-oriented interpretation of the
data can corrupt it. As such, it does not appear safe to modify erlware
commons' conversion functions since it may not be safe for all its
users.

Instead, rebar3 reimplements a subset of them (only converting
atoms and chardata, ignoring numbers) with the explicit purpose of
handling unicode string data.

Tests were left as unchanged as possible. This may impact the ability to
run rebar3's own suites in a unicode path, but respects a principle of
least change for such a large patch.

Includes improvments and function documentation for all modules (in
alphabetical order) up to rebar_core, and may have included more in
other modules as I saw fit to dig and understand more of the internals.

It turns out that pretty-printing uses throw-catch and thus garbages out
the stack trace sometimes, so we should get it only once.

Insert a newline before printing the stacktrace so that the term is
easier to read and copy. This is a more conventional way to print
traces, and is, for instance, the way it's done by make and python.

Setting DEBUG/QUIET environment variable to the empty string now acts
the same as unsetting it. Unsetting is not always easy/possible.

Formatted errors can accidentally contain substrings which are control
sequences for io:format/2. This is a naive attempt to handle such cases.

One example is running xref on the following module
(assuming module m does not exist)

```
-module(handle_error).

-export([f/0]).

f() -> m:'bobby~stables'().
```

```
$ rebar3 xref
===> Verifying dependencies...
===> Compiling myapp
===> Running cross reference analysis...
escript: exception error: bad argument
  in function  io:format/3
     called as io:format(<0.23.0>,
                         "\e[0;31m===> \e[1mWarning: handle_error:f/0 is unused export (Xref)\nWarning: handle_error:f/0 calls undefined function m:bobby~stables/0 (Xref)\n\n\e[0m\e[0m",
                         [])
  in call from rebar3:handle_error/1 (/Users/gomoripeti/git/rebar3/_build/default/lib/rebar/src/rebar3.erl, line 279)
```

Instead of reading every time that rebar_dir:global_cache_dir/1 is
called

This reverts commit 4c32c52b55.

The provider is used for debugging to help displaying current rebar's state.

Usage:
  rebar3 state

The current code could not cope with missing dependencies, as they would
prevent the rebar3 app from loading or properly building its config,
which prevented the log state from being carried along with default
values. This in turn would turn in an escript-level error that
obfuscated the true source of failure.

This patch bypasses the whole state setup and logging macros and logs an
error message manually when a dependency such as crypto or SSL is
missing from the Erlang install.