This allows to reduce the number of noise to only checking deps' app
files when they're built, rather than on every run.

Since main apps and checkouts are still compiled every time, the linting
takes place there and then with a higher frequency.

Compile time with parse transforms needs to have the deps first in scope
before the plugins

Due to building dependencies with potential artifacts such as parse
transforms or macros, project builder plugins should be included in the
path, but _after_ deps are loaded.

Doing otherwise means that if any of the dependencies is required at
compile time, those of a plugin might get used first.

- Only set paths that need to be put as a priority
- Clean up paths before leaving API mode

The first point accounted for some performance cost, but the latter one
explains the 40% overhead in test runs: since rebar3 calls rebar3 a lot
with a bunch of fake apps, and that the new mechanism for path handling
by default does not _remove_ paths, it just _orders_ them, we would end
up in a situation where as the tests ran, more and more fake paths would
get added to the VM.

By the time the run was over, all path handling would take longer since
more paths needed filtering every time. By resetting paths at the end of
an API run, we prevent a given 'project' from polluting another one's
runtime and performance once the API successfully returns.

Also handle some formatting

Move path management out of rebar_utils manual code path function
handling (which we leave there for backwards compat), and centralize
them to allow easier coordination of paths between plugins and deps.

On top of path handling, do a check of loaded modules to only purge and
reload those that actually need it done in order to prevent all kinds of
weird interaction and accidental purge kills. It also allows the
possible cohabitation of both at once, with a "in case of conflict pick
X" as a policy

Changing path handling in providers also highlighted a bunch of bugs in
some tests and appears to fix some in other providers, specifically
around plugins.

* add compile type for dynamic project compilation

* new rebar_compiler abstraction for running multiple compilers

rebar_compiler is a new behaviour that a plugin can implement to
be called on any ues of the compile provider to compile source
files and keep track of their dependencies.

* fix check that modules in .app modules list are from src_dirs

* use project_type to find module for building projects

* allow plugins to add project builders and compilers

Priv and include dirs need the virtual symlink in order to preserve hook
functionality in some edge cases.

This would cause crashes on linux and force people to have a src_dirs
config that is strictly matching what is on the file system rather than
acting as a specification of those that are valid.

To compare, if lib_dirs worked the same, then any repo that did not both
have apps/ and lib/ would crash, as the spec mentions both options as
valid.

Rather than the caller having to think of what to purge or not, use
erlang:check_process_code/2 to detect if the caller (rebar3) may die
because of the operation.

If so, do a soft purge with a conditional delete instead of a hard purge
with a mandatory delete.

Prevents the killing of a plugin with itself

The Erlang compiler runs based on a global state built from currently
loaded libraries and the configured code path that is available. For
this reason, the rebar3 compiler job unloads all plugin paths before
calling the Erlang compiler.

However, this causes a problem when an application uses a custom
resource handler with a dynamic version in their .app.src file since the
plugin that can be used to find the version has been unloaded.

Fortunately, the compile phase that runs the version handling is
distinct from the phase that uses the Erlang compiler. This patch fixes
the problem by re-loading the plugins' paths in memory before generating
the .app file, and before unloading them afterwards.

It appears that unloading them is unnecessary because the hooks after
that will re-load them, but it is likely better to play it safe with
that global state and clean up after ourselves. It offers better
protection for future changes.

Fixes #1657

The compiling of OTP applications is done by first topographically
sorting them according to their dependencies, deps-first. This allows
all compilation to take place in order. In the current code, the same
logic extends to top-level applications in an umbrella project.

Unfortunately, there are cases where this is not going to be true: when
an application has extra_src_dirs entries (or additional directories or
files) to conditionally compile under some profiles, it may start
depending on another top-level application dedicated to that profile for
include files.

However, such an app will never make it to production and neither will
the compilation artifacts that create the dependency. Under that
scenario, current rebar3 is unusable.

This patch makes it so that the compilation provider instead changes the
logic for top-level apps: rather than copying their directories one by
one and compiling them in order, it:

1. copies all top-level apps to the build directory so the files are in
   their proper locations
2. adds the top-level apps to the path (after the global hooks have run,
   so the existing scope and env has not changed)
3. runs the compilation as usual.

Fixes #1651

This is done through 3 main change groups:

- replacing `~s` by `~ts` in format strings, so that strings that
  contain unicode are properly printed rather than crashing
- adding the `unicode` argument to all function of the `re` module to
  ensure transformations on strings containing unicode data are valid
  instead of crashing (see issue #1302)
- replacing `ec_cnv:to_binary/1` and `ec_cnv:to_list/1` with matching
  functions in `rebar_utils`.

The last point has been done, rather than modifying and updating erlware
commons, because binary and list conversions can be a contentious
subject. For example, if what is being handled is actually bytes from a
given binary stream, then forcing a byte-oriented interpretation of the
data can corrupt it. As such, it does not appear safe to modify erlware
commons' conversion functions since it may not be safe for all its
users.

Instead, rebar3 reimplements a subset of them (only converting
atoms and chardata, ignoring numbers) with the explicit purpose of
handling unicode string data.

Tests were left as unchanged as possible. This may impact the ability to
run rebar3's own suites in a unicode path, but respects a principle of
least change for such a large patch.

erlc_compile, for before/after compiling .erls to
.beams, but before .app.src to .app

if these directories actually exist they'll be added to the path ahead
of the release/standard distribution directories and they'll break eunit
and/or ct execution

fixes #950

previously rebar3 dropped suites declared at the root of the project (via
`--suite=whatever_SUITE' probably) and warned. this was because the compiler
would recursively copy and compile everything in the directory indicated by
the test suite. this changes the copy mechanism to only copy erl source files
and directories that end with `_SUITE_data' into the `extras' dir in `_build'

* modify compiler interface to work on either application objects or
  directories containing source files
* compile all sources in `src_dirs` to the application `ebin` dir and
  all sources in `extra_src_dirs` to a directory mirroring it's
  position in the app's `_build` directory. for example, `apps/foo/more`
  would compile to `_build/default/lib/foo/more`
  for `extra_src_dirs` in the root of a project with multiple
  applications (so orphan directories that don't "belong" to an
  application) compile to `_build/default/extras/more`
* copy directories specified in `extra_src_dirs` into the `_build`
  directory so tools like `ct` and `xref` that expect source to be
  in a particular location still work
* clean compiled artifacts from all `extra_src_dirs`
* alter `eunit`, `ct` and `cover` to work with the new directory
  structure
* billions of new tests

This reverts commit b06a6ecadd.

- Crashes in providers lib when no providers in a namespace are bare
- Making sure bareness matches semantics; i.e. a bare provider is
  visible, a non-bare provider is hidden.

adding it results in ct test suites being compiled before they're
properly copied so changes made to suites between runs are not
reflected if a run fails before test dirs are copied

note that now ALL `src_dirs` across included profiles are
compiled. previously only the last included profile's `src_dirs`
were used

like `src_dirs`, `extra_src_dirs` are directories to be copied to
the `_build` dir and compiled. unlike `src_dirs` they are not added
to the .app specification