This commit changes mainly the `rebar_paths` module by adding a
new `runtime` target. When using that target, `rebar_paths:get_apps/2`
will, for each project app, find all their `applications` and
`included_applicactions` and filter those that could be found
in rebar3's state.

This was due to a subtle mistake in the new code in
https://github.com/erlang/rebar3/pull/2322 which optimized the search
behaviour of files not found by EPP. The patch mistakenly always kept
the top-level src/ directory for an app even if it did a lookup in
recursive ones.

This patch tracks and maintains the proper subdirectory hierarchy. To
keep performance adequate, the lists:keyfind/3 function is used, which
is also now a NIF to the same extent as lists:member/2; manual lookups
would likely end up reducing performance, particularly in deep
hierarchies.

A test case has been added to track the regression.
Reported by @elbrujohalcon

Saves a few seconds when loading DAG with thousands of files.
Unfortunately, uses internal knowledge about digraph implementation.

Instead of sequentially running *.erl (or any other) files
one-by-one, spawn a separate process, in a scatter-gather way,
then wait for all processes to complete.
Combined with the previous change it introduces a significant
speed-up for large repositories, up to a number of cores that
the host has.

This commit makes rebar_compiler_erl to start a gen_server that acts
as a filesystem cache, avoiding re-scanning it over and over.

rebar3 would crash if the option `{keep_logs, N}` was present and
there were more than N run directories in the log dir, so if the
option is found we keep the N-1 newest directories and delete the
rest.

While we are at it, as this should fix the crash, let's limit the
number of logs runs to keep from test runs in rebar3 to 5.

Modify also the load_file function by adding a
way to check an error, and abort the program
smoothly.

Co-authored-by: Fred Hebert <mononcqc@ferd.ca>

this fixes the issue where using 'rebar3 plugins upgrade P'
would result in a plugin's .app file having an empty
modules list. The code this replaces hadn't been touched
since rebar3 3.0.0-beta.3 and a lot of improvements have
gone into dep handling and the compiler since then. This
change should take advantage of those changes.

This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:

1. The src directories remain recursive. We turned it on by accident at
   some point in the past and now people rely on it, so we're stuck with
   it. However a new test ensures that the feature can be turned off
   on-demand as documented on the website.

2. The test directories are no longer recursive by default. The fix is
   done by properly fixing how rebar3.erl does its feature injection by
   mandating the default value there.
   I'm somewhat nervous that this change could negatively impact some
   users and older compiler module versions, but if users stick to the
   rebar_dir interface, everything should keep working transparently.

3. The test directories' configuration is no longer silently dropped.
   Due to how rebar3.erl injected test state without looking for what
   the user may have specified, multiple extra_src_dirs entries existed
   at once and were run; one with the recursion set to true and one with
   whatever the user specified. If the user disabled recursion of the
   "test" extra_src_dir, then the injected value still ran it once...

4. The handling of extra files in the compiler module is fixed to
   actually use the rebar_dir interface properly, and reinjects
   non-default directory recursion settings into the swapped options
   for the shimmed extra apps. Not doing this annotation step resulted
   in the write for swapped opts to actually drop the configured
   recursion value and make everything recursive all the time.

A single new test actually validates all of that behaviour and seems to
work fine.

This commit changes the location where rebar3 will extract
its libs upon calling a `local` command from `~/.cache/rebar3/lib`
to `~/.cache/rebar3/vsns/VERSION/lib`, with `VERSION` being
rebar3 current version.

It also updates the bin script to reflect these changes.

validate_app in rebar_otp_app does not need to set the vsn and
original_vsn because they are set from the app details during
discovery.

The original_vsn was set here before becuase when there is no
.app.src or .app.src.script the app file does not go through
preprocess/3 and preprocess/3 is where vsn is set based on evaluating
the vsn from .app.src. But if neither of these files exist it
means the discovery vsn was the correct version and there is
no need to update.

As reported by Alexey Pronin, this would let people potentially pass in
URLs that can call to command execution.

While running arbitrary code is kind of our bread and butter, it makes
sense to at least play nicely and escape more characters:

- the pipe (`|`) which can be used to run subcommands
- the tab (`\t`) which the shell interprets as a space.
- the home directory (`~`) just because the expansion could behave funny
- redirection (`<` and `>`) to avoid problems with data setting up
  streams

Before, running `rebar3 ct` on such an empty project would crash with an
internal badmatch since no directory to run CT on could be found.

Now this case will terminate with a sensible error message and won't try
to run any further.

Commit 029cbcc7 changed the return value of `rebar_git_resource:lock_/2`

None of the version checks are considered safe enough by mix
maintainers, and so the agreed upon mechanism is to just set this
environment variable, which will be supported implicitly by rebar3
versions that can handle it.

This lets build artifacts (aside from some priv/ issues with plugins) to
properly be located within their build folder rather than the deps
folder, as the --outdir switch would otherwise have done.

Not all of the changes to the git resource were reverted when
the sparse checkout support was moved to a new resource,
git_subdir_resource. This fixes issues caused by these changes
that were accidentally left.

Since the compiler refactor changed the order of steps and all hooks get
applied at once, knowing which app is being escriptized isn't really
obvious, particularly when many apps individually define their own 'main
app' and they get run at once.

This adds the name of the main app being run when escriptizing it to make
it clear what goes on, especially if there are failures.

This was reported in https://github.com/erlang/rebar3/pull/2211#issuecomment-615937628

Fixes #2273

This patch aims to make rebar3 delete a dependency by first looking for
its name (fixing `del` on hex dependencies) and if that isn't found
remove it as always, keeping its current behaviour with the `del` override.

Starting with OTP-22, Erlang started changing how volume names are handled in
windows: filename:join/1 drops drive letters, filename:split/2 and
file:absname/1 use c:/ as a form instead of c:\\

We adjust the file_utils and their tests accordingly.

Rebar3.14 has new tests that were not windows ready and those are fixed.

Relx has two failing tests that are not covered in this branch.

Tested in Windows 10 with Powershell.