Mostly changing the format of debug messages, fixing one URL that was
bad in a warning.

Since we switched to DIAGNOSTIC=1 in the crashdump, the only way to make
DEBUG=1 discoverable (outside of docs) is in the help provider.

Didn't handle tuple-based dep definitions right

Upgrading dependencies already checked for package updates before running
the upgrade task. Plugins didn't have that mechanism in place and that
made upgrading plugins over hex trickier since users could be stuck a
long time without running an update manually.

This fix mostly readapts existing dep code to the context of plugins to
look them up one at a time and run the update as part of the upgrade.

This should cover most cases, including hooks.
What this specifically does is change some less useful messages for more
contextualized ones, including tying specific configurations to actions
taking place. This is particularly visible in hooks, and hopefully
provides some amount of discoverability.

Also of note:
- ebin/ directory copying is called explicitly when done since people
  have issues with that
- paths for compiled files are made relative since the analysis list
  shows the full paths already
- hooks are a bit more verbose but also more useful.

Move most stuff that have to do with effectiveness to the debug
level, and some stuff that has to do with implementation details to
diagnostics

The change was possible since R13B04 when macros with many arities
became supported. We never really cleaned that stuff up before so I'm
picking it up while I'm updating that code

Diagnostic mode is intended to be used for rebar3 maintainers to debug
and understand issues. It shows all the things and all the data and
works with the assumption that people has intent to use the source code
to debug things.

Debug mode is promoted to user-oriented debugging, based on the idea
that what they care about is figuring out their interactions with the
tool. A major improvement in usability would be to directly tie the user
config to the interpreted config (ie merged profiles) to clearly hint at
seen rebar3’s behaviour.  If the debug output can show that and even tie
some behaviours to the config value that impacts things, we could get
some discoverability out of it too, something missing from declarative
tools like this.

Co-authored-by: Fred Hebert <mononcqc@ferd.ca>

This commit changes rebar3 to check for name clashes of the
modules created from its templates with existing Erlang modules
to produce a non-blocking warning for each clashing module.

This is required for compile order to be maintained in a post-3.14
world.

... and e.g. an unknown type is detected

This commit changes mainly the `rebar_paths` module by adding a
new `runtime` target. When using that target, `rebar_paths:get_apps/2`
will, for each project app, find all their `applications` and
`included_applicactions` and filter those that could be found
in rebar3's state.

This was due to a subtle mistake in the new code in
https://github.com/erlang/rebar3/pull/2322 which optimized the search
behaviour of files not found by EPP. The patch mistakenly always kept
the top-level src/ directory for an app even if it did a lookup in
recursive ones.

This patch tracks and maintains the proper subdirectory hierarchy. To
keep performance adequate, the lists:keyfind/3 function is used, which
is also now a NIF to the same extent as lists:member/2; manual lookups
would likely end up reducing performance, particularly in deep
hierarchies.

A test case has been added to track the regression.
Reported by @elbrujohalcon

Saves a few seconds when loading DAG with thousands of files.
Unfortunately, uses internal knowledge about digraph implementation.

Instead of sequentially running *.erl (or any other) files
one-by-one, spawn a separate process, in a scatter-gather way,
then wait for all processes to complete.
Combined with the previous change it introduces a significant
speed-up for large repositories, up to a number of cores that
the host has.

This commit makes rebar_compiler_erl to start a gen_server that acts
as a filesystem cache, avoiding re-scanning it over and over.

rebar3 would crash if the option `{keep_logs, N}` was present and
there were more than N run directories in the log dir, so if the
option is found we keep the N-1 newest directories and delete the
rest.

While we are at it, as this should fix the crash, let's limit the
number of logs runs to keep from test runs in rebar3 to 5.

Modify also the load_file function by adding a
way to check an error, and abort the program
smoothly.

Co-authored-by: Fred Hebert <mononcqc@ferd.ca>

this fixes the issue where using 'rebar3 plugins upgrade P'
would result in a plugin's .app file having an empty
modules list. The code this replaces hadn't been touched
since rebar3 3.0.0-beta.3 and a lot of improvements have
gone into dep handling and the compiler since then. This
change should take advantage of those changes.

This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:

1. The src directories remain recursive. We turned it on by accident at
   some point in the past and now people rely on it, so we're stuck with
   it. However a new test ensures that the feature can be turned off
   on-demand as documented on the website.

2. The test directories are no longer recursive by default. The fix is
   done by properly fixing how rebar3.erl does its feature injection by
   mandating the default value there.
   I'm somewhat nervous that this change could negatively impact some
   users and older compiler module versions, but if users stick to the
   rebar_dir interface, everything should keep working transparently.

3. The test directories' configuration is no longer silently dropped.
   Due to how rebar3.erl injected test state without looking for what
   the user may have specified, multiple extra_src_dirs entries existed
   at once and were run; one with the recursion set to true and one with
   whatever the user specified. If the user disabled recursion of the
   "test" extra_src_dir, then the injected value still ran it once...

4. The handling of extra files in the compiler module is fixed to
   actually use the rebar_dir interface properly, and reinjects
   non-default directory recursion settings into the swapped options
   for the shimmed extra apps. Not doing this annotation step resulted
   in the write for swapped opts to actually drop the configured
   recursion value and make everything recursive all the time.

A single new test actually validates all of that behaviour and seems to
work fine.

This commit changes the location where rebar3 will extract
its libs upon calling a `local` command from `~/.cache/rebar3/lib`
to `~/.cache/rebar3/vsns/VERSION/lib`, with `VERSION` being
rebar3 current version.

It also updates the bin script to reflect these changes.