This should cover most cases, including hooks.
What this specifically does is change some less useful messages for more
contextualized ones, including tying specific configurations to actions
taking place. This is particularly visible in hooks, and hopefully
provides some amount of discoverability.
Also of note:
- ebin/ directory copying is called explicitly when done since people
have issues with that
- paths for compiled files are made relative since the analysis list
shows the full paths already
- hooks are a bit more verbose but also more useful.
The change was possible since R13B04 when macros with many arities
became supported. We never really cleaned that stuff up before so I'm
picking it up while I'm updating that code
This patch contains two behaviour changes and reasserts other behaviours
that now line things up with user and documentation expectations:
1. The src directories remain recursive. We turned it on by accident at
some point in the past and now people rely on it, so we're stuck with
it. However a new test ensures that the feature can be turned off
on-demand as documented on the website.
2. The test directories are no longer recursive by default. The fix is
done by properly fixing how rebar3.erl does its feature injection by
mandating the default value there.
I'm somewhat nervous that this change could negatively impact some
users and older compiler module versions, but if users stick to the
rebar_dir interface, everything should keep working transparently.
3. The test directories' configuration is no longer silently dropped.
Due to how rebar3.erl injected test state without looking for what
the user may have specified, multiple extra_src_dirs entries existed
at once and were run; one with the recursion set to true and one with
whatever the user specified. If the user disabled recursion of the
"test" extra_src_dir, then the injected value still ran it once...
4. The handling of extra files in the compiler module is fixed to
actually use the rebar_dir interface properly, and reinjects
non-default directory recursion settings into the swapped options
for the shimmed extra apps. Not doing this annotation step resulted
in the write for swapped opts to actually drop the configured
recursion value and make everything recursive all the time.
A single new test actually validates all of that behaviour and seems to
work fine.
We run the analysis for an extra_src_dir by making a fake app for it,
but send in the app alone for analysis.
In there, the DAG pruning routine looks at the DAG and files submitted
and goes "this right here is a project with 99% of its apps deleted". It
then tries to prune the whole DAG except for some test files. The code
"works" simply because there's a false-positive check that makes sure
the file is on disk before removing it for the DAG.
This ends up making extra runs where ~80% of the time is spent
double-checking the false positives for file deletions.
This commit fixes this by merging in all extra_src fake apps and making them
run in a single analysis phase, meaning we only pay the cost of the DAG
pruning once for the whole project, making it faster than any sparse
repo.
There's also a small patch needed for the root-level extra src dirs;
turns out that since the context-handling in the `rebar_compiler` uses a
map to store content, running single-pass analysis clobbered entries for
a given app if they had more than one extra_src_dir in there.
I also took the time to clean up the ordering of that file.
The propagation was confusing source files and artifacts; the artifact
ordering was flipped, and the tagging non-mandatory (aside from as
edges), which made things hard to identify when plugins for compilers
are used.
This allows to do quicker re-compile option validation by not requiring
to access the disk and check all candidate erlang files for option
changes. This also opens up the way to compilers that produce more than
one artifact per file.
This moves the old DAG stuff out from the main code paths and
substitutes it with the new epp-based DAG work.
Also fixes previous work for integration tests:
- consider ptrans from erl_opts in app ordering
- display proper app compilation info messages due to reordering
While leaving the old one in place, prep the ground for new analysis
phases for the DAG work. The new DAG functions are added, but not hooked
in yet.
Fixes to the EPP handling have also been added due to issues in
resolving include_lib information
This commit is a transition point that makes some assumptions about new
callbacks for the compiler modules, which will likely not hold when it
comes to making epp be able to read and handle multiple applications at
once (so it can resolve parse transforms and behaviours across OTP
apps).
As such, a follow-up commit is going to be completing that one and
changing its API in incompatible ways; if you find this commit during a
bisect, it's probably not a good one in which to run a bisection.
This is another tricky commit towards replacing the current module
analysis with EPP. The compiler DAG is now being shared across multiple
applications being compiled, rather than a per-application basis, which
promises to allow better ordering, parallelism, and more thorough
invalidation of runtime dependencies when they are modified.
This however required changes:
- The compiler DAG is no longer private to `rebar_compiler`, and has
been extracted to the `rebar_compiler_dag` module
- The compiler DAG is now started by the `rebar_prv_compile` module,
which oversees the calls to `rebar_compiler` for each OTP application
- The compiler DAG has been refactored to use a "dirty flag" to know if
it was modified, rather than just tracking modifications in a
functional manner, since the scope change (going multi-app) makes it
impossible to cleanly use the functional approach without much larger
changes
- The DAG used to be cached within each OTP application. This is no
longer possible since it is shared. Instead the DAG is stored in the
state's deps_dir, which allows to cleanly split caches between regular
apps for the user's project and plugins
- The DAG supported a "label" mode that was used to store distinct DAGs
for extra_src_dir runs and regular modules; this label is now used
(and extended to `rebar_prv_compile` internals) to distinguish between
"compile runs", such as "project_apps", or just "apps" (deps). The
label is optional (i.e. not used by plugins which have no such need)
- The extra_src_dirs for each app is now compiled using the main app's
DAG, but the run takes place later in the compilation process. This
may need changing to detect and prevent dependencies from src_dirs
into extra_src_dirs, but this should not technically be a problem for
runtime anyway.
- Reworked the support for extra_src_dirs that are at the root of an
umbrella project (and therefore do not belong to any single app) to
use the new structure, also as part of the project_apps DAG.
All tests keep passing, and this puts us in a better place to use EPP
with cross-app support in the near-future.
This patch does two things:
1. it broadens the interface for the compiler module so that
non-first-file modules can possibly be parallelized. This is done by
dynamically switching on `[ListOfFiles]`, which remains sequential as
before, or `{[SeqPriority], [Parallel]}`, which divides regular files
between higher priority ones and those that can be in parallel
2. implements this mechanism in the rebar compiler, based on the erl
file digraph. If a file has an in-neighbour, it is depended on by
another file. The mechanism therefore makes it so all files that have
dependants get compiled in their strict relative sequential order
first, and then the undepended-on files get compiled together in
parallel.
By running:
./rebar3 ct --suite test/rebar_compile_SUITE.erl --case \
recompile_when_parse_transform_inline_changes --repeat 50
the previous iteration of this would rapidly fail, and this one succeeds
every time.
Support for parallel compilation of *.erl file was dropped before 3.0 release.
However, our tests for a project containing ~500 source files show substantial gain, lowering compilation time from 58 seconds to 18 on a MacBook Pro 15" (4 cores, 8 threads), and to just 10 seconds on Xeon-D machine.
This change fixes cases where changes in .hrl files would not be picked
up in .erl files that are in extra source directories (such as those
defined with `extra_src_dirs` or modules in the test/ directory during a
CT or Eunit run).
The problem was due to the way the Directed Acyclic Graph (DAG) of
dependencies between files was being loaded and stored by the compiler
modules.
Prior to this fix, a single DAG would be used for all runs. On a regular
run, the prior DAG is loaded from disk, re-checked, and if changed, it
would get re-written to disk with the changes deciding what to
re-compile. However, whenever extra source directories were specified, a
second run would be done which swaps target directories around in the
compiler modules.
Bug 1: this second run was done without properly tracking the private .hrl
files (in src/), so the changes were invisible. This has been fixed by
re-adding the paths.
The problem is that the DAG handling is self-contained; just invoking it
was sufficient to get it loaded and rewritten to disk. But since runs
with extra src dirs were done on different sets, the compilation of
extra src dirs would be done with bad historical data (all the modules
in src/ are dropped, all those in test/ are re-added); this DAG was then
written to disk once again, polluting the next non-extra run.
This is bug 2, and it is fixed by adding an optional label to each run
so that a regular or extra compile round can be distinguished, each
tracking their own files in their own DAG.
A single test (and a lot of diffing) were sufficient for this.
* add compile type for dynamic project compilation
* new rebar_compiler abstraction for running multiple compilers
rebar_compiler is a new behaviour that a plugin can implement to
be called on any ues of the compile provider to compile source
files and keep track of their dependencies.
* fix check that modules in .app modules list are from src_dirs
* use project_type to find module for building projects
* allow plugins to add project builders and compilers
Fred Hebert
Maxim Fedorov
Tino Breddin
Tristan Sloughter