As reported by Alexey Pronin, this would let people potentially pass in
URLs that can call to command execution.
While running arbitrary code is kind of our bread and butter, it makes
sense to at least play nicely and escape more characters:
- the pipe (`|`) which can be used to run subcommands
- the tab (`\t`) which the shell interprets as a space.
- the home directory (`~`) just because the expansion could behave funny
- redirection (`<` and `>`) to avoid problems with data setting up
streams
- Normalize path behaviour to always be "/" across versions
- Normalize port visibility behaviour to match input string when
appending
- restore rebar_utils function to avoid breaking random plugins that
peek into our libs for their stuff
- Support URIOpts equivalent and protocol-based overrides; the behaviour
there is not necessarily consistent across versions in terms of what
ports are returned (http_uri always returned a port but uri_string
only returns it if explicitly specified -- we choose the latter
behaviour on newer releases), but the calls work to ensure URI parsing
consistently works across versions
The latter judgement call is a bit of an odd one; for consistency we
could always mandate ports, but this would come at a performance penalty
when appending URL paths (i.e. when fetching packages).
The reason for this is that we standardize on the new uri_string
behaviour for path appending (if the port wasn't specified, we don't add
it), and with the http_uri rules, we need to do a kind of parsing round
that checks if the port was included or not to make it equivalent. This
is costly, and _not_ returning the port when it isn't specified lets us
do this transparently.
This allows to maintain backwards compat in older append functions, and
nobody aside from us currently uses the new rebar_uri module so we can
decide to introduce this potential inconsistency if we wish to.
Solves #2071, IPV6 support for proxy, by trying to resolve proxy
host name without specific IP address family. If it fails, it
attempts a different IP family, and in case that succeeds, it
sets up HTTP client profile to use IP family that worked.
The problem is that the shell util handlers receive a byte stream, which
we handle as a list coming from a port. However the byte stream is utf-8
encoded in most systems, which breaks encoding expectations (binaries
only) for Erlang.
This patch turns the list of characters to a binary for all shell calls,
before re-encoding it properly as unicode lists to keep the format compatible.
Fixes#2205
To avoid a conflict the Kernel's default handler have to be disabled
before the new default handler is added
{kernel,
[{logger,
[{handler, default, undefined}
]}
]},
{my_app,
[{logger,
[{handler, default, my_handler, #{}}
]}
]}
To support this behavior in shell it's necessary to process the
handler-default-undefined tuple.
It's not clear what to do with the simple handler (logger_simple_h)
however. It's added at the start and removed if a default handler was
added. Should it be added in reread_logger_config/1 if there's no
default handler?
OTP kernel application use "logger_level" configuration for configuring
level in primary configuration.
rebar3 uses "logger_info" for this purpose - ths is little bit confusing
and probably mistake.
This commit will unify behavior between kernel and rebar3o
Fixes: 0303567d95 ("Reload logger config in shell")
- vendor in hex_core at v0.5.0
- Change where repo_name should be the org and not the parent
- Changed rebar_utils:url_append_path/2 to not explicitly add a '?',
this is returned in the Query chunk by http_uri:parse/1 (e.g., "?foo=bar")
- update organization_merging test to expect the sub-repo as the repo_name
- Add tests for rebar_utils:url_append_path/2
- Stop referencing/setting "organization" in config and use new
organization settings (api_repository and repo_organization)
- Do not set (assume) the read key is valid for any/every repo
- Set repo_organization and api_repository to org
- Update tests to check for new config opts
Dollar Variable expansion (`$VAR`) was inadvertently disabled for
windows variables, although %VARIABLES% already worked. This reduced the
portability of hooks in general.
Additionally, tests would fail on windows due to bad quoting of paths:
the path C:/a/b/c would fail when passed to the command
`cmd /q /c C:/a/b/c` because it would interpret /a /b and /c as 3
options. Using quotes makes the tests pass.
app_info was turning any vsn into a binary which the vcs_vsn
function interprets as being an actual version and not something
like <<"git">>. original_vsn shouldn't be converted as it is then
not longer "original".
* add compile type for dynamic project compilation
* new rebar_compiler abstraction for running multiple compilers
rebar_compiler is a new behaviour that a plugin can implement to
be called on any ues of the compile provider to compile source
files and keep track of their dependencies.
* fix check that modules in .app modules list are from src_dirs
* use project_type to find module for building projects
* allow plugins to add project builders and compilers
* update to hex_core for hex-v2 repo support (#1865)
* update to hex_core for hex-v2 repo support
This patch adds only single repo hex-v2 support through hex_core.
Packages no longer filtered out by buildtool metadata and the
package index is updated per-package instead of fetched as one
large ets dump.
* tell travis to also build hex_core branch
* support list of repos for hex packages (#1866)
* support list of repos for hex packages
repos are defined under the hex key in rebar configs. They can be
defined at the top level of a project or globally, but not in
profiles and the repos configured in dependencies are also ignored.
Searching for packages involves first checking for a match in the
local repo index cache, in the order repos are defined. If not found
each repo is checked through the hex api for any known versions of
the package and the first repo with a version that fits the constraint
is used.
* add {repos, replace, []} for overriding the global & default repos
* add hex auth handling for repos (#1874)
auth token are kept in a hex.config file that is modified by the
rebar3 hex plugin.
Repo names that have a : separating a parent and child are considered
organizations. The parent repo's auth will be included with the child.
So an organization named hexpm:rebar3_test will include any hexpm
auth tokens found in the rebar3_test organization's configuration.
* move packages to top level of of hexpm cache dir (#1876)
* move packages to top level of of hexpm cache dir
* append organization name to parent's repo_url when parsing repos
* only eval config scripts and apply overrides once per app (#1879)
* only eval config scripts and apply overrides once per app
* move new resource behaviour to rebar_resource_v2 and keep v1
* cleanup use of rebar_resource module and unused functions
* cleanup error messages and unused code
* when discovering apps support mix packages as unbuilt apps (#1882)
* use hex_core tarball unpacking support in pkg resource (#1883)
* use hex_core tarball unpacking support in pkg resource
* ignore etag if package doesn't exist and delete if checksum fails
* add back tests for bad package checksums
* improve bad registry checksum error message
This requires some fancy dynamic work since the logger is started as
part of the kernel and we lost the sys.config from users when working
from there.
We start conservatively by making it an optional thing, turning it on
only where we know it to be safe.
The changes are applied _after_ having loaded the rest of configs so if
an inoffensive error happens, the shell works (with a bad error message)
rather than plain exploding.
Rather than the caller having to think of what to purge or not, use
erlang:check_process_code/2 to detect if the caller (rebar3) may die
because of the operation.
If so, do a soft purge with a conditional delete instead of a hard purge
with a mandatory delete.
This replaces all deprecated function usage by alternative ones based on
a version switch enacted at compile time, preventing all warnings.
This will likely introduce some possible runtime errors in using a
Rebar3 compiled on OTP-20 or OTP-21 back in versions 19 and earlier, but
we can't really work around that.
A bunch of dependencies have been updated to support OTP-21 without
warnings as well.
When a global plugin is used both locally and within the project, there
are cases when the rebar3 program will hard crash (killed in do_boot).
This has been traced to plugin-handling in compilation, where the same
code path may be purged twice in a row without further reloading for the
compile operation.
This of course yields the result where the code handling on the VM kills
all processes holding references to the module in memory, in this case
the rebar3 process itself.
By deduplicating the paths first, we ensure at most one purge before
reloading plugins and paths, and this prevents a hard crash.
This is done through 3 main change groups:
- replacing `~s` by `~ts` in format strings, so that strings that
contain unicode are properly printed rather than crashing
- adding the `unicode` argument to all function of the `re` module to
ensure transformations on strings containing unicode data are valid
instead of crashing (see issue #1302)
- replacing `ec_cnv:to_binary/1` and `ec_cnv:to_list/1` with matching
functions in `rebar_utils`.
The last point has been done, rather than modifying and updating erlware
commons, because binary and list conversions can be a contentious
subject. For example, if what is being handled is actually bytes from a
given binary stream, then forcing a byte-oriented interpretation of the
data can corrupt it. As such, it does not appear safe to modify erlware
commons' conversion functions since it may not be safe for all its
users.
Instead, rebar3 reimplements a subset of them (only converting
atoms and chardata, ignoring numbers) with the explicit purpose of
handling unicode string data.
Tests were left as unchanged as possible. This may impact the ability to
run rebar3's own suites in a unicode path, but respects a principle of
least change for such a large patch.
rebar_file_utils:cp_r uses rebar_utils:escape_chars to ensure that the file names are safe to use, but it doesn't escape double and single quotes. If the file name includes those characters, they disappear when the shell processes them and we get "file not found" errors.
The main culprit here is eunit, that creates reports whose names are `TEST-file_"myfile.app".xml`, and I wish it didn't but I think escape_chars should still do its job all the way.
Includes improvments and function documentation for all modules (in
alphabetical order) up to rebar_core, and may have included more in
other modules as I saw fit to dig and understand more of the internals.
this is slower than the compile time check but i guess packaging rebars with
repos is still a thing and i think only the eunit and ct providers call it
anyways
- proper segregation of comparison between tuple terms and non-tuple
terms. Guards weren't specific enough and that meant the wrong clauses
of guards would be triggered
- proper deduplication of entries in the list. An additional N passes
are required (we co-opt the reverse step to be more efficient) because
while the original lists:umerge easily removes dupes, this is
requiring more logic here since `[a,{a,b},{a,b,c},a,{a,b,c}]` is a
possible interleaving and we'd want `[a,{a,b},{a,b,c}]` -- comparison
of direct neighbours isn't enough.
yida_young
Fred Hebert
Michael Klishin
Tristan Sloughter
Maxim Fedorov
Dmitri Vereshchagin
Jan Chochol
Anders Wei
Bryan Paxton
Bryan Paxton
Carl-Johan Kjellander
Vlad Dumitrescu
Artem Pervin
Alexander Sedov
alisdair sullivan
alisdair sullivan