Some users of Jiffy have experienced issues when decoding large JSON
documents. Normally Jiffy expects smallish documents and returns any
strings as sub-binaries. When dealing with large documents these
sub-binary references can keep a large amount of RAM around unless the
user goes through and applies `binary:copy/1` on every string returned
from Jiffy. This however causes a large amount of CPU usage to do
something that Jiffy could do as it builds the JSON structure.

The `copy_strings` decoder option does exactly this. Instead of
returning sub-binaries Jiffy now copies every string into a newly
allocated binary. Users report that this fixes the memory issues while
also not negatively affecting performance significantly.

You can no optionally request that keys are deduplicate inside of Jiffy
instead of having to perform that operation in Erlang.

When "\u"-escaping a Unicode character, the esc_extra value doesn't
need to include the number of bytes in the input string.  That is, if
a three-byte UTF-8 character is being escaped to a six-byte "\uXXXX"
sequence, esc_extra only needs to be increased by 3.

Originally reported by @NorthNick on apache/couchdb-jiffy.

Each call to enif_map_iterator_create() must be paired with a call to
enif_map_iterator_destroy(). Otherwise, we'll leak memory.

Fixes #112

In the original PR for `return_trailer` @vlm pointed out that I wasn't
using enif_consume_timeslice correctly. This fixes that by changing out
its called.

Previously we attempted to define the total number of bytes to decode or
encode in a single NIF call and then would consume as much of the
timeslice as we processed. This is wrong because we may start the NIF
call with less than an entire timeslice left.

The new approach is to define the number of bytes to encode or decode
per reduction and then iteratively call enif_consume_timeslice until it
indicates that we should return.

Previously Jiffy would throw an error about trailing data if there is
any non-whitespace character encounter after the first term had been
decoded.

This patch adds a decoder option `return_trailer` that will instead
return a sub-binary starting at the first non-whitespace character. This
allows users to be able to decode multiple terms from a single iodata()
term.

Thanks to @vlm for the original patch.

This brings back escaping forward slashes as an option during encoding.
Default is still not to escape.

This is pretty much a generalisation of the use_nil option to support an
arbitrary atom.

This sounds rather insane to me but I've managed to show that `(char)
-1` is converted to 255 on some platforms. This was reproduced on
ppc64el via Qemu on OS X. A simple program that does `fprintf(stderr,
"%d\r\n", (char) -1);` prints 255 to the console. Rather than rely on
the signedness of a char I've just updated things to use an unsigned
char (which hopefully is never signed) and replaced -1 with 255 for the
sentinel value when converting hex values.

Thanks to Balint Reczey (@rbalint) for the report.

Fixes #74

This fixes a leak when encoding a bare bignum. Technically it would be
possible to hit this memory leak randomly with bignums in objects but
the chances are highly unlikely.

Thanks to @miriampena for the issue.

Fixes #69

The `val` variable is a register value that we need to be able to return
at any time from `decode_iter`. If it happened that a yield was
triggered while processing trailing whitespace the lack of persistance
caused decode to return a term intialized from a random integer value.
Obviously the Erlang VM did not enjoy this.

Thanks to @michalpalka for the report.

Fixes #66

This updates encoder errors to report the actual Erlang value that
caused the error. This should make it easier to debug errors when
generating JSON.

I must've managed to miss the PR update from Stanislav the other day
when merging this.

This implements the `use_nil` option as discussed on issue #64. Passing
the atom `use_nil` as an option to both encode and decode will replace
the atom `null` with `nil` when decoding and encode `nil` as `null` when
encoding values.

Fixes #64
Fixes #68

Rather than worry about truncation casting from a possibly 64bit value
down to a possibly 32bit size_t we just limit the total bytes per
invocation to 4G using an unsigned integer.

Thanks to @seriyps for the report.

Fixes #61

This patch adds initial support for decoding/encoding to/from the new
maps data type.

I'd like to thank Jihyun Yu (yjh0502) for the initial versions of this
work.

This adds a configurable limit on the number of bytes produced by
the encoder before yielding back to the Erlang VM. This is to avoid the
infamous scheduler collapse issues.

The `jiffy:encode/2` now takes an option `{bytes_per_iter,
pos_integer()}` that controls the yield frequency. The default value is
2048.

This adds a configurable limit on the number of bytes consumed by
the decoder before yielding back to the Erlang VM. This is to avoid the
infamous scheduler collapse issues.

The `jiffy:decode/2` now takes an option `{bytes_per_iter,
pos_integer()}` that controls the yield frequency. The default value is
2048.

This is ground work to allow Jiffy to yield back to the scheduler.
Creating an encoder resource will allow for the necessary state to be
carried across NIF function invocations.

This is ground work to allow Jiffy to yield back to the scheduler.
Creating a decoder resource will allow for the necessary state to be
carried across NIF function invocations.

Properly decode them and accept to encode them.

This was clarified by Unicode Technical Committee:
http://www.unicode.org/versions/corrigendum9.html

Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>

Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>

Signed-off-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>

A single quote input was causing segfaults due to sneaking past the
string termination logic. This patch corrects that lapse in conditional
by only parsing strings where a closing quote was found. All other
strings are rejected as invalid.

Big thanks to Jean-Charles Campagne (@jccampagne) for reporting the
issue.

My truth table was a bit busted for extreme value doubles.

While using rebar's dependency management was surprisingly easy it
broke apps that tried to build Jiffy as a dependency due to relative
path #includes.

This also fixes a few other issues. Most notably it removes the use of
the ECMAScript compatible encoding due to JSON's lack of support for +/-
Inf and NaN.

Floating point numbers are no longer encoded as a one to one mapping
of their binary representation, but as short as possible (while still
being acurate). The double-conversion library [1] is used to do the
hard work.

The ECMAScript compatible conversion is used.

[1] https://code.google.com/p/double-conversion/

After a few requests and some reflection I've decided to stop escaping
forward slashes in strings while still accepting the escaped version
through the decoder. This appears to mimic the behavior of other popular
JSON libraries I've checked (Ruby and Python).

Apparently I missed this when refactoring a do/while loop into a
standard while loop. It was a harmless extra invocation to check if the
stack was empty.

As it turns out I did not understand the documenation for
load/upgrade/unload correctly. load/upgrade are called conditionally if
there's code in the VM for the NIF. Ie, no code means load is called,
where as if code exists, upgrade is called.

unload is called regardless once per load/unload. This means that
load/upgrade in Jiffy's case will each create a state object and unload
will free it each time. I was missing the fact that unload is called
every time and hence I don't need to clear the state in upgrade.

The docs aren't entirely clear on the order of calls for upgrades so
this is mostly just in case old_priv ever happens to not be what load
returned in priv.

By default Jiffy is quite strict in what it encodes. By default it will
not allow invalid UTF-8 to be produced. This can cause issues when
attempting to encode JSON that was decoded by other libraries as UTF-8
semantics are not uniformly enforced.

This patch adds an option 'force_utf8' to the encoder. If encoding hits
an error for an invalid string it will forcefully mutate the object to
contain only valid UTF-8 and return the resulting encoded JSON.

For the most part this means it will strip any garbage data from
binaries replacing it replacement codepoint U+FFFD. Although, it will
also try and the common error of encoding surrogate pairs as three-byte
sequences and reencode them into UTF-8 properly.

There was a bug that was resetting the extra amount of space needed when
encoding unicode characters with the `uescape` option enabled.

I had a silly direction mistake in a bit shift that was causing the high
portion of all combining characters to be printed as \uD800 which is
obviously wrong. This bug only affects people using the non-default
uescape option during encoding.

Numbers like 1.0 were being encoded as <<"1">> which can lead to a bit
of confusion. This merely checks if a decimial point exists and if not
it appends ".0" to the value.

jiffy:encode/2 now accepts a 'pretty' option that will format the JSON
output using two-space indents.

It was possible to pass some types of invalid UTF-8 through Jiffy's
encoder. Specifically, if uescaping isn't used, values that would decode
from 0xD800 to 0xDFFFF, 0xFFFE, 0xFFFF, and values greater than 0x10FFFF
would not be flagged as invalid. Now they are.

The encoding length for unicode values was applied to the wrong index
variable.

Thanks to Paul Guyot for the detailed bug report and fix.

Fixes #7