This bug was due to an interaction between two optimizations. If we
attempt to flush the buffer before any bytes are used we refused.
However, in enc_ensure we were not checking whether the buffer was
actually flushed so we would allocate a new buffer for the request.
The easiest way to encounter this issue was by encoding a raw binary
longer than 2041 bytes (i.e., `jiffy:encode(<<"stuff...">>).`).
If the input contained a mismatched end-of-array/object, the stack
could become empty before a call to dec_curr, which would look
beyond the bounds of the stack. If the value at this invalid
position happened to be st_array, we would pop too much from the
stack and overwrite the data that came before it.
This commit fixes this by letting dec_pop return the previous
state or st_invalid if the stack is empty, letting us exit
gracefully if the state isn't what we expect it to be.
dec_pop_assert is identical to the old dec_pop, tearing down the
emulator on internal errors.
number of items on `hexvals` is 128 while table size is 256, so
remaining 128 items are filled with zero. As a result, values in
\xf0-\xff will be treated as zero while should be rejected.
Some users of Jiffy have experienced issues when decoding large JSON
documents. Normally Jiffy expects smallish documents and returns any
strings as sub-binaries. When dealing with large documents these
sub-binary references can keep a large amount of RAM around unless the
user goes through and applies `binary:copy/1` on every string returned
from Jiffy. This however causes a large amount of CPU usage to do
something that Jiffy could do as it builds the JSON structure.
The `copy_strings` decoder option does exactly this. Instead of
returning sub-binaries Jiffy now copies every string into a newly
allocated binary. Users report that this fixes the memory issues while
also not negatively affecting performance significantly.