In CouchDB's CI the small_pipeline test intermittently fails with
something like:
```
in function ibrowse_tests:'-small_pipeline/0-fun-5-'/1 (test/ibrowse_tests.erl, line 150)
{line,150},
{expression,"Counts"},
{expected,"\n\n\n\n\n\n\n\n\n\n"},
{value,"\n\n\n\n\n\n\n\n\t\n"}]}
```
Increasing the timeout a bit seems to help in local tests.
Algorithm change had bug where ets:select return
value was incorrectly assumed to be the object key
and not the entire object causing the following
delete attempt based on a matchspec and the key to fail.
This meant that ets was not updated to reflect the completed
requests on each connection and causing exaustion of pipelines
event though connections were idle. Functional test added which
demonstrated the problem.
Big commit. Switched algorithm to one which will favor
the connection with the smallest pipeline first
(deciding ties by timestamp of last finished request,
and then by pid as ultimate tie breaker).
Note: this also drastically changes the internal
representation of the connection in ets and is dependent
on specific order of operations when changing key values
to limit risk of race conditions between loadbalancer
and a given connection.
Also removed connection reporting of start of request
as this was no longer necessary since the load balancer
tees up the entry into ets with a 1.
Nick Vatamaniuc
getong
Chandrashekhar Mullaparthi
benjaminplee