SisMaker
/
ibrowse

# ibrowse [![Build Status](https://secure.travis-ci.org/cmullaparthi/ibrowse.png)](http://travis-ci.org/cmullaparthi/ibrowse)

ibrowse is a HTTP client written in erlang.
**License:** ibrowse is available under two different licenses.  LGPL or the BSD license.
**Comments to:** chandrashekhar.mullaparthi@gmail.com
**Current Version:** 4.2
**Latest Version:** git://github.com/cmullaparthi/ibrowse.git


## Features

*  [RFC2616](http://www.ietf.org/rfc/rfc2616.txt) compliant (AFAIK)*  supports GET, POST, OPTIONS, HEAD, PUT, DELETE, TRACE,   MKCOL, PROPFIND, PROPPATCH, LOCK, UNLOCK, MOVE and COPY*  Understands HTTP/0.9, HTTP/1.0 and HTTP/1.1*  Understands chunked encoding*  Can generate requests using [Chunked Transfer-Encoding](http://en.wikipedia.org/wiki/Chunked_transfer_encoding)*  Pools of connections to each webserver*  Pipelining support*  Download to file*  Asynchronous requests. Responses are streamed to a process*  Basic authentication*  Supports proxy authentication*  Supports socks5*  Can talk to secure webservers using SSL*  *Any other features in the code not listed here :)*


## Usage Examples

Remember to start ibrowse first:
```erlang5> ibrowse:start().{ok,<0.94.0>}```


### Synchronous Requests

A simple `GET` request:
```erlang6> ibrowse:send_req("http://intranet/messenger/", [], get).{ok,"200",    [{"Server","Microsoft-IIS/5.0"},     {"Content-Location","http://intranet/messenger/index.html"},     {"Date","Fri, 17 Dec 2004 15:16:19 GMT"},     {"Content-Type","text/html"},     {"Accept-Ranges","bytes"},     {"Last-Modified","Fri, 17 Dec 2004 08:38:21 GMT"},     {"Etag","\"aa7c9dc313e4c41:d77\""},     {"Content-Length","953"}],    "<html>...</html>"}```

A `GET` using a proxy:
```erlang7> ibrowse:send_req("http://www.google.com/", [], get, [],                 [{proxy_user, "XXXXX"},                  {proxy_password, "XXXXX"},                  {proxy_host, "proxy"},                  {proxy_port, 8080}], 1000).{ok,"302",    [{"Date","Fri, 17 Dec 2004 15:22:56 GMT"},     {"Content-Length","217"},     {"Content-Type","text/html"},     {"Set-Cookie",      "PREF=ID=f58155c797f9..."},     {"Server","GWS/2.1"},     {"Location",      "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D110329..."},     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],    "<HTML>...</HTML>\r\n"}```

A `GET` response saved to file. A temporary file is created and thefilename returned. The response will only be saved to file if thestatus code is in the `200` range. The directory to download to canbe set using the application env var `download_dir` - the defaultis the current working directory:
```erlang8> ibrowse:send_req("http://www.erlang.se/", [], get, [],                 [{proxy_user, "XXXXX"},                  {proxy_password, "XXXXX"},                  {proxy_host, "proxy"},                  {proxy_port, 8080},                  {save_response_to_file, true}], 1000).{error,req_timedout}
9> ibrowse:send_req("http://www.erlang.se/", [], get, [],                 [{proxy_user, "XXXXX"},                  {proxy_password, "XXXXX"},                  {proxy_host, "proxy"},                  {proxy_port, 8080},                  {save_response_to_file, true}], 5000).{ok,"200",    [{"Transfer-Encoding","chunked"},     {"Date","Fri, 17 Dec 2004 15:24:36 GMT"},     {"Content-Type","text/html"},     {"Server","Apache/1.3.9 (Unix)"},     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],    {file,"/Users/chandru/code/ibrowse/src/ibrowse_tmp_file_1103297041125854"}}```

Setting the size of the connection pool and pipeline. This sets thenumber of maximum connections to the specified server to `10` and the pipelinesize to `1`. Connections are assumed to be already setup.
```erlang11> ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10},                                             {max_pipeline_size, 1}]).ok```

Example using the `HEAD` method:
```erlang56> ibrowse:send_req("http://www.erlang.org", [], head).{ok,"200",    [{"Date","Mon, 28 Feb 2005 04:40:53 GMT"},     {"Server","Apache/1.3.9 (Unix)"},     {"Last-Modified","Thu, 10 Feb 2005 09:31:23 GMT"},     {"Etag","\"8d71d-1efa-420b29eb\""},     {"Accept-ranges","bytes"},     {"Content-Length","7930"},     {"Content-Type","text/html"}],    []}```

Example using the `OPTIONS` method:
```erlang62> ibrowse:send_req("http://www.sun.com", [], options).{ok,"200",    [{"Server","Sun Java System Web Server 6.1"},     {"Date","Mon, 28 Feb 2005 04:44:39 GMT"},     {"Content-Length","0"},     {"P3p",      "policyref=\"http://www.sun.com/p3p/Sun_P3P_Policy.xml\", CP=\"CAO DSP COR CUR ADMa DEVa TAIa PSAa PSDa CONi TELi OUR  SAMi PUBi IND PHY ONL PUR COM NAV INT DEM CNT STA POL PRE GOV\""},     {"Set-Cookie",      "SUN_ID=X.X.X.X:169191109565879; EXPIRES=Wednesday, 31-Dec-2025 23:59:59 GMT; DOMAIN=.sun.com; PATH=/"},     {"Allow",      "HEAD, GET, PUT, POST, DELETE, TRACE, OPTIONS, MOVE, INDEX, MKDIR, RMDIR"}],    []}```


### Asynchronous Requests

Example of an asynchronous `GET` request:
```erlang18> ibrowse:send_req("http://www.google.com", [], get, [],                     [{proxy_user, "XXXXX"},                      {proxy_password, "XXXXX"},                      {proxy_host, "proxy"},                      {proxy_port, 8080},                      {stream_to, self()}]).{ibrowse_req_id,{1115,327256,389608}}
19> flush().Shell got {ibrowse_async_headers,{1115,327256,389608},           "302",           [{"Date","Thu, 05 May 2005 21:06:41 GMT"},            {"Content-Length","217"},            {"Content-Type","text/html"},            {"Set-Cookie",             "PREF=ID=b601f16bfa32f071:CR=1:TM=1115327201:LM=1115327201:S=OX5hSB525AMjUUu7; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com"},            {"Server","GWS/2.1"},            {"Location",             "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1115327201:S%3DDS9pDJ4IHcAuZ_AS&prev=/"},            {"Via",             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}Shell got {ibrowse_async_response,{1115,327256,389608},           "<HTML>...</HTML>\r\n"}Shell got {ibrowse_async_response_end,{1115,327256,389608}}ok```

Another asynchronous `GET` request:
```erlang24> ibrowse:send_req("http://yaws.hyber.org/simple_ex2.yaws", [], get, [],                     [{proxy_user, "XXXXX"},                      {proxy_password, "XXXXX"},                      {proxy_host, "proxy"},                      {proxy_port, 8080},                      {stream_to, self()}]).{ibrowse_req_id,{1115,327430,512314}}
25> flush().Shell got {ibrowse_async_headers,{1115,327430,512314},           "200",           [{"Date","Thu, 05 May 2005 20:58:08 GMT"},            {"Content-Length","64"},            {"Content-Type","text/html;charset="},            {"Server",             "Yaws/1.54 Yet Another Web Server"},            {"Via",             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}Shell got {ibrowse_async_response,{1115,327430,512314},           "<html>...</html>\n"}Shell got {ibrowse_async_response_end,{1115,327430,512314}}```

Example of request which fails when using the async option. Herethe `{ibrowse_req_id, ReqId}` is not returned. Instead the error code isreturned.
```erlang68> ibrowse:send_req("http://www.earlyriser.org", [], get, [], [{stream_to, self()}]).{error,conn_failed}```


### Other Examples

Example of request using both Proxy-Authorization and authorizationby the final webserver:
```erlang17> ibrowse:send_req("http://www.erlang.se/lic_area/protected/patches/erl_756_otp_beam.README",                     [], get, [],                     [{proxy_user, "XXXXX"},                      {proxy_password, "XXXXX"},                      {proxy_host, "proxy"},                      {proxy_port, 8080},                      {basic_auth, {"XXXXX", "XXXXXX"}}]).{ok,"200",    [{"Accept-Ranges","bytes"},     {"Date","Thu, 05 May 2005 21:02:09 GMT"},     {"Content-Length","2088"},     {"Content-Type","text/plain"},     {"Server","Apache/1.3.9 (Unix)"},     {"Last-Modified","Tue, 03 May 2005 15:08:18 GMT"},     {"ETag","\"1384c8-828-427793e2\""},     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],    "Patch Id:\t\terl_756_otp_beam\n..."}```

Example of a `TRACE` request. Very interesting! yaws.hyber.org didn'tsupport this. Nor did www.google.com. But good old BBC supports this:
```erlang37> ibrowse:send_req("http://www.bbc.co.uk/", [], trace, [],                         [{proxy_user, "XXXXX"},                          {proxy_password, "XXXXX"},                          {proxy_host, "proxy"},                          {proxy_port, 8080}]).{ok,"200",    [{"Transfer-Encoding","chunked"},     {"Date","Thu, 05 May 2005 21:40:27 GMT"},     {"Content-Type","message/http"},     {"Server","Apache/2.0.51 (Unix)"},     {"Set-Cookie",      "BBC-UID=7452e72a..."},     {"Set-Cookie",      "BBC-UID=7452e72a..."},     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],    "TRACE / HTTP/1.1\r\nHost: www.bbc.co.uk\r\nConnection: keep-alive\r\nX-Forwarded-For: 172.24.28.29\r\nVia: 1.1 hatproxy01 (NetCache NetApp/5.6.2)\r\nCookie: BBC-UID=7452e...\r\n\r\n"}```
A `GET` using a socks5:
```erlangibrowse:send_req("http://google.com", [], get, [],  [{socks5_host, "127.0.0.1"},  {socks5_port, 5335}]).
ibrowse:send_req("http://google.com", [], get, [],  [{socks5_host, "127.0.0.1"},  {socks5_port, 5335},  {socks5_user, "user4321"},  {socks5_pass, "pass7654"}]).```