SisMaker
/
ibrowse
1
0
複製 0
程式碼 問題管理 0 合併請求 0 Projects 0 版本發佈 30 Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
303 Commit
6 分支
1.1 MiB
 
 
目錄樹: d775155a13
分支列表 標籤列表
${ item.name }
Create branch ${ searchTerm }
from 'd775155a13'
${ noResults }
Nick Vatamaniuc d775155a13
Increase timeout in pipelining test (#174) 		3 年之前
doc Added stream_full_chunks option 8 年之前
include Merge branch 'improve_pipeline_balance' of https://github.com/benjaminplee/ibrowse into merge_pull_req_123 9 年之前
priv Tagging master with new version 4.0.2 12 年之前
src Add strictly monotonic timestamp (#173) 3 年之前
test Increase timeout in pipelining test (#174) 3 年之前
.gitignore Bug fixes 9 年之前
.travis.yml add erlang 19, 20 for test 7 年之前
BSD_LICENSE Version 4.1.0 11 年之前
CHANGELOG Prefer IPv4 by default. Use the option `{prefer_ipv6, true}` to force ibrowse to use ipv6 where available. 6 年之前
CONTRIBUTORS new release bundling community contributions 6 年之前
LICENSE Version 4.1.0 11 年之前
Makefile Bug fix for spurious timeout messages being sent to the client in certain cases. Other cleanup 9 年之前
README.md README: Use SVG badge (#168) 4 年之前
bootstrap_travis.sh fix travis tests 9 年之前
erlang.mk Bug fix for spurious timeout messages being sent to the client in certain cases. Other cleanup 9 年之前
rebar.config add erlang 19, 20 for test 7 年之前
rebar.lock move to rebar3 9 年之前

README.md

ibrowse Build Status

ibrowse is a HTTP client written in erlang.

License: ibrowse is available under two different licenses. LGPL or the BSD license.

Comments to: chandrashekhar.mullaparthi@gmail.com

Current Version: 4.4.2

Latest Version: git://github.com/cmullaparthi/ibrowse.git

Features

  • RFC2616 compliant (AFAIK)
  • supports GET, POST, OPTIONS, HEAD, PUT, DELETE, TRACE, MKCOL, PROPFIND, PROPPATCH, LOCK, UNLOCK, MOVE and COPY
  • Understands HTTP/0.9, HTTP/1.0 and HTTP/1.1
  • Understands chunked encoding
  • Can generate requests using Chunked Transfer-Encoding
  • Pools of connections to each webserver
  • Pipelining support
  • Download to file
  • Asynchronous requests. Responses are streamed to a process
  • Basic authentication
  • Supports proxy authentication
  • Supports SOCKS5
    • Authentication methods 0 (No authentication) and 2(Username/password) supported
  • Can talk to secure webservers using SSL
  • Any other features in the code not listed here :)

Usage Examples

Remember to start ibrowse first:

5> ibrowse:start().
{ok,<0.94.0>}

Synchronous Requests

A simple GET request:

6> ibrowse:send_req("http://intranet/messenger/", [], get).
{ok,"200",
    [{"Server","Microsoft-IIS/5.0"},
     {"Content-Location","http://intranet/messenger/index.html"},
     {"Date","Fri, 17 Dec 2004 15:16:19 GMT"},
     {"Content-Type","text/html"},
     {"Accept-Ranges","bytes"},
     {"Last-Modified","Fri, 17 Dec 2004 08:38:21 GMT"},
     {"Etag","\"aa7c9dc313e4c41:d77\""},
     {"Content-Length","953"}],
    "<html>...</html>"}

A GET using a proxy:

7> ibrowse:send_req("http://www.google.com/", [], get, [],
                 [{proxy_user, "XXXXX"},
                  {proxy_password, "XXXXX"},
                  {proxy_host, "proxy"},
                  {proxy_port, 8080}], 1000).
{ok,"302",
    [{"Date","Fri, 17 Dec 2004 15:22:56 GMT"},
     {"Content-Length","217"},
     {"Content-Type","text/html"},
     {"Set-Cookie",
      "PREF=ID=f58155c797f9..."},
     {"Server","GWS/2.1"},
     {"Location",
      "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D110329..."},
     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],
    "<HTML>...</HTML>\r\n"}

A GET response saved to file. A temporary file is created and the filename returned. The response will only be saved to file if the status code is in the 200 range. The directory to download to can be set using the application env var download_dir - the default is the current working directory:

8> ibrowse:send_req("http://www.erlang.se/", [], get, [],
                 [{proxy_user, "XXXXX"},
                  {proxy_password, "XXXXX"},
                  {proxy_host, "proxy"},
                  {proxy_port, 8080},
                  {save_response_to_file, true}], 1000).
{error,req_timedout}

9> ibrowse:send_req("http://www.erlang.se/", [], get, [],
                 [{proxy_user, "XXXXX"},
                  {proxy_password, "XXXXX"},
                  {proxy_host, "proxy"},
                  {proxy_port, 8080},
                  {save_response_to_file, true}], 5000).
{ok,"200",
    [{"Transfer-Encoding","chunked"},
     {"Date","Fri, 17 Dec 2004 15:24:36 GMT"},
     {"Content-Type","text/html"},
     {"Server","Apache/1.3.9 (Unix)"},
     {"Via","1.1 netapp01 (NetCache NetApp/5.5R2)"}],
    {file,"/Users/chandru/code/ibrowse/src/ibrowse_tmp_file_1103297041125854"}}

Setting the size of the connection pool and pipeline. This sets the number of maximum connections to the specified server to 10 and the pipeline size to 1. Connections are assumed to be already setup.

11> ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10},
                                             {max_pipeline_size, 1}]).
ok

Example using the HEAD method:

56> ibrowse:send_req("http://www.erlang.org", [], head).
{ok,"200",
    [{"Date","Mon, 28 Feb 2005 04:40:53 GMT"},
     {"Server","Apache/1.3.9 (Unix)"},
     {"Last-Modified","Thu, 10 Feb 2005 09:31:23 GMT"},
     {"Etag","\"8d71d-1efa-420b29eb\""},
     {"Accept-ranges","bytes"},
     {"Content-Length","7930"},
     {"Content-Type","text/html"}],
    []}

Example using the OPTIONS method:

62> ibrowse:send_req("http://www.sun.com", [], options).
{ok,"200",
    [{"Server","Sun Java System Web Server 6.1"},
     {"Date","Mon, 28 Feb 2005 04:44:39 GMT"},
     {"Content-Length","0"},
     {"P3p",
      "policyref=\"http://www.sun.com/p3p/Sun_P3P_Policy.xml\", CP=\"CAO DSP COR CUR ADMa DEVa TAIa PSAa PSDa CONi TELi OUR  SAMi PUBi IND PHY ONL PUR COM NAV INT DEM CNT STA POL PRE GOV\""},
     {"Set-Cookie",
      "SUN_ID=X.X.X.X:169191109565879; EXPIRES=Wednesday, 31-Dec-2025 23:59:59 GMT; DOMAIN=.sun.com; PATH=/"},
     {"Allow",
      "HEAD, GET, PUT, POST, DELETE, TRACE, OPTIONS, MOVE, INDEX, MKDIR, RMDIR"}],
    []}

Asynchronous Requests

Example of an asynchronous GET request:

18> ibrowse:send_req("http://www.google.com", [], get, [],
                     [{proxy_user, "XXXXX"},
                      {proxy_password, "XXXXX"},
                      {proxy_host, "proxy"},
                      {proxy_port, 8080},
                      {stream_to, self()}]).
{ibrowse_req_id,{1115,327256,389608}}

19> flush().
Shell got {ibrowse_async_headers,{1115,327256,389608},
           "302",
           [{"Date","Thu, 05 May 2005 21:06:41 GMT"},
            {"Content-Length","217"},
            {"Content-Type","text/html"},
            {"Set-Cookie",
             "PREF=ID=b601f16bfa32f071:CR=1:TM=1115327201:LM=1115327201:S=OX5hSB525AMjUUu7; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com"},
            {"Server","GWS/2.1"},
            {"Location",
             "http://www.google.co.uk/cxfer?c=PREF%3D:TM%3D1115327201:S%3DDS9pDJ4IHcAuZ_AS&prev=/"},
            {"Via",
             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}
Shell got {ibrowse_async_response,{1115,327256,389608},
           "<HTML>...</HTML>\r\n"}
Shell got {ibrowse_async_response_end,{1115,327256,389608}}
ok

Another asynchronous GET request:

24> ibrowse:send_req("http://yaws.hyber.org/simple_ex2.yaws", [], get, [],
                     [{proxy_user, "XXXXX"},
                      {proxy_password, "XXXXX"},
                      {proxy_host, "proxy"},
                      {proxy_port, 8080},
                      {stream_to, self()}]).
{ibrowse_req_id,{1115,327430,512314}}

25> flush().
Shell got {ibrowse_async_headers,{1115,327430,512314},
           "200",
           [{"Date","Thu, 05 May 2005 20:58:08 GMT"},
            {"Content-Length","64"},
            {"Content-Type","text/html;charset="},
            {"Server",
             "Yaws/1.54 Yet Another Web Server"},
            {"Via",
             "1.1 hatproxy01 (NetCache NetApp/5.6.2)"}]}
Shell got {ibrowse_async_response,{1115,327430,512314},
           "<html>...</html>\n"}
Shell got {ibrowse_async_response_end,{1115,327430,512314}}

Example of request which fails when using the async option. Here the {ibrowse_req_id, ReqId} is not returned. Instead the error code is returned.

68> ibrowse:send_req("http://www.earlyriser.org", [], get, [], [{stream_to, self()}]).
{error,conn_failed}

Other Examples

Example of request using both Proxy-Authorization and authorization by the final webserver:

17> ibrowse:send_req("http://www.erlang.se/lic_area/protected/patches/erl_756_otp_beam.README",
                     [], get, [],
                     [{proxy_user, "XXXXX"},
                      {proxy_password, "XXXXX"},
                      {proxy_host, "proxy"},
                      {proxy_port, 8080},
                      {basic_auth, {"XXXXX", "XXXXXX"}}]).
{ok,"200",
    [{"Accept-Ranges","bytes"},
     {"Date","Thu, 05 May 2005 21:02:09 GMT"},
     {"Content-Length","2088"},
     {"Content-Type","text/plain"},
     {"Server","Apache/1.3.9 (Unix)"},
     {"Last-Modified","Tue, 03 May 2005 15:08:18 GMT"},
     {"ETag","\"1384c8-828-427793e2\""},
     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],
    "Patch Id:\t\terl_756_otp_beam\n..."}

Example of a TRACE request. Very interesting! yaws.hyber.org didn't support this. Nor did www.google.com. But good old BBC supports this:

37> ibrowse:send_req("http://www.bbc.co.uk/", [], trace, [],
                         [{proxy_user, "XXXXX"},
                          {proxy_password, "XXXXX"},
                          {proxy_host, "proxy"},
                          {proxy_port, 8080}]).
{ok,"200",
    [{"Transfer-Encoding","chunked"},
     {"Date","Thu, 05 May 2005 21:40:27 GMT"},
     {"Content-Type","message/http"},
     {"Server","Apache/2.0.51 (Unix)"},
     {"Set-Cookie",
      "BBC-UID=7452e72a..."},
     {"Set-Cookie",
      "BBC-UID=7452e72a..."},
     {"Via","1.1 hatproxy01 (NetCache NetApp/5.6.2)"}],
    "TRACE / HTTP/1.1\r\nHost: www.bbc.co.uk\r\nConnection: keep-alive\r\nX-Forwarded-For: 172.24.28.29\r\nVia: 1.1 hatproxy01 (NetCache NetApp/5.6.2)\r\nCookie: BBC-UID=7452e...\r\n\r\n"}

A GET using a socks5:

ibrowse:send_req("http://google.com", [], get, [],
  [{socks5_host, "127.0.0.1"},
  {socks5_port, 5335}]).

ibrowse:send_req("http://google.com", [], get, [],
  [{socks5_host, "127.0.0.1"},
  {socks5_port, 5335},
  {socks5_user, "user4321"},
  {socks5_password, "pass7654"}]).