V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
xdf331
V2EX  ›  Python

python 如何连续读取一个由服务器发来的非阻塞请求?

  •  
  •   xdf331 · 2014-11-22 18:39:29 +08:00 · 4725 次点击
    这是一个创建于 3680 天前的主题,其中的信息可能已经有所发展或是发生改变。
    tornado.web.xn--asynchronous,self-2s50ae30c16witdm75aj9cjstsv9boe7n1nycnnua.flush().">假设服务端采用tornado,对一个handler使用@tornado.web.asynchronous装饰器,并在一定时间内就self.flush()。

    python的客户端如何连续读取这样的请求?

    有人说使用pycurl,但是好像在win下需要别的dll,如何在不使用外部dll的下情况下实现?
    使用原生库最好了。
    7 条回复    2014-11-23 03:45:25 +08:00
    mulog
        1
    mulog  
       2014-11-22 20:12:56 +08:00
    什么叫客户端连续读取请求。。。?

    我想你是说非阻塞地发送请求? tornado不是带了个AsyncHTTPClient么
    mhycy
        2
    mhycy  
       2014-11-22 20:15:24 +08:00
    越说越复杂就是没说清事情本质.....
    lianghui
        3
    lianghui  
       2014-11-22 21:37:03 +08:00
    如果是tornado4, 可以用chunked transfer enoding传送,那么客户端支持chunk编码就可以做数据接收了。http://en.wikipedia.org/wiki/Chunked_transfer_encoding。

    tornado3不支持chunk:




    import tornado.ioloop
    import tornado.iostream
    import socket

    ```python
    class WebRequest(object):

    LINE_END = b'\r\n'
    def __init__(self, host, port=80, callback=None):
    self.host = host
    self.port =port
    self.headers = {}
    self.data = ''
    self.callback = callback
    self.transfer = 'stream'
    self.http_ver_with_status = ''

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
    self.stream = tornado.iostream.IOStream(s)
    self.stream.connect((self.host, self.port), self.send_request)


    def send_request(self):
    self.stream.write(b"GET / HTTP/1.0\r\nHost: %s\r\n\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36\r\n" %self.host)
    self.stream.read_until(self.LINE_END*2, self.on_headers)

    def on_headers(self, data):
    for line in data.split(self.LINE_END):
    parts = line.split(b":", 1)
    if len(parts) == 2:
    self.headers[parts[0].strip()] = parts[1].strip()
    else:
    self.http_ver_with_status = line
    if 'Content-Length' in self.headers:
    if int(self.headers[b"Content-Length"]) > 0:
    print self.headers
    print 'content_length: ', self.headers[b"Content-Length"]
    self.transfer = 'content_length'
    self.stream.read_bytes(int(self.headers[b"Content-Length"]), self.on_body)
    else:
    self.callback(self)
    self.stream.close()

    elif self.headers.get(b'Transfer-Encoding') == 'chunked':
    print 'chunk'
    self.transfer = 'chunked'
    self.on_chunk()
    else:
    self.transfer = 'stream'
    callback = lambda data: self.on_stream(data, True)
    streaming_callback = lambda data: self.on_stream(data, False)
    self.stream.read_until_close(callback, streaming_callback)

    def on_body(self, data):
    self.data += data
    self.callback(self)
    self.stream.close()

    def on_chunk(self):
    self.stream.read_until_regex(self.LINE_END, self.on_chunk_header)

    def on_chunk_header(self, data):
    length = int(data)
    if length > 0:
    self.stream.read_bytes(length+len(self.LINE_END), self.on_chunk_data)
    else:
    self.stream.read_bytes(length+len(self.LINE_END), self.on_chunk_end)


    def on_chunk_end(self, data):
    self.callback(self)
    self.stream.close()


    def on_chunk_data(self, data):
    print data
    self.data += data
    self.on_chunk()


    def on_stream(self, data, finish=False):
    # do some thing
    if finish:

    self.data += data
    self.callback(self)
    self.stream.close()
    else:
    self.data += data


    def callback(req):
    print req.http_ver_with_status
    print "Transfer :" , req.transfer
    print "Headers: ", req.headers
    print "Data: ", req.data
    req = WebRequest('cn.bing.com', 80, on_bing)


    def on_bing(req):
    print req.http_ver_with_status
    print "Transfer :" , req.transfer
    print "Headers: ", req.headers
    print "Data: ", req.data
    tornado.ioloop.IOLoop.instance().stop()

    if __name__ == '__main__':
    req = WebRequest('www.baidu.com', 80, callback)
    tornado.ioloop.IOLoop.instance().start()
    ```
    xdf331
        5
    xdf331  
    OP
       2014-11-22 22:48:31 +08:00 via Android   ❤️ 1
    @lianghui
    问题已经解决,和你的方法差不多,也是使用socket来模仿一个HTTP请求。但是我没有用ttornado.IOstream进行包装,我是直接用socket.recv(),不停的接收内容再处理就行了。
    lianghui
        6
    lianghui  
       2014-11-22 23:05:29 +08:00
    @xdf331 复杂的,可能还需要处理follow,以及if-modify-sine等,和断点请求
    ryd994
        7
    ryd994  
       2014-11-23 03:45:25 +08:00 via Android
    request.read(1)行吗?
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1425 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 23ms · UTC 17:21 · PVG 01:21 · LAX 09:21 · JFK 12:21
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.