请问爬虫高手为什么每次 scrapy 用 xpath 的 extract_first() 或者 get() 总是会有这个 class 的前缀出现???

[helloworld000]

比如 stackoverflow 的 这个 userpage https://stackoverflow.com/users/542251/liam

我要得到用户的自我介绍这个 class 里的内容

response.xpath("//div[@class='grid--cell mt16 s-prose profile-user--bio ']").get()

最后的结果(如下)

'<div class="grid--cell mt16 s-prose profile-user--bio ">\r\n<p>Father, Husband, Rock Climber and Developer with 20 years experience in the industry (in that order).</p>\n\n<hr>\n\n<p><strong>Any fool can write code that a computer can understand. Good programmers write code that humans can understand.</strong></p>\n\n<p><a href="https://en.wikiquote.org/wiki/Martin_Fowler" rel="nofollow noreferrer">Martin Fowler (2008)</a></p>\n\n<hr>\n\n<p>Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. <strong>We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil</strong>. Yet we should not pass up our opportunities in that critical 3%.</p>\n\n<p><a href="https://en.wikiquote.org/wiki/Donald_Knuth" rel="nofollow noreferrer">Donald Ervin Knuth (Professor Emeritus at Stanford University, and winner of the 1974 Turing Award)</a></p>\n\n<hr>\n    </div>'

怎么前面和后面还是有这个<div class="grid--cell mt16 s-prose profile-user--bio ">?

难道这个前缀不应该没了吗? 如果我在后面加了个 text()的话就只剩下 ( '\r\n' ) 了...

scrapy 里的 xpath 不能像 lxml 里的 html 一样直接取 text_content()吗?

[crella]

看看有没有.innerHtml 的函数？不懂 python

[Kobayashi]

不能。把标签内所有内容拼接起来一次返回，目前只有 lxml.html 和 bs4 支持，后者其实也是封装前者。

scrapy 内置解析器是 parsel，也是依赖 lxml 。先取出这个 div HTML 代码，临时引入 lxml.html 包装为 HTML Element，再调用 text_content() 就可以了。

[Guidance3204]

xpath 写的不对，大概应该是 //div[@class="classname"]//p//text(), 数据没在 <div> 下面，在 <p> 里面呢