K.K. K.K. - 1 month ago 15x
Python Question

lxml XPath - extracting text from multi p nodes

Please have a look at lxml XPath position() does not work first.

Since XPath does not support to extract text from multi nodes, I decided to write for loop to get 30 stuffs.

for i in range(1,31):
content = "string(//div[@id='article']/p[" + (print(i)) + "]/.)"

I imagined it would return like,


However, obviously it does not work as I expected.. I got error message as following.

TypeError: Can't convert 'NoneType' object to str implicitly

What should I do? Any other elegant approach to solve this problem?


In Python3, print is a function which prints to the screen and returns None. (In Python2, print is a statement and the code would have raised an error since you can't put a statement in the middle of an expression.) Instead, to build a string use the format method:

content = "string(//div[@id='article']/p[{}]/.)".format(i)

And by the way, you should be able to use position() just fine with lxml. For instance,

import lxml.html as LH
content = '''\
        <title lang="eng">Harry Potter</title>
        <title lang="eng">Learning XML</title>
        <title lang="eng">Things Fall Apart</title>
        <title lang="eng">Blood Meridian</title>
root = LH.fromstring(content)

# Compare with http://stackoverflow.com/a/39242701/190597
print(root.xpath('//book[position()>=1 and position()<=last()]/title/text()'))
# ['Harry Potter', 'Learning XML', 'Things Fall Apart', 'Blood Meridian']

# But note that it is equivalent to 
# ['Harry Potter', 'Learning XML', 'Things Fall Apart', 'Blood Meridian']



['Harry Potter', 'Learning XML']

which shows that you can select the first N books without having to loop.

As Tomalak mentions, the XPath string function only returns the string representation of the first node. For example,


only prints

Harry Potter

If you want a list of strings, then don't use string.

If, as Daniel Haley points out, the desired text is in a mixture of nested nodes and child elements, e.g. <title lang="eng">Harry <b>Potter</b></title>, then you can extract the desired text using the text_content method:

[title.text_content() for title in root.xpath('//book[position()<3]/title')]