lxml iterparse in python can't handle namespaces

Question

from lxml import etree
import StringIO

data= StringIO.StringIO('<root xmlns="http://some.random.schema"><a>One</a><a>Two</a><a>Three</a></root>')
docs = etree.iterparse(data,tag='a')
a,b = docs.next()


Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "iterparse.pxi", line 478, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:95348)
  File "iterparse.pxi", line 534, in lxml.etree.iterparse._read_more_events (src/lxml/lxml.etree.c:95938)
StopIteration

Works fine untill I add the namespace to the root node. Any ideas as to what I can do as a work around, or the correct way of doing this? I need to be event driven due to very large files.

Which part is the namespace , please ?

eyquem
– eyquem

2011-08-10 22:32:37 +00:00
Commented Aug 10, 2011 at 22:32 — eyquem
– eyquem, Commented Aug 10, 2011 at 22:32
xmlns="http://some.random.schema"

James Townley
– James Townley

2011-08-10 22:34:31 +00:00
Commented Aug 10, 2011 at 22:34 — James Townley
– James Townley, Commented Aug 10, 2011 at 22:34

jwhitlock · Accepted Answer · 2019-11-13 21:08:38Z

14

When there is a namespace attached, the tag isn't a, it's {http://some.random.schema}a. Try this (Python 3):

from lxml import etree
from io import BytesIO

xml = '''\
<root xmlns="http://some.random.schema">
  <a>One</a>
  <a>Two</a>
  <a>Three</a>
</root>'''
data = BytesIO(xml.encode())
docs = etree.iterparse(data, tag='{http://some.random.schema}a')
for event, elem in docs:
    print(f'{event}: {elem}')

or, in Python 2:

from lxml import etree
from StringIO import StringIO

xml = '''\
<root xmlns="http://some.random.schema">
  <a>One</a>
  <a>Two</a>
  <a>Three</a>
</root>'''
data = StringIO(xml)
docs = etree.iterparse(data, tag='{http://some.random.schema}a')
for event, elem in docs:
    print event, elem

This prints something like:

end: <Element {http://some.random.schema}a at 0x10941e730>
end: <Element {http://some.random.schema}a at 0x10941e8c0>
end: <Element {http://some.random.schema}a at 0x10941e960>

As @mihail-shcheglov pointed out, a wildcard * can also be used, which works for any or no namespace:

from lxml import etree
from io import BytesIO

xml = '''\
<root xmlns="http://some.random.schema">
  <a>One</a>
  <a>Two</a>
  <a>Three</a>
</root>'''
data = BytesIO(xml.encode())
docs = etree.iterparse(data, tag='{*}a')
for event, elem in docs:
    print(f'{event}: {elem}')

See lxml.etree docs for more.

edited Nov 13, 2019 at 21:08

answered Aug 10, 2011 at 23:40

jwhitlock

4,8824 gold badges43 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Partap Davis Over a year ago

That's what I've been doing, but now I have the problem that I can't use xpath because the tag names are invalid.

Mikhail Shcheglov Over a year ago

Also you can use wildcard to ignore namespace (link to documentation): docs=etree.iterparse(data, tag='{*}a')

eyquem · Accepted Answer · 2011-08-10 23:38:31Z

Why not with a regular expression ?

1)

Using lxml is slower than using a regex.

from time import clock
import StringIO



from lxml import etree

times1 = []
for i in xrange(1000):
    data= StringIO.StringIO('<root ><a>One</a><a>Two</a><a>Three\nlittle pigs</a><b>Four</b><a>another</a></root>')
    te = clock()
    docs = etree.iterparse(data,tag='a')
    tf = clock()
    times1.append(tf-te)
print min(times1)

print [etree.tostring(y) for x,y in docs]




import re

regx = re.compile('<a>[\s\S]*?</a>')

times2 = []
for i in xrange(1000):
    data= StringIO.StringIO('<root ><a>One</a><a>Two</a><a>Three\nlittle pigs</a><b>Four</b><a>another</a></root>')
    te = clock()
    li = regx.findall(data.read())
    tf = clock()
    times2.append(tf-te)
print min(times2)

print li

result

0.000150298431784
['<a>One</a>', '<a>Two</a>', '<a>Three\nlittle pigs</a>', '<a>another</a>']
2.40253998762e-05
['<a>One</a>', '<a>Two</a>', '<a>Three\nlittle pigs</a>', '<a>another</a>']

0.000150298431784 / 2.40253998762e-05 is 6.25
lxml is 6.25 times slower than a regex

.

2)

No problem if namespace:

import StringIO
import re

regx = re.compile('<a>[\s\S]*?</a>')

data= StringIO.StringIO('<root xmlns="http://some.random.schema"><a>One</a><a>Two</a><a>Three\nlittle pigs</a><b>Four</b><a>another</a></root>')
print regx.findall(data.read())

result

['<a>One</a>', '<a>Two</a>', '<a>Three\nlittle pigs</a>', '<a>another</a>']

Collectives™ on Stack Overflow

lxml iterparse in python can't handle namespaces

2 Answers 2

2 Comments

1)

2)

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1)

2)

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related