Skip to content

Frequent Asked Questions

Why do selectors not work the same in Beautiful Soup 4.7+?

Soup Sieve is the official CSS selector library in Beautiful Soup 4.7+, and with this change, Soup Sieve introduces a number of changes that break some of the expected behaviors that existed in versions prior to 4.7.

In short, Soup Sieve follows the CSS specifications fairly close, and this broke a number of non-standard behaviors. These non-standard behaviors were not allowed according to the CSS specifications. Soup Sieve has no intentions of bringing back these behaviors.

For more details on specific changes, and the reasoning why a specific change is considered a good change, or simply a feature that Soup Sieve cannot/will not support, see Beautiful Soup Differences.

How does <template> handling work?

New in 4.0

The contents of an HTML <template> are not part of the document. They are held for later use, and in a web browser, they are kept in a separate document fragment that is available from the content property of the element. Soup Sieve treats them the same way: selectors do not descend into a <template>, and elements inside one can't see anything outside of it. This is true of everything, including combinators and pseudo-classes like :lang(), :disabled, and :default. A <template> itself is an element like any other that can be selected, but it is considered to have no children.

Beautiful Soup has no content property, so to select the contents, call select on the <template>.

>>> from bs4 import BeautifulSoup as bs
>>> html = """
... <html>
... <head></head>
... <body>
... <div>
... <template><p>Template content</p></template>
... </div>
... </body>
... </html>
... """
>>> soup = bs(html, 'html5lib')
>>> print(soup.select('p'))
[]
>>> print(soup.template.select('p'))
[<p>Template content</p>]
>>> print(soup.template.select('div p'))
[]

Any <template> within the contents is not entered, select from that <template> to get its contents.

This only applies to HTML elements. Something like <svg><template> is not an HTML <template>, so it has normal children, but only parsers that provide namespaces can make that distinction.

How does <iframe> handling work?

Changed in 4.0

In a web browser, the document that an <iframe> displays comes from its src or srcdoc attribute and is kept separately from the document the <iframe> is in. It is not the content of the <iframe> element, and selectors called on the outer document never reach it. Beautiful Soup does not load these documents, so Soup Sieve never sees them.

What is between <iframe> and </iframe> is something else. Browsers parse it as text, and recent versions of Python's html.parser, lxml, and html5lib do the same, so in HTML, an <iframe> only ever contains text. Like in a browser, that text is part of the <iframe>: the <iframe> is not :empty, the text is used to find the direction of an element with dir="auto", and :-soup-contains() sees it.

Note

Older versions of parsers may not escape raw HTML under <iframe> elements. In those cases, the raw elements under it will be treated like normal children.

>>> from bs4 import BeautifulSoup as bs
>>> html = """
... <html>
... <head></head>
... <body>
... <div>
... <iframe><p>Frame content</p></iframe>
... </div>
... </body>
... </html>
... """
>>> soup = bs(html, 'html.parser')
>>> print(soup.select('iframe p'))
[]
>>> print(soup.select('iframe:-soup-contains("<p>")'))
[<iframe>&lt;p&gt;Frame content&lt;/p&gt;</iframe>]

Elements only end up inside an <iframe> in XHTML, older versions of a parser, or when they are added to the tree by hand. Browsers treat these as normal children of the <iframe>, and so does Soup Sieve. They can be selected with combinators like any other element, and they are part of the same document, so they inherit things like the language, direction, and editability of the elements around them, and they belong to the forms around them.