API
Soup Sieve implements most of the selectors from the stable specification and even many from the latest draft specification. Selectors can be used to detect and filter elements. To learn more about which specific selectors are implemented, see CSS Selectors.
Soup Sieve will detect the document type being used from the Beautiful Soup object that is given to it, and depending on the document type, its behavior may be slightly different.
When detecting XHTML, Soup Sieve simply looks to see if the root element of an XML document is under the XHTML namespace
and does not currently look at the doctype. If in the future there is a need for stricter XHTML detection, this may
change.
-
HTML document types (HTML, HTML5) will have their tag names and attribute names treated without case sensitivity, like most browsers do.
-
XML document types (including XHTML) will have their tag names and attribute names treated with case sensitivity.
-
HTML5, XHTML and XML documents will have namespaces evaluated per the document's support (provided via the parser). Some additional configuration is required when using namespaces, see Namespace for more information.
Getting Proper Namespaces
The
html5libparser provides proper namespaces for HTML5, butlxml's HTML parser will not. If you need namespace support for HTML5, consider usinghtml5lib.For XML, the
lxml-xmlparser (xmlfor short) will provide proper namespaces. It is generally suggested thatlxml-xmlis used to parse XHTML documents to take advantage of namespaces. -
While attribute values are generally treated as case sensitive, HTML5 and HTML treat the
typeattribute special. Thetypeattribute's value is always case insensitive. This is generally how most browsers treattype. If you needtypeto be sensitive, you can use thesflag:[type="submit" s].
While Soup Sieve access is exposed through Beautiful Soup's API, Soup Sieve's API can always be imported and accessed directly for more controlled tag selection if needed.
Selector Limits
Starting in 2.8.4, number of selectors in a given pattern are arbitrarily capped at ~8192. This limitation was added to prevent cases where impractical, massive selectors.
Some selectors are defined as a series of other pre-defined selectors which also contribute towards the count.
In 3.0, this has become a configurable option via the max_selectors parameter.
Flags
soupseive.DEBUG
Print debug output when parsing a selector.
>>> import soupsieve as sv
>>> sv.compile('p:has(#id) > span.some-class:-soup-contains(text)', flags=sv.DEBUG)
## PARSING: 'p:has(#id) > span.some-class:-soup-contains(text)'
TOKEN: 'tag' --> 'p' at position 0
TOKEN: 'pseudo_class' --> ':has(' at position 1
is_pseudo: True
is_open: True
is_relative: True
TOKEN: 'id' --> '#id' at position 6
TOKEN: 'pseudo_close' --> ')' at position 9
TOKEN: 'combine' --> ' > ' at position 10
TOKEN: 'tag' --> 'span' at position 13
TOKEN: 'class' --> '.some-class' at position 17
TOKEN: 'pseudo_contains' --> ':-soup-contains(text)' at position 28
## END PARSING
SoupSieve(pattern='p:has(#id) > span.some-class:-soup-contains(text)', namespaces=None, custom=None, flags=1)
soupseive.NOCACHE
Warning
Disabling the cache will disable performance optimizations and may open you up to poor performance. This is provided only to allow users to circumvent the cache if the cache is introducing issues itself, and should not be disabled in time critical systems where bad performance cannot be tolerated.
Disable caching optimizations. Results should be the same with or without except in cases where the tree is
intentionally mutated during lazy iterations such as iselect().
>>> from bs4 import BeautifulSoup
>>> text = """
... <div>
... <!-- These are animals -->
... <p class="a">Cat</p>
... <p class="b">Dog</p>
... <p class="c">Mouse</p>
... </div>
... """
>>> soup = BeautifulSoup(text, 'html5lib')
>>> import soupsieve as sv
>>> sv.select('p:nth-child(2n + 1)', soup)
[<p class="a">Cat</p>, <p class="c">Mouse</p>]
>>> sv.select('p:nth-child(2n + 1)', soup, flags=sv.NOCACHE)
[<p class="a">Cat</p>, <p class="c">Mouse</p>]
soupseive.NOSTRICT
New in 3.0
Warning
Disabling the strict mode will enable more complex behavior which may introduce poor performance. NOSTRICT should
not be enabled in time critical systems where bad performance cannot be tolerated.
Soup Sieve provides more strict rules by default as of version 3.0. To relax these rules, at the cost of possible
performance concerns, the NOSTRICT flag can be used to disable the strict rules and allow for some more flexible
rules. Any selectors whose rules become relaxed with no NOSTRICT will specifically make mention of it.
>>> from bs4 import BeautifulSoup
>>> text = """
... <div>
... <!-- These are animals -->
... <p class="a">Cat</p>
... <p class="b">Dog</p>
... <p class="c">Mouse</p>
... </div>
... """
>>> soup = BeautifulSoup(text, 'html5lib')
>>> import soupsieve as sv
>>> try:
... sv.select('div:has(> p:has(+ .b + .c))', soup)
... except sv.SelectorSyntaxError:
... print('Not allowed')
...
Not allowed
>>> sv.select('div:has(> p:has(+ .b + .c))', soup, flags=sv.NOSTRICT)
[<div>
<!-- These are animals -->
<p class="a">Cat</p>
<p class="b">Dog</p>
<p class="c">Mouse</p>
</div>]
soupsieve.select_one()
select_one will return the first tag under the given tag that matches the given CSS selectors provided, or it will
return None if a suitable tag was not found.
select_one accepts a CSS selector string, a Tag/BeautifulSoup object, an optional namespace
dictionary, and flags.
soupsieve.select()
def select(select, tag, namespaces=None, limit=0, flags=0, **kwargs):
"""Select the specified tags."""
select will return all tags under the given tag that match the given CSS selectors provided. You can also limit the
number of tags returned by providing a positive integer via the limit parameter (0 means to return all tags).
select accepts a CSS selector string, a Tag/BeautifulSoup object, an optional namespace dictionary,
a limit, and flags.
>>> import soupsieve as sv
>>> sv.select('p:is(.a, .b, .c)', soup)
[<p class="a">Cat</p>, <p class="b">Dog</p>, <p class="c">Mouse</p>]
soupsieve.iselect()
def iselect(select, node, namespaces=None, limit=0, flags=0, **kwargs):
"""Select the specified tags."""
iselect is exactly like select except that it returns a generator instead of a list.
soupsieve.closest()
def closest(select, tag, namespaces=None, flags=0, **kwargs):
"""Match closest ancestor to the provided tag."""
closest returns the tag closest to the given tag that matches the given selector. The element found must be a direct
ancestor of the tag or the tag itself.
closest accepts a CSS selector string, a Tag/BeautifulSoup object, an optional namespace
dictionary, and flags.
soupsieve.match()
The match function matches a given tag with a given CSS selector.
match accepts a CSS selector string, a Tag/BeautifulSoup object, an optional namespace dictionary,
and flags.
>>> nodes = sv.select('p:is(.a, .b, .c)', soup)
>>> sv.match('p:not(.b)', nodes[0])
True
>>> sv.match('p:not(.b)', nodes[1])
False
Matching Many Tags
Soup Sieve caches what it learns about a document while matching, such as the text of an element for
:-soup-contains() or whether it has a matching descendant for :has(), but only for the duration
of one call. Each call of match starts over, so calling it for each element of a large or deeply nested document
can take time that grows faster than the size of the document. To test many tags, use select,
or filter with a list of the tags, which share what they learn between the tags of the same
document.
soupsieve.filter()
filter takes an iterable containing HTML nodes and will filter them based on the provided CSS selector string. If
given a Tag/BeautifulSoup object, it will iterate the direct children filtering them.
filter accepts a CSS selector string, an iterable containing nodes, an optional namespace dictionary,
and flags.
When given a list, or any other iterable, the tags can be from different documents. The tags from the same document,
or from the same detached tree, share what Soup Sieve learns while matching, as one call of
select does, so filtering many tags of a document takes time that grows with the size of the
document, not faster. A selector that uses :scope or & is the exception: each tag is the scope when it is
matched, as with match, so each tag is matched separately.
soupsieve.escape()
escape is used to escape CSS identifiers. It follows the CSS specification and escapes any character that
would normally cause an identifier to be invalid.
>>> sv.escape(".foo#bar")
'\\.foo\\#bar'
>>> sv.escape("()[]{}")
'\\(\\)\\[\\]\\{\\}'
>>> sv.escape('--a')
'--a'
>>> sv.escape('0')
'\\30 '
>>> sv.escape('\0')
'�'
New in 1.9.0
escape is a new API function added in 1.9.0.
soupsieve.compile()
compile will pre-compile a CSS selector pattern returning a SoupSieve object. The SoupSieve object has the same
selector functions available via the module without the need to specify the selector, namespaces, or flags.
class SoupSieve:
"""Match tags in Beautiful Soup with CSS selectors."""
def match(self, tag):
"""Match."""
def closest(self, tag):
"""Match closest ancestor."""
def filter(self, iterable):
"""Filter."""
def select_one(self, tag):
"""Select a single tag."""
def select(self, tag, limit=0):
"""Select the specified tags."""
def iselect(self, tag, limit=0):
"""Iterate the specified tags."""
soupsieve.purge()
Soup Sieve caches compiled patterns for performance. If for whatever reason, you need to purge the cache, simply call
purge.
Custom Selectors
The custom selector feature is loosely inspired by the css-extensions proposal. In its current
form, Soup Sieve allows assigning a complex selector to a custom pseudo-class name. The pseudo-class name must start
with :-- to avoid conflicts with any future pseudo-classes.
To create custom selectors, you simply need to pass a dictionary containing the custom pseudo-class names (keys) with
the associated CSS selectors that the pseudo-classes are meant to represent (values). For best results, selectors should
be pre-compiled with soupsieve.CustomSelectors for best performance, especially if reusing these selectors.
It is important to remember that pseudo-class names are not case sensitive, so even though a dictionary will allow you to specify multiple keys with the same name (as long as the character cases are different), Soup Sieve will not and will throw an exception if you attempt to do so.
In the following example, we will define our own custom selector called :--header that will be an alias for
h1, h2, h3, h4, h5, h6.
>>> import soupsieve as sv
>>> import bs4
>>> markup = """
... <html>
... <body>
... <h1 id="1">Header 1</h1>
... <h2 id="2">Header 2</h2>
... <p id="3"></p>
... <p id="4"><span>child</span></p>
... </body>
... </html
... """
>>> soup = bs4.BeautifulSoup(markup, 'html5lib')
>>> sv.select(':--header', soup, custom=sv.CustomSelectors({':--header': 'h1, h2, h3, h4, h5, h6'}))
[<h1 id="1">Header 1</h1>, <h2 id="2">Header 2</h2>]
Custom selectors can also be dependent upon other custom selectors. The order is important, so if a selector depends on another custom selector, the selector should be defined after its dependency.
Change in Behavior 3.0
Previous to 3.0 order did not matter and custom selectors were compiled as needed recursively. Now all custom custom selectors will be compiled up front and must be organized such that selector dependencies must come first.
Assuming the same markup as in the first example, we will now create a custom selector that should find any element that
has child elements, we will call the selector :--parent. Then we will create another selector called
:--parent-paragraph that will use the :--parent selector to find <p> elements that are also
parents:
>>> custom = sv.CustomSelectors(
... {
... ":--parent": ":has(> *|*)",
... ":--parent-paragraph": "p:--parent"
... }
... )
>>> sv.select(':--parent-paragraph', soup, custom=custom)
[<p id="4"><span>child</span></p>]
Namespaces
Many of Soup Sieve's selector functions take an optional namespace dictionary. Namespaces, just like CSS, must be
defined for Soup Sieve to evaluate ns|tag type selectors. This is analogous to CSS's namespace at-rule:
A namespace dictionary should have keys (prefixes) and values (namespaces). An empty key string for a key would denote the default key. An empty value would essentially represent a null namespace. To represent the above CSS example for Soup Sieve, we would configure it like so:
namespace = {
"": "http://www.w3.org/1999/xhtml", # Default namespace is for XHTML
"svg": "http://www.w3.org/2000/svg", # The SVG namespace defined with prefix of "svg"
}
Prefixes used in the namespace dictionary do not have to match the prefixes in the document. The provided prefix is
never compared against the prefixes in the document, only the namespaces are compared. The prefixes in the document are
only there for the parser to know which tags get which namespace. And the prefixes in the namespace dictionary are only
defined in order to provide an alias for the namespaces when using the namespace selector syntax: ns|name.
Tags do not necessarily have to have a prefix for Soup Sieve to recognize them either. For instance, in HTML5, SVG should automatically get the SVG namespace. Depending how namespaces were defined in the document, tags may inherit namespaces in some conditions. Namespace assignment is mainly handled by the parser and exposed through the Beautiful Soup API. Soup Sieve uses the Beautiful Soup API to then compare namespaces for supported documents.
Ignore Pseudo-class
Soup Sieve implements a number of pseudo classes, but some (e.g. :has()
with the NOSTRICT flag) have potential performance concerns if exposed to untrusted user
inputs. While Beautiful Soup (along with Soup Sieve) are not necessarily recommended for time
critical, high performance systems, if you are in an environment where the risk of using a specific pseudo-class is not
tolerable, you can use the ignore option to specify and fail if they are used.
>>> import soupsieve as sv
>>> sv.compile('*:has(a)')
SoupSieve(pattern='*:has(a)', namespaces=None, custom=None, flags=0)
>>> try:
... sv.compile('*:has(a)', ignore=[':has'])
... except sv.SelectorSyntaxError:
... print('Not allowed')
...
Not allowed
A name is matched however it is written in the selector: in any case, and with CSS escapes, such as :HAS() or
:h\61 s(), and wherever it is used, such as inside :is() or :not().
Each name is ignored on its own, so a pseudo-class that has an alias must have every name it goes by listed. For
example, :-soup-contains() is also available under its deprecated name :contains(), and :is()
under :matches():
>>> import soupsieve as sv
>>> sv.compile('p:contains(text)', ignore=[':-soup-contains'])
SoupSieve(pattern='p:contains(text)', namespaces=None, custom=None, flags=0)
>>> try:
... sv.compile('p:contains(text)', ignore=[':-soup-contains', ':contains'])
... except sv.SelectorSyntaxError:
... print('Not allowed')
...
Not allowed
Selector Limits
New in 3.0
By default, selectors are limited 8192 selectors (or 0x2000). To give users more control, a new parameter called
max_selectors has been added and can be used to override the default selector limit with whatever is most appropriate
for the environment Soup Sieve is running in.
>>> import soupsieve as sv
>>> sv.compile('.a, .b, .c')
SoupSieve(pattern='.a, .b, .c', namespaces=None, custom=None, flags=0)
>>> try:
... sv.compile('.a, .b, .c', max_selectors=2)
... except ValueError:
... print('Too many selectors')
...
Too many selectors
Selector Length
max_selectors limits the number of selectors and the parts they are made of. It does not limit the length of the
selector string, or the number of values in the argument of a pseudo-class, such as the language ranges of
:lang(en, fr, de), which together count as one.
Soup Sieve does not limit the length of selector strings, and does not guarantee protection against long ones. The time to match grows with the length of the selector, whatever the selector is made of, and no option of Soup Sieve bounds it. If long selector strings are a concern, for example because selectors come from untrusted input, limiting their length is the responsibility of the application, before it passes them to Soup Sieve.
Thread Safety
New in 4.1
Soup Sieve is thread safe. Selectors can be compiled in any thread, and a compiled selector can be shared and used to match in any number of threads at the same time. This holds for free-threaded builds of Python as well.
Soup Sieve only reads a document, so matching in many threads at the same time is safe as long as no thread modifies the document while it is being matched. Beautiful Soup is not thread safe, so modifying a document must not happen at the same time as anything else that uses it.
An iterator from iselect() should only be used by one thread.