Utility functions for the engines

Back to top

Utility functions for the engines

searx.utils.XPathSpecType: TypeAlias = str | lxml.etree.XPath : Type alias used by searx.utils.get_xpath, searx.utils.eval_xpath and other XPath selectors.

searx.utils.searxng_useragent() → str[source] : Return the SearXNG User Agent

searx.utils.gen_useragent(os_string: str | None = None) → str[source] : Return a random browser User Agent

See searx/data/useragents.json

class searx.utils.HTMLTextExtractor[source] : Internal class to extract text from HTML

searx.utils.html_to_text(html_str: str) → str[source] : Extract text from a HTML string

Args:
:   - html\_str (str): string HTML

Returns:
:   - str: extracted text

Examples:
:   ```
    >>> html_to_text('Example <span id=“42”>#2</span>')
    'Example #2'
    ```

    ```
    >>> html_to_text('<style>.span { color: red; }</style><span>Example</span>')
    'Example'
    ```

    ```
    >>> html_to_text(r'regexp: (?&lt;![a-zA-Z]')
    'regexp: (?<![a-zA-Z]'
    ```

    ```
    >>> html_to_text(r'<p><b>Lorem ipsum </i>dolor sit amet</p>')
    'Lorem ipsum </i>dolor sit amet</p>'
    ```

    ```
    >>> html_to_text(r'&#x3e; &#x3c; &#97;')
    '> < a'
    ```

searx.utils.markdown_to_text(markdown_str: str) → str[source] : Extract text from a Markdown string

Args:
:   - markdown\_str (str): string Markdown

Returns:
:   - str: extracted text

Examples:
:   ```
    >>> markdown_to_text('[example](https://example.com)')
    'example'
    ```

    ```
    >>> markdown_to_text('## Headline')
    'Headline'
    ```

searx.utils.extract_text(xpath_results: list[ElementBase | _Element] | ElementBase | _Element | str | Number | bool | None, allow_none: bool = False) → str | None[source] : Extract text from a lxml result

- If `xpath_results` is a list of `ElementType` objects, extract
  the text from each result and concatenate the list in a string.
- If `xpath_results` is a `ElementType` object, extract all the
  text node from it ( [`lxml.html.tostring`](https://lxml.de/apidoc/lxml.html.html#lxml.html.tostring), `method=“text”` )
- If `xpath_results` is of type [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) or `Number`,
  [`bool`](https://docs.python.org/3/builtins/functions.html#bool) the string value is returned.
- If `xpath_results` is of type `None` a [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) is raised,
  except `allow_none` is `True` where `None` is returned.

searx.utils.normalize_url(url: str, base_url: str) → str[source] : Normalize URL: add protocol, join URL with base_url, add trailing slash if there is no path

Args:
:   - url (str): Relative URL
    - base\_url (str): Base URL, it must be an absolute URL.

Example:
:   ```
    >>> normalize_url('https://example.com', 'http://example.com/')
    'https://example.com/'
    >>> normalize_url('//example.com', 'http://example.com/')
    'http://example.com/'
    >>> normalize_url('//example.com', 'https://example.com/')
    'https://example.com/'
    >>> normalize_url('/path?a=1', 'https://example.com')
    'https://example.com/path?a=1'
    >>> normalize_url('', 'https://example.com')
    'https://example.com/'
    >>> normalize_url('/test', '/path')
    raise ValueError
    ```

Raises:
:   - lxml.etree.ParserError

Returns:
:   - str: normalized URL

searx.utils.extract_url(xpath_results: list[ElementBase | _Element] | ElementBase | _Element | str | Number | bool | None, base_url: str) → str[source] : Extract and normalize URL from lxml Element

Example:
:   ```
    >>> def f(s, search_url):
    >>>    return searx.utils.extract_url(html.fromstring(s), search_url)
    >>> f('<span id="42">https://example.com</span>', 'http://example.com/')
    'https://example.com/'
    >>> f('https://example.com', 'http://example.com/')
    'https://example.com/'
    >>> f('//example.com', 'http://example.com/')
    'http://example.com/'
    >>> f('//example.com', 'https://example.com/')
    'https://example.com/'
    >>> f('/path?a=1', 'https://example.com')
    'https://example.com/path?a=1'
    >>> f('', 'https://example.com')
    raise lxml.etree.ParserError
    >>> searx.utils.extract_url([], 'https://example.com')
    raise ValueError
    ```

Raises:
:   - ValueError
    - lxml.etree.ParserError

Returns:
:   - str: normalized URL

searx.utils.dict_subset(dictionary: MutableMapping[Any, Any], properties: set[str]) → MutableMapping[str, Any][source] : Extract a subset of a dict

Examples:
:   ```
    >>> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'C'])
    {'A': 'a', 'C': 'c'}
    >>> >> dict_subset({'A': 'a', 'B': 'b', 'C': 'c'}, ['A', 'D'])
    {'A': 'a'}
    ```

searx.utils.humanize_bytes(size: int | float, precision: int = 2)[source] : Determine the human readable value of bytes on 1024 base (1KB=1024B).

searx.utils.humanize_number(size: int | float, precision: int = 0)[source] : Determine the human readable value of a decimal number.

searx.utils.convert_str_to_int(number_str: str) → int[source] : Convert number_str to int or 0 if number_str is not a number.

searx.utils.extr(txt: str, begin: str, end: str, default: str = '') → str[source] : Extract the string between begin and end from txt

Parameters:
:   - **txt** – String to search in
    - **begin** – First string to be searched for
    - **end** – Second string to be searched for after `begin`
    - **default** – Default value if one of `begin` or `end` is not
      found. Defaults to an empty string.

Returns:
:   The string between the two search-strings `begin` and `end`.
    If at least one of `begin` or `end` is not found, the value of
    `default` is returned.

Examples:
:   ```
    >>> extr("abcde", "a", "e")
    "bcd"
    >>> extr("abcde", "a", "z", deafult="nothing")
    "nothing"
    ```

searx.utils.int_or_zero(num: list[str] | str) → int[source] : Convert num to int or 0. num can be either a str or a list. If num is a list, the first element is converted to int (or return 0 if the list is empty). If num is a str, see convert_str_to_int

searx.utils.to_string(obj: Any) → str[source] : Convert obj to its string representation.

searx.utils.ecma_unescape(text: str) → str[source] : Python implementation of the unescape javascript function

&lt;https://www.ecma-international.org/ecma-262/6.0/#sec-unescape-string>
&lt;https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/unescape>

Examples:
:   ```
    >>> ecma_unescape('%u5409')
    '吉'
    >>> ecma_unescape('%20')
    ' '
    >>> ecma_unescape('%F3')
    'ó'
    ```

searx.utils.remove_pua_from_str(text: str)[source] : Removes unicode’s “PRIVATE USE CHARACTER”s (PUA) from a string.

searx.utils.get_engine_from_settings(name: str) → dict[str, dict[str, str]][source] : Return engine configuration from settings.yml of a given engine name

searx.utils.get_xpath(xpath_spec: str | XPath) → XPath[source] : Return cached compiled lxml.etree.XPath object.

`TypeError`:
:   Raised when `xpath_spec` is neither a [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) nor a
    [`lxml.etree.XPath`](https://lxml.de/apidoc/lxml.etree.html#lxml.etree.XPath).

`SearxXPathSyntaxException`:
:   Raised when there is a syntax error in the *XPath* selector (`str`).

searx.utils.eval_xpath(element: ElementBase | _Element, xpath_spec: str | XPath) → Any[source] : Equivalent of element.xpath(xpath_str) but compile xpath_str into a lxml.etree.XPath object once for all. The return value of xpath(..) is complex, read XPath return values for more details.

`TypeError`:
:   Raised when `xpath_spec` is neither a [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) nor a
    [`lxml.etree.XPath`](https://lxml.de/apidoc/lxml.etree.html#lxml.etree.XPath).

`SearxXPathSyntaxException`:
:   Raised when there is a syntax error in the *XPath* selector (`str`).

`SearxEngineXPathException:`
:   Raised when the XPath can’t be evaluated (masked
    `lxml.etree..XPathError`).

searx.utils.eval_xpath_list(element: ElementBase | _Element, xpath_spec: str | XPath, min_len: int | None = None) → list[Any][source] : Same as searx.utils.eval_xpath, but additionally ensures the return value is a list. The minimum length of the list is also checked (if min_len is set).

searx.utils.eval_xpath_getindex(element: ElementBase | _Element, xpath_spec: str | XPath, index: int, default: Any = <searx.utils._NotSetClass object>) → Any[source] : Same as searx.utils.eval_xpath_list, but returns item on position index from the list (index starts with 0).

The exceptions known from [`searx.utils.eval_xpath`](#searx.utils.eval_xpath) are thrown. If a
default is specified, this is returned if an element at position `index`
could not be determined.

searx.utils.get_embedded_stream_url(url: str) → str[source] : Converts a standard video URL into its embed format.

Supported services include:

- Youtube
- Facebook
- Instagram
- TikTok
- Dailymotion
- Bilibili

The function is suitable for the field `MainResult.iframe_src`. If `url` is not from one of
the supported services, an empty string is returned.

searx.utils.js_obj_str_to_python(js_obj_str: str) → Any[source] : Convert a javascript variable into JSON and then load the value

It does not deal with all cases, but it is good enough for now.
chompjs has a better implementation.

searx.utils.js_find_matching_bracket(text: str, start: int, open_char: str = '{', close_char: str = '}') → int[source] : Return index of the matching closing bracket, safely ignoring JS string literals.

searx.utils.js_split_args(args_str: str) → list[str][source] : Split a JS argument list, respecting strings and nested brackets.

searx.utils.js_find_variable_positions(js_obj: str) → list[tuple[str, int]][source] : Find all variables referenced in the JavaScript object.

searx.utils.parse_duration_string(duration_str: str) → timedelta | None[source] : Parse a time string in format MM:SS or HH:MM:SS and convert it to a datetime.timedelta object.

Returns `None` if the provided string doesn’t match any of the formats.

searx.utils.solve_altcha(parameters: dict[str, Any], maxCounter: int = 1000) → tuple[str, int] | None[source] : Solves Altcha CAPTCHAs. It derives keys using PBKDF2 until the derived key starts with the keyPrefix` from the challenge. If the solver does not finish after maxCounter iterations, returns None.

Copyright © SearXNG team

Made with Furo

💾 🌐