Unicode

Functions for Unicode strings.

List of functions

  • Unicode::IsUtf(String) -> Bool Checks whether a string is a valid UTF-8 sequence. For example, the string "\xF0" isn’t a valid UTF-8 sequence, but the string "\xF0\x9F\x90\xB1" correctly describes a UTF-8 cat emoji.
  • Unicode::GetLength(Utf8{Flags:AutoMap}) -> Uint64 Returns the length of a utf-8 string in unicode code points. Surrogate pairs are counted as one character.
  • Unicode::Find(Utf8{Flags:AutoMap}, Utf8, [Uint64?]) -> Uint64?
  • Unicode::RFind(Utf8{Flags:AutoMap}, Utf8, [Uint64?]) -> Uint64?
  • Unicode::Substring(Utf8{Flags:AutoMap}, from:Uint64?, len:Uint64?) -> Utf8 Returns a substring starting with from with the length of len characters. If the len argument is omitted, the substring is moved to the end of the source string.

  • The Unicode::Normalize... functions convert the passed UTF-8 string to a normalization form:

  • Unicode::Normalize(Utf8{Flags:AutoMap}) -> Utf8 — NFC

  • Unicode::NormalizeNFD(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::NormalizeNFC(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::NormalizeNFKD(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::NormalizeNFKC(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::Translit(Utf8{Flags:AutoMap}, [String?]) -> Utf8 Transliterates with Latin letters the words from the passed string, consisting entirely of characters of the alphabet of the language passed by the second argument. If no language is specified, the words are transliterated from Russian. Available languages: “kaz”, “rus”, “tur”, and “ukr”.
  • Unicode::LevensteinDistance(Utf8{Flags:AutoMap}, Utf8{Flags:AutoMap}) -> Uint64 Calculates the Levenshtein distance for the passed strings.
  • Unicode::Fold(Utf8{Flags:AutoMap}, [ Language:String?, DoLowerCase:Bool?, DoRenyxa:Bool?, DoSimpleCyr:Bool?, FillOffset:Bool? ]) -> Utf8 A case folding is performed on the passed string. Language is set according to the same rules as in Unicode::Translit(). DoLowerCase converts a string to lowercase letters, defaults to true.
  • Unicode::ReplaceAll(Utf8{Flags:AutoMap}, Utf8, Utf8) -> Utf8 Arguments: input, find, replacement. Replaces all occurrences of the find string in the input with replacement.
  • Unicode::ReplaceFirst(Utf8{Flags:AutoMap}, Utf8, Utf8) -> Utf8 Arguments: input, findSymbol, replacementSymbol. Replaces the first occurrence of the findSymbol character in the input with replacementSymbol. The character can’t be a surrogate pair.
  • Unicode::ReplaceLast(Utf8{Flags:AutoMap}, Utf8, Utf8) -> Utf8 Arguments: input, findSymbol, replacementSymbol. Replaces the last occurrence of the findSymbol character in the input with replacementSymbol. The character can’t be a surrogate pair.
  • Unicode::RemoveAll(Utf8{Flags:AutoMap}, Utf8) -> Utf8 The second argument is interpreted as an unordered set of characters to be removed. Removes all occurrences.
  • Unicode::RemoveFirst(Utf8{Flags:AutoMap}, Utf8) -> Utf8 The second argument is interpreted as an unordered set of characters to be removed. Removes the first occurrence.
  • Unicode::RemoveLast(Utf8{Flags:AutoMap}, Utf8) -> Utf8 The second argument is interpreted as an unordered set of characters to be removed. Removes the last occurrence.
  • Unicode::ToCodePointList(Utf8{Flags:AutoMap}) -> List<Uint32>
  • Unicode::FromCodePointList(List<Uint32>{Flags:AutoMap}) -> Utf8
  • Unicode::Reverse(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::ToLower(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::ToUpper(Utf8{Flags:AutoMap}) -> Utf8
  • Unicode::ToTitle(Utf8{Flags:AutoMap}) -> Utf8

  • Unicode::SplitToList( Utf8?, Utf8, [ DelimeterString:Bool?, SkipEmpty:Bool?, Limit:Uint64? ]) -> List<Utf8> The first argument is the source string The second argument is a delimiter The third argument includes the following parameters:

  • DelimeterString:Bool? — treating a delimiter as a string (true, by default) or a set of characters “any of” (false)

  • SkipEmpty:Bool? - whether to skip empty strings in the result, is false by default
  • Limit:Uint64? - Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last item
  • Unicode::JoinFromList(List<Utf8>{Flags:AutoMap}, Utf8) -> Utf8
  • Unicode::ToUint64(Utf8{Flags:AutoMap}, [Uint16?]) -> Uint64 The second optional argument specifies the number system, by default 0 — automatic detection by prefix. Supported prefixes : 0x(0X) - base-16, 0 - base-8. The default system is base-10. The ‘-‘ sign before the number is interpreted as in unsigned C arithmetic, for example -0x1 -> UI64_MAX. If there are incorrect characters in the string or the number goes beyond the boundaries of ui64, the function terminates with an error.
  • Unicode::TryToUint64(Utf8{Flags:AutoMap}, [Uint16?]) -> Uint64? Similar to the Unicode::ToUint64() function, but returns Nothing instead of an error.

Examples

1. SELECT Unicode::Fold("Eylül", "Turkish" AS Language); -- "eylul"
2. SELECT Unicode::GetLength("жніўня");                  -- 6

Unicode - 图1