Url

Normalize

  • Url::Normalize(String) -> String?

Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, and so on.
The normalization result only depends on the URL itself. The normalization DOES NOT include operations depending on the external data: transformation based on duplicates, mirrors, etc.

Returned value:

  • Normalized URL.
  • NULL, if the passed string argument can’t be parsed as a URL.

Examples

  1. SELECT Url::Normalize("hTTp://wWw.yDb.TECH/"); -- "http://www.ydb.tech/"
  2. SELECT Url::Normalize("http://ydb.tech#foo");      -- "http://ydb.tech/"

Url - 图1

NormalizeWithDefaultHttpScheme

  • Url::NormalizeWithDefaultHttpScheme(String?) -> String?

Normalizes similarly to Url::Normalize, but inserts the http:// schema in case there is no schema.

Returned value:

  • Normalized URL.
  • Source URL, if the normalization has failed.

Examples

  1. SELECT Url::NormalizeWithDefaultHttpScheme("wWw.yDb.TECH");    -- "http://www.ydb.tech/"
  2. SELECT Url::NormalizeWithDefaultHttpScheme("http://ydb.tech#foo"); -- "http://ydb.tech/"

Url - 图2

Encode / Decode

Encode a UTF-8 string to the urlencoded format (Url::Encode) and back (Url::Decode).

List of functions

  • Url::Encode(String?) -> String?
  • Url::Decode(String?) -> String?

Examples

  1. SELECT Url::Decode("http://ydb.tech/%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0"); 
  2.   -- "http://ydb.tech/page"
  3. SELECT Url::Encode("http://ydb.tech/page");                                         
  4.   -- "http://ydb.tech/%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0"

Url - 图3

Parse

Parses the URL into parts.

  • Url::Parse(Parse{Flags:AutoMap}) -> Struct< Frag: String?, Host: String?, ParseError: String?, Pass: String?, Path: String?, Port: String?, Query: String?, Scheme: String?, User: String? >

Examples

  1. SELECT Url::Parse(
  2.   "https://en.wikipedia.org/wiki/Isambard_Kingdom_Brunel?s=24&g=h-24#Great_Western_Railway");
  3. /*
  4. (
  5.   "Frag": "Great_Western_Railway",
  6.   "Host": "en.wikipedia.org",
  7.   "ParseError": null,
  8.   "Pass": null,
  9.   "Path": "/wiki/Isambard_Kingdom_Brunel",
  10.   "Port": null,
  11.   "Query": "s=24&g=h-24",
  12.   "Scheme": "https",
  13.   "User": null
  14. )
  15. */

Url - 图4

Get…

Get a component of the URL.

List of functions

  • Url::GetScheme(String{Flags:AutoMap}) -> String

  • Url::GetHost(String?) -> String?

  • Url::GetHostPort(String?) -> String?

  • Url::GetSchemeHost(String?) -> String?

  • Url::GetSchemeHostPort(String?) -> String?

  • Url::GetPort(String?) -> String?

  • Url::GetTail(String?) -> String? — everything following the host: path + query + fragment

  • Url::GetPath(String?) -> String?

  • Url::GetFragment(String?) -> String?

  • Url::GetCGIParam(String?, String) -> String? — The second parameter is the name of the intended CGI parameter.

  • Url::GetDomain(String?, Uint8) -> String? — The second parameter is the required domain level.

  • Url::GetTLD(String{Flags:AutoMap}) -> String

  • Url::IsKnownTLD(String{Flags:AutoMap}) -> Bool — Registered on http://www.iana.org/

  • Url::IsWellKnownTLD(String{Flags:AutoMap}) -> Bool — Belongs to a small whitelist of com, net, org, ru, and so on.

  • Url::GetDomainLevel(String{Flags:AutoMap}) -> Uint64

  • Url::GetSignificantDomain(String{Flags:AutoMap}, [List<String>?]) -> String
    Returns a second-level domain in most cases and a third-level domain for the hostnames like: ***.XXX.YY, where XXX is com, net, org, co, gov, or edu. You can redefine this list using an optional second argument

  • Url::GetOwner(String{Flags:AutoMap}) -> String
    Returns the domain that’s most likely owned by an individual or organization. Unlike Url::GetSignificantDomain, it uses a special whitelist. Besides the ***.co.uk domains, it can return a third-level domain used by free hosting sites and blogs (for example: something.livejournal.com)s

Examples

  1. SELECT Url::GetScheme("https://ydb.tech");           -- "https://"
  2. SELECT Url::GetDomain("http://www.ydb.tech", 2); -- "ydb.tech"

Url - 图5

Cut…

  • Url::CutScheme(String?) -> String?
    Returns the passed URL without the schema (http://, https://, etc.).

  • Url::CutWWW(String?) -> String?
    Returns the passed domain without the “www.” prefix (if any).

  • Url::CutWWW2(String?) -> String?
    Returns the passed domain without the prefixes like “ www.”, “ www2.”, “ wwww777.” (if any).

  • Url::CutQueryStringA­ndFragment(String{Flags:AutoMap}) -> String
    Returns a copy of the passed URL, stripping out all the CGI parameters and fragments (“?foo=bar” and/or “#baz”).

Examples

  1. SELECT Url::CutScheme("http://www.ydb.tech"); -- "www.ydb.tech"
  2. SELECT Url::CutWWW("www.ydb.tech");           -- "yydb.tech"

Url - 图6

…Punycode…

Punycode transformations.

List of functions

  • Url::HostNameToPunycode(String{Flag:AutoMap}) -> String?
  • Url::ForceHostNameToPunycode(String{Flag:AutoMap}) -> String
  • Url::PunycodeToHostName(String{Flag:AutoMap}) -> String?
  • Url::ForcePunycodeToHostName(String{Flag:AutoMap}) -> String
  • Url::CanBePunycodeHostName(String{Flag:AutoMap}) -> Bool

Examples

  1. SELECT Url::PunycodeToHostName("xn--80aniges7g.xn--j1aef"); -- "экзампл.ком"

Url - 图7

…Query…

Query transformations.

List of functions

  1. Url::QueryStringToList(String{Flag:AutoMap}, [
  2.   KeepBlankValues:Bool?,  -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false.
  3.   Strict:Bool?,           -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true.
  4.   MaxFields:Uint32?,      -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>.
  5.   Separator:String?       -- A key-value pair separator, defaults to '&'.
  6. ]) -> List<Tuple<String, String>>
  7. Url::QueryStringToDict(String{Flag:AutoMap}, [
  8.   KeepBlankValues:Bool?,  -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false.
  9.   Strict:Bool?,           -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true.
  10.   MaxFields:Uint32?,      -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>.
  11.   Separator:String?       -- A key-value pair separator, defaults to '&'.
  12. ]) -> Dict<String, List<String>>
  13. Url::BuildQueryString(Dict<String, List<String?>>{Flag:AutoMap}, [
  14.   Separator:String?       -- A key-value pair separator, defaults to '&'.
  15. ]) -> String
  16. Url::BuildQueryString(Dict<String, String?>{Flag:AutoMap}, [
  17.   Separator:String?       -- A key-value pair separator, defaults to '&'.
  18. ]) -> String
  19. Url::BuildQueryString(List<Tuple<String, String?>>{Flag:AutoMap}, [
  20.   Separator:String?       -- A key-value pair separator, defaults to '&'.
  21. ]) -> String

Url - 图8

Examples

  1. SELECT Url::QueryStringToList("a=1&b=2&a=3");                       -- [("a", "1"), ("b", "2"), ("a", "3")]
  2. SELECT Url::QueryStringToDict("a=1&b=2&a=3");                       -- {"b" : ["2"], "a" : ["1", "3"]}
  3. SELECT Url::BuildQueryString([("a", "1"), ("a", "3"), ("b", "2")]); -- "a=1&a=3&b=2"
  4. SELECT Url::BuildQueryString({"a" : "1", "b" : "2"});               -- "b=2&a=1"
  5. SELECT Url::BuildQueryString({"a" : ["1", "3"], "b" : ["2", "4"]}); -- "b=2&b=4&a=1&a=3"

Url - 图9