Technical Documentation: sparklemotion/nokogiri
ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference
Overview (`README.md`)
Nokogiri
Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby. It provides a sensible, easy-to-understand API for reading, writing, modifying, and querying documents. It is fast and standards-compliant by relying on native parsers like libxml2, libgumbo, and xerces.
Guiding Principles
Some guiding principles Nokogiri tries to follow:
- be secure-by-default by treating all documents as untrusted by default
- be a thin-as-reasonable layer on top of the underlying parsers, and don't attempt to fix behavioral differences between the parsers
Features Overview
- DOM Parser for XML, HTML4, and HTML5
- SAX Parser for XML and HTML4
- Push Parser for XML and HTML4
- Document search via XPath 1.0
- Document search via CSS3 selectors, with some jquery-like extensions
- XSD Schema validation
- XSLT transformation
- "Builder" DSL for XML and HTML documents
Status
Support, Getting Help, and Reporting Issues
All official documentation is posted at https://nokogiri.org (the source for which is at https://github.com/sparklemotion/nokogiri.org/, and we welcome contributions).
Reading
Your first stops for learning more about Nokogiri should be:
- API Documentation
- Tutorials
- An excellent community-maintained Cheat Sheet
Ask For Help
There are a few ways to ask exploratory questions:
- The Nokogiri mailing list is active at https://groups.google.com/group/nokogiri-talk
- Open an issue using the "Help Request" template at https://github.com/sparklemotion/nokogiri/issues
- Open a discussion at https://github.com/sparklemotion/nokogiri/discussions
Please do not mail the maintainers at their personal addresses.
Report A Bug
The Nokogiri bug tracker is at https://github.com/sparklemotion/nokogiri/issues
Please use the "Bug Report" or "Installation Difficulties" templates.
Security and Vulnerability Reporting
Please report vulnerabilities at https://github.com/sparklemotion/nokogiri/security
See SECURITY.md for full information and description of our security policy.
Semantic Versioning Policy
Nokogiri follows Semantic Versioning (since 2017 or so).
We bump Major.Minor.Patch versions following this guidance:
Major: (we've never done this)
- Significant backwards-incompatible changes to the public API that would require rewriting existing application code.
Minor:
- Features and bugfixes.
- Updating packaged libraries for non-security-related reasons.
- Dropping support for EOLed Ruby versions. Some folks find this objectionable, but SemVer says this is OK if the public API hasn't changed.
- Backwards-incompatible changes to internal or private methods and constants. These are detailed in the "Changes" section of each changelog entry.
- Removal of deprecated methods or parameters, after a generous transition period; usually when those methods or parameters are rarely-used or dangerous to the user. Essentially, removals that do not justify a major version bump.
Patch:
- Bugfixes.
- Security updates.
- Updating packaged libraries for security-related reasons.
Sponsorship
You can help sponsor the maintainers of this software through one of these organizations:
- github.com/sponsors/flavorjones
- opencollective.com/nokogiri
- tidelift.com/subscription/pkg/rubygems-nokogiri
Installation
Requirements:
- Ruby >= 3.2
- JRuby >= 10.0
If you are compiling the native extension against a system version of libxml2:
- libxml2 >= 2.9.2 (recommended >= 2.12.0)
Native Gems: Faster, more reliable installation
"Native gems" contain pre-compiled libraries for a specific machine architecture. On supported platforms, this removes the need for compiling the C extension and the packaged libraries, or for system dependencies to exist. This results in much faster installation and more reliable installation, which as you probably know are the biggest headaches for Nokogiri users.
Supported Platforms
Nokogiri ships pre-compiled, "native" gems for the following platforms:
- Linux:
x86_64-linux-gnu,aarch64-linux-gnu, andarm-linux-gnu(req:glibc >= 2.29)x86_64-linux-musl,aarch64-linux-musl, andarm-linux-musl
- Darwin/MacOS:
x86_64-darwinandarm64-darwin - Windows:
x64-mingw-ucrt - Java: any platform running JRuby 10.0 or higher
To determine whether your system supports one of these gems, look at the output of bundle platform or ruby -e 'puts Gem::Platform.local.to_s'.
If you're on a supported platform, either gem install or bundle install should install a native gem without any additional action on your part. This installation should only take a few seconds, and your output should look something like:
$ gem install nokogiri
Fetching nokogiri-1.11.0-x86_64-linux.gem
Successfully installed nokogiri-1.11.0-x86_64-linux
1 gem installedOther Installation Options
Because Nokogiri is a C extension, it requires that you have a C compiler toolchain, Ruby development header files, and some system dependencies installed.
The following may work for you if you have an appropriately-configured system:
gem install nokogiriIf you have any issues, please visit Installing Nokogiri for more complete instructions and troubleshooting.
How To Use Nokogiri
Nokogiri is a large library, and so it's challenging to briefly summarize it. We've tried to provide long, real-world examples at Tutorials.
Parsing and Querying
Here is example usage for parsing and querying a document:
#! /usr/bin/env ruby
require 'nokogiri'
require 'open-uri'
# Fetch and parse HTML document
doc = Nokogiri::HTML(URI.open('https://nokogiri.org/tutorials/installing_nokogiri.html'))
# Search for nodes by css
doc.css('nav ul.menu li a', 'article h2').each do |link|
puts link.content
end
# Search for nodes by xpath
doc.xpath('//nav//ul//li/a', '//article//h2').each do |link|
puts link.content
end
# Or mix and match
doc.search('nav ul.menu li a', '//article//h2').each do |link|
puts link.content
endEncoding
Strings are always stored as UTF-8 internally. Methods that return
text values will always return UTF-8 encoded strings. Methods that
return a string containing markup (like to_xml, to_html andinner_html) will return a string encoded like the source document.
WARNING
Some documents declare one encoding, but actually use a different
one. In these cases, which encoding should the parser choose?
Data is just a stream of bytes. Humans add meaning to that stream. Any
particular set of bytes could be valid characters in multiple
encodings, so detecting encoding with 100% accuracy is not
possible. libxml2 does its best, but it can't be right all the time.
If you want Nokogiri to handle the document encoding properly, your
best bet is to explicitly set the encoding. Here is an example of
explicitly setting the encoding to EUC-JP on the parser:
doc = Nokogiri.XML('<foo><bar /></foo>', nil, 'EUC-JP')Technical Overview
Guiding Principles
As noted above, two guiding principles of the software are:
- be secure-by-default by treating all documents as untrusted by default
- be a thin-as-reasonable layer on top of the underlying parsers, and don't attempt to fix behavioral differences between the parsers
Notably, despite all parsers being standards-compliant, there are behavioral inconsistencies between the parsers used in the CRuby and JRuby implementations, and Nokogiri does not and should not attempt to remove these inconsistencies. Instead, we surface these differences in the test suite when they are important/semantic; or we intentionally write tests to depend only on the important/semantic bits (omitting whitespace from regex matchers on results, for example).
CRuby
The Ruby (a.k.a., CRuby, MRI, YARV) implementation is a C extension that depends on libxml2 and libxslt (which in turn depend on zlib and possibly libiconv).
These dependencies are met by default by Nokogiri's packaged versions of the libxml2 and libxslt source code, but a configuration option --use-system-libraries is provided to allow specification of alternative library locations. See Installing Nokogiri for full documentation.
We provide native gems by pre-compiling libxml2 and libxslt (and potentially zlib and libiconv) and packaging them into the gem file. In this case, no compilation is necessary at installation time, which leads to faster and more reliable installation.
See LICENSE-DEPENDENCIES.md for more information on which dependencies are provided in which native and source gems.
JRuby
The Java (a.k.a. JRuby) implementation is a Java extension that depends primarily on Xerces and NekoHTML for parsing, though additional dependencies are on isorelax, nekodtd, jing, serializer, xalan-j, and xml-apis.
These dependencies are provided by pre-compiled jar files packaged in the java platform gem.
See LICENSE-DEPENDENCIES.md
for more information on which dependencies are provided in which native and source gems.
Contributing
See CONTRIBUTING.md for an intro guide to developing Nokogiri.
Code of Conduct
See the CODE_OF_CONDUCT.md.
License
This project is licensed under the terms of the MIT license.
See LICENSE.md.
Dependencies
Some additional libraries may be distributed with your version of Nokogiri.
See LICENSE-DEPENDENCIES.md for a discussion of the variations as well as the licenses thereof.
Authors
- Mike Dalessio
- Aaron Patterson
- Yoko Harada
- Akinori MUSHA
- John Shahid
- Karol Bucek
- Sam Ruby
- Craig Barnes
- Stephen Checkoway
- Lars Kanis
- Sergio Arbeo
- Timothy Elliott
- Nobuyoshi Nakada
Sponsors
These folks have generously sponsored Nokogiri's development. Thank you for your support!
Overview (`gumbo-parser/src/README.md`)
libgumbo
This is an internal fork of the libgumbo library, which was copied and
later modified under the terms of the Apache 2.0 license. See lua-gumbo
commit 0a04728 for details of the original import.
Since importing the code, the following notable fixes and improvements
have been made:
91cef89: Re-implementadjust_foreign_attributes()with a gperf hashb11abe7: PassTagSetarrays into functions by reference instead of valueb73dc03: Simplifymaybe_replace_codepoint()functiond5d0bb3: Remove special handling of<menuitem>tag7bd5162: Remove special handling of `` taga5c1b0e: Userealloc(3)instead ofmalloc(3)inenlarge_vector_if_full()dcbebd7: Userealloc(3)instead ofmalloc(3)inmaybe_resize_string_buffer()df15262: Makedestroy_node()function non-recursive2df37f5: Fix signedness of some format specifiers176553e: Add maximum element nesting limitbed0f4a: Annotategumbo_debug()withPRINTFmacro and fix warnings7ffc218: Annotateprint_message()withPRINTFmacro and fix warnings1bd8ab5,9136507,53a1f9a: Deduplicate some identicalTagSetarraysa7a9065: Add some GCC/Clang function attributes8d3d4e4: Remove custom allocator support8d3b006: Fix recording of source positions for</form>end tags1a8d763: Replace linear search inmaybe_replace_codepoint()with a lookup table6dca79e: Replacestrcasecmp()andstrncasecmp()with ascii-only equivalents17ab1d2: FixTAGSET_INCLUDESmacro to work properly with multiple bit flags7e56d45: Re-implementgumbo_normalize_svg_tagname()with a gperf hasha518d35: Replace linear array search inadjust_svg_attributes()with a gperf hasha4a7433: Fix duplicateTagSetinitializer being ignored inis_special_node()8137fcd: Add support for<dialog>tag4b35471: Add missingstaticqualifiers to hide symbols that shouldn't be externdf57c59,03101f3,ea62330: Replace use of locale-dependantctype.hfunctions
with custom, ASCII-only equivalents
Doc Keyword Arguments (`doc/keyword_arguments.md`)
Keyword Arguments
Many \Nokogiri methods take optional positional arguments.
Beginning in version 1.17.0 (December 2024),
some methods are being "modernized" to take optional keyword arguments,
which are far more versatile.
Example:
# Before.
XML::Document.parse(xml_s, nil, nil, options)
# After.
XML::Document.parse(xml_s, options: options)About the Examples
Examples on this page assume that the following code has been executed:
require 'nokogiri'
include Nokogiri
xml_s = '<root />'
url = 'www.site.com'
encoding = 'UTF-16'
options = XML::ParseOptions::STRICTBefore
Before the changes, the calling sequence for a method might have
trailing optional positional arguments:
For example, the calling sequence XML::Document.parse was:
XML::Document.parse(
# Required leading argument.
input,
# Optional positional arguments.
url = nil,
encoding = nil,
options = XML::ParseOptions::DEFAULT_XML
)That calling sequence requires leading argument input,
and allows any of these:
- No optional arguments.
- Optional argument
urlonly. - Optional arguments
urlandencodingonly. - Optional arguments
url,encoding, andoptions.
To pass arguments input and options,
a method call would also have to pass arguments url and encoding:
XML::Document.parse(xml_s, nil, nil, options)After
The updated calling sequence allows trailing keyword arguments.
The updated calling sequence for XML::Document.parse, for example,
allows optional keyword arguments url, encoding, and options.
The updated calling sequence may be thought of as:
XML::Document.parse(
# Required leading argument.
input,
# Optional keyword arguments.
url:,
encoding:,
options:
)where url, encoding, and options are optional keyword arguments.
Thus, to pass arguments input and options,
a method call need only pass those two arguments (and not arguments url and encoding):
XML::Document.parse(xml_s, options: options)Each of the optional keyword arguments may be given or omitted;
they may be given in any combination and in any order:
XML::Document.parse(xml_s, options: options, encoding: encoding, url: url)The new calling sequence is fully compatible with the old,
so that this is still a valid call:
XML::Document.parse(xml_s, url, encoding, options)Details
The updated calling sequence retains the optional positional arguments,
but adds trailing keyword arguments;
the default value for each keyword argument comes from the given (or default)
value of a positional argument.
The actual updated calling sequence for XML::Document.parse, for example, is:
XML::Documnent.parse(
# Required leading argument.
input,
# Optional positional arguments.
url_ = nil,
encoding_ = nil,
options_ = XML::ParseOptions::DEFAULT_XML,
# Optional keyword arguments; each defaults to a positional argument value.
url: url_,
encoding: encoding_,
options: options_
)Valid calls to the method include:
# Positional arguments only.
XML::Document.parse(xml_s, url, encoding, options)
# Keyword arguments only, any order.
XML::Document.parse(xml_s, url: url, encoding: encoding, options: options)
XML::Document.parse(xml_s, encoding: encoding, options: options, url: url)
# Mixture of leading positional arguments and trailing keyword arguments.
XML::Document.parse(xml_s, url, options: options, encoding: encoding)Doc Xml Parsing (`doc/xml/parsing.md`)
Parsing
This page shows how \Nokogiri parses an XML string into \Nokogiri objects.
The string has text consisting of character data and markup.
For a \Nokogiri parsing method, the string is passed
either as a String object
or as an IO object
from which the string is to be read.
Most of the sections below link to a relevant section in the W3C document
Extensible Markup Language (XML) 1.0 (Fifth Edition).
On this page, each example uses either:
Method Nokogiri::XML::parse (shorthand for Nokogiri::XML::Document.parse)
to parse a string into a tree of \Nokogiri objects.
The topmost object is a Nokogiri::XML::Document object,
which we will usually refer to as a document.A document may be childless (i.e., it may have no immediate child object),
or it may have a single immediate child Nokogiri::XML::Element object -- its root.Method Nokogiri::XML::DocumentFragment.parse
to parse a string into a tree of \Nokogiri objects.
The topmost object is a Nokogiri::XML::DocumentFragment object,
which we will usually refer to as a fragment;
the fragment may have other objects as children.A document fragment may have multiple immediate child objects of various types.
Text
The string to be parsed is text, consisting of
character data and markup.
Character Data
All text that is not markup it character data.
Markup
Comments
\Nokogiri parses an XML comment
into a Nokogiri::XML::Comment object.
A comment may be in the document itself or in a tag:
xml = '<root></root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Comment " Comment. "),
#(Element: {
name = "root",
children = [ #(Comment " Another comment. ")]
})]
})Processing Instructions
\Nokogiri parses an XML processing instruction
into a Nokogiri::XML::ProcessingInstruction object:
xml = '<?xml-stylesheet type="text/xsl" href="style.xsl"?>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(ProcessingInstruction: {
name = "xml-stylesheet"
})]
})CDATA Sections
\Nokogiri parses an XML CDATA section
into a Nokogiri::XML::CDATA object:
xml = '<root><![CDATA[<greeting>Hello, world!</greeting>]]></root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [
#(CDATA "<greeting>Hello, world!</greeting>")]
})]
})Prolog (XML Declaration)
\Nokogiri parses an XML declaration
into values put onto the parsed document:
xml = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
doc = Nokogiri::XML.parse(xml)
doc.version # => "1.0"
doc.encoding # => "UTF-8"Document Type Declaration
\Nokogiri parses an XML document type declaration
into a Nokogiri::XML::DTD object:
xml = '<!DOCTYPE greeting SYSTEM "hello.dtd">'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [ #(DTD: { name = "greeting" })]
})Tags
\Nokogiri parses an XML tag
into a Nokogiri::XML::Element object.
In this example, a single tag is parsed into a document whose only child
is the root element parsed from the tag:
xml = '<root/>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [ #(Element: { name = "root" })]
})A tag may have nested tags:
xml = '<root><foo><goo/><moo/></foo><bar><car/><far/></bar></root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [
#(Element: {
name = "foo",
children = [
#(Element: { name = "goo" }),
#(Element: { name = "moo" })]
}),
#(Element: {
name = "bar",
children = [
#(Element: { name = "car" }),
#(Element: { name = "far" })]
})]
})]
})A tag may have nested text:
xml = '<root>One<foo>Two</foo>Three<bar>Four</bar>Five</root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [
#(Text "One"),
#(Element: {
name = "foo",
children = [ #(Text "Two")]
}),
#(Text "Three"),
#(Element: {
name = "bar",
children = [ #(Text "Four")]
}),
#(Text "Five")]
})]
})A tag may have nested markup of other types (such as comments):
xml = '<root><foo/><bar/></root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [
#(Element: { name = "foo" }),
#(Comment " Comment text. "),
#(Element: { name = "bar" })]
})]
})Tag Attributes
\Nokogiri parses an XML tag attribute
into a Nokogiri::XML::Attr object:
xml = '<root foo="0" bar="1"/>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
attribute_nodes = [
#(Attr: { name = "foo", value = "0" }),
#(Attr: { name = "bar", value = "1" })]
})]
})Element Type Declarations
\Nokogiri parses an XML element type declaration
into a Nokogiri::XML::ElementDecl object:
xml = <<DOCTYPE
<!DOCTYPE note [
<!ELEMENT note (to,from,heading,body)>
<!ELEMENT to (#PCDATA)>
<!ELEMENT from (#PCDATA)>
<!ELEMENT heading (#PCDATA)>
<!ELEMENT body (#PCDATA)>
]>
DOCTYPE
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(DTD: {
name = "note",
children = [
#(ElementDecl: { "<!ELEMENT note (to , from , heading , body)>\n" }),
#(ElementDecl: { "<!ELEMENT to (#PCDATA)>\n" }),
#(ElementDecl: { "<!ELEMENT from (#PCDATA)>\n" }),
#(ElementDecl: { "<!ELEMENT heading (#PCDATA)>\n" }),
#(ElementDecl: { "<!ELEMENT body (#PCDATA)>\n" })]
})]
})Attribute-List Declarations
\Nokogiri parses an XML attribute-list declaration
into a Nokogiri::XML::AttributeDecl object:
xml = <<DOCTYPE
<!DOCTYPE note [
<!ELEMENT payment (#PCDATA)>
<!ATTLIST payment type CDATA "check">
]>
DOCTYPE
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(DTD: {
name = "note",
children = [
#(ElementDecl: { "<!ELEMENT payment (#PCDATA)>\n" }),
#(AttributeDecl: { "<!ATTLIST payment type CDATA \"check\">\n" })]
})]
})Conditional Sections
\Nokogiri parses an XML conditional section
into a Nokogiri::XML::EntityDecl object:
xml = <<DOCTYPE
<!DOCTYPE note [
<!ENTITY % draft 'INCLUDE' >
]>
DOCTYPE
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(DTD: {
name = "note",
children = [ #(EntityDecl: { "<!ENTITY % draft \"INCLUDE\">\n" })]
})]
})Character References
\Nokogiri parses an XML character reference
(such as &9792;)
and replaces it with a character such as ('♀'):
xml = <<ELE
<root>
<name>
<vorname>Marie</vorname>
<nachname>Müller</nachname>
<geschlecht>♀</geschlecht>
</name>
</root>
ELE
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [
#(Text "\n "),
#(Element: {
name = "name",
children = [
#(Text "\n "),
#(Element: { name = "vorname", children = [ #(Text "Marie")] }),
#(Text "\n "),
#(Element: { name = "nachname", children = [ #(Text "Müller")] }),
#(Text "\n "),
#(Element: { name = "geschlecht", children = [ #(Text "♀")] }),
#(Text "\n ")]
}),
#(Text "\n")]
})]
})Entity References
\Nokogiri parses an XML entity reference
(such as <)
and replaces it with text such as ('<'):
xml = '<root>An entity reference is needed for the less-than character (<).</root>'
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "root",
children = [ #(Text "An entity reference is needed for the less-than character (<).")]
})]
})Entity Declarations
\Nokogiri parses an XML entity declaration
into a Nokogiri::XML::EntityDecl object:
xml = <<DTD
<!DOCTYPE note [
<!ENTITY company "Example Corp">
]>
DTD
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(DTD: {
name = "note",
children = [ #(EntityDecl: { "<!ENTITY company \"Example Corp\">\n" })]
})]
})Document Fragments
When an XML string has more than one top-level tag,
\Nokogiri document parsing captures only the first top-level tag
(which becomes the root element)
and ignores other top-level tags (and their children);
this may not be the desired result:
xml = <<FRAGMENT
<top0>
<ele0/>
<ele1/>
</top0>
<top1/>
<top2/>
FRAGMENT
doc = Nokogiri::XML.parse(xml)
doc
# =>
#(Document: {
name = "document",
children = [
#(Element: {
name = "top0",
children = [
#(Text "\n "),
#(Element: { name = "ele0" }),
#(Text "\n "),
#(Element: { name = "ele1" }),
#(Text "\n")]
})]
})```
To capture all top-level tags, use \Nokogiri *fragment parsing*
via method Nokogiri::XML::DocumentFragment.parse:xml = <<FRAGMENT
FRAGMENT
fragment = Nokogiri::XML::DocumentFragment.parse(xml)
fragment
=>
#(DocumentFragment: {
name = "#document-fragment",
children = [
#(Element: {
name = "top0",
children = [
#(Text "\n "),
#(Element: { name = "ele0" }),
#(Text "\n "),
#(Element: { name = "ele1" }),
#(Text "\n")]
}),
#(Text "\n"),
#(Element: { name = "top1" }),
#(Text "\n"),
#(Element: { name = "top2" }),
#(Text "\n")]
})
Note that:
- The returned object is a Nokogiri::XML::DocumentFragment object
(not a Nokigiri::XML::Document object).
- The fragment has three children of class Nokogiri::XML::Element
(which in a document is not allowed).
---
--- METRICS ---
- Files Extracted: 4
- Estimated Token Budget: ~7533 tokens
- Canonical Reference: https://codewiki.google/github.com/sparklemotion/nokogiri