ferret Documentation

Declarative data automation language and Go runtime for structured extraction workflows.

6,012 stars Go Go 1 file · ~11,932 tokens #browser-automation#chrome-devtools-protocol#data-automation#data-extraction#dsl#go#golang#golang-library
RAW Doc

1. Project Overview & Quickstart (MontFerret/ferret)

File: README.md

Ferret

Explore Ferret v2

Ferret v2 is currently in alpha. You can try the new syntax in the playground and read more about the design behind the new runtime and language capabilities:

Notice: This branch contains the upcoming Ferret v2. For the stable v1 release, please visit Ferret v1.

What is it?

Ferret is a declarative-first, expression-oriented embedded language and runtime for data automation.

FQL combines querying, transformation, synchronization, and structured results with host-defined values and capabilities. Applications can embed the runtime and decide exactly which functions, modules, data, and external operations a program can use.

The language keeps a declarative core and adds domain-oriented orchestration plus constrained mutable state for automation that cannot be expressed as a pure data transformation. It is deliberately focused rather than a general-purpose scripting language.

Features

  • Purpose-built declarative language for querying, transforming, synchronizing, and automating structured data
  • Embeddable Go runtime with reusable compiled plans and isolated execution sessions
  • Capability-based host values for exposing application objects, resources, and external systems directly to FQL
  • Unified query model for browsers, APIs, databases, documents, and custom data sources
  • Extensible runtime through namespaced functions, modules, hooks, and custom value types
  • Event-driven synchronization and dispatch for interacting with asynchronous and stateful resources
  • Managed resource lifecycle for files, connections, cursors, streams, and other host resources
  • Bytecode VM and portable programs for efficient repeated execution and precompiled artifacts

Getting started

bash
go get github.com/MontFerret/ferret/v2@latest

There are currently two ways to start with Ferret v2:

  • Native v2 API - recommended for new projects
  • compat module - recommended as a first migration step for existing v1 integrations

New projects

Use the native v2 API built around the following flow:

text
Engine -> compile query -> create session -> run
go
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/MontFerret/ferret/v2/pkg/engine"
)

func main() {
	ctx := context.Background()

	eng, err := engine.New()
	if err != nil {
		log.Fatal(err)
	}
	defer eng.Close()

	plan, err := eng.Compile(`return 1 + 1`)
	if err != nil {
		log.Fatal(err)
	}

	session, err := plan.NewSession()
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	result, err := session.Run(ctx)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result.Content)
}

Migration from v1

Ferret v2 introduces a new architecture and public API, so existing Go applications should migrate in two stages: first to the v2 compatibility API, then incrementally to the native v2 API.

Run the ferret migrate command from anywhere inside the application's Go module:

bash
ferret migrate --dry-run # List the files that would change
ferret migrate --print   # Print a unified diff without changing files
ferret migrate           # Apply the migration

The command rewrites the documented v1 imports to their v2 compatibility packages, updates go.mod and go.sum as required by those rewrites, and formats changed Go files. Generated, vendored, and nested-module files are left untouched. Unsupported v1 imports are reported as manual follow-up; if the project vendors dependencies, run go mod vendor after applying the migration.

This is only the mechanical compatibility stage. The command does not convert application logic to the native v2 API or migrate drivers and other unsupported v1 packages. Running it again on an already-migrated compatibility project is a no-op and does not upgrade the Ferret v2 dependency merely because the CLI is newer.

After applying the migration, address any reported manual follow-up, build and test the application, and then migrate to the native v2 API over time. The compatibility layer is a migration aid, not the long-term preferred API; new projects should use the native v2 packages directly.

Alpha status

Ferret v2 is currently in active development.

Alpha releases are intended for early adopters, experimentation, and feedback. Some APIs and language features may still change before the stable v2 release.

Maintainers

Support Ferret

Ferret is supported by organizations and community members who help fund its continued development. View all supporters.

File: docs/maintainers/core-api-reference.md

Ferret Core API Artifacts

Ferret publishes a versioned API Reference and sibling API Catalog with the canonical identity
montferret/core. This identity describes optional built-in functionality
provided by Ferret itself. It is not an installable Registry module and must not
be added to a module manifest as a dependency.

Authority and scope

The function definitions produced by stdlib.Full() are the authority for the
published API. Generation enumerates that in-process registry, including every
fixed arity, variadic definition, overload, root function, and nested namespace.
Unregistered Go declarations are not published.

The generator then loads pkg/stdlib source with go/packages and resolves each
registered function to its declaration. The literal testing assertion catalog is
resolved statically: each entry supplies its public name, statically named
descriptor, and explicit negation support. Descriptor args.min and args.max
bounds define the fixed overloads for both t and, when enabled, t::not. Any
unresolved declaration, dynamic or malformed catalog entry, documentation error,
arity contradiction, or runtime/source mismatch fails generation without writing
a partial artifact.

Host-function qualified names are case-insensitive in FQL and have one canonical
lowercase presentation in the registry and generated API Reference. This includes
every namespace segment; casing compatibility is represented by lookup rather
than duplicate aliases.

The API Reference, API Catalog, and discovery-index wire contracts belong to
github.com/MontFerret/specs. Ferret pins
the released Specs version in the independent generator and publisher modules,
which validate every completed reference and index. Ferret's root module does
not carry those tooling dependencies. Ferret does not depend on Barn for
generation or publication.

Documentation authoring

Registered declarations use the strict structured format parsed by
api.ParseDocumentation:

go
// split divides a string at each separator.
// @param value {String} Source string.
// @param separator {String} Separator string.
// @return {String[]} Split values.
// @throws {TypeError} The supplied values cannot be converted.

The rules are intentionally strict:

  • Every generated signature has non-empty prose and exactly one @return.
  • @param uses @param name {Type} Description; JSDoc optional-name brackets
    and dash separators are invalid.
  • Fixed declarations document exactly their registered parameter count.
  • Variadic declarations document at least one logical parameter and are emitted
    only as variadic signatures.
  • Overload helpers carry their own complete documentation. Fixed overloads are
    ordered by arity before any variadic signature.
  • Parameter and @throws order is retained as authored.
  • Parameters are flat. Describe nested map fields in the parent parameter's
    description instead of using names such as params.mode.
  • Assertion descriptor prose is namespace-neutral because the same descriptor
    documents both positive and t::not overloads. The descriptor documents its
    maximum argument list; each fixed overload receives the corresponding prefix.

Run the focused authoring and parity checks after changing stdlib registration
or documentation:

sh
GOWORK=off go -C tools/apiref test ./internal/analyzer

Local generation

Run the generator from the repository root with an unprefixed canonical SemVer:

sh
GOWORK=off go -C tools/apiref run . \
  -version 2.0.0-alpha.45 \
  -o /tmp/montferret-core-api.json \
  -catalog /tmp/montferret-core-catalog.json

The command emits diagnostics only on stderr and writes deterministic,
two-space-indented JSON with one trailing newline. api.json remains the
canonical callable API. catalog.json contains presentation categories for
global and namespaced functions. Each catalog member identifies a function by
its namespace and name; the empty namespace identifies a global function.
Categories such as math, io, and testing are presentation concepts, not
callable Ferret namespaces. The generator contains no deployment domain.

Release publication

The Publish Ferret Core API Artifacts workflow runs independently from
dependent-release notifications. It accepts release.published events and
manual dispatches, verifies that the selected canonical v2 tag belongs to a
published GitHub release, checks out that exact tag, strips the leading v, and
generates both artifacts.

The existing gh-pages branch is updated through the repository-local
scripts/publish-core-api.sh wrapper. Publication validates the existing index
and every referenced artifact before mutation, preserves unrelated Pages files,
atomically installs one version directory containing api.json and
catalog.json, recomputes the unchanged API index, commits once, and performs a
normal non-force push. Version directories, index entries, and hrefs are
immutable and cannot be republished even when bytes match. Existing API-only
version directories remain valid legacy state and are never backfilled. A stale
push fails; published history is never rewritten.

All gh-pages writers use the repository-wide gh-pages-writer concurrency key
with cancellation disabled. Unit and integration benchmarks still execute in
parallel, but their two results are persisted downstream by one serialized job.

Publication starts with the first release that contains this generator. Older
v2 alpha tags are not backfilled. A prerelease-only index omits latest; once a
stable release exists, latest identifies the greatest stable SemVer.

The public documents are:

  • https://ferretlang.org/ferret/index.json
  • https://ferretlang.org/ferret/versions/<version>/api.json
  • https://ferretlang.org/ferret/versions/<version>/catalog.json

These URLs share the existing Ferret Pages root, so benchmark history and other
site files remain alongside the API Reference without being regenerated.

2. Official Technical Reference & Guides (MontFerret/MontFerret.github.io)

File: README.md

Ferret Website

Built with Hugo

Prerequisites

Getting Started

Installing dependencies

bash
go mod tidy && mage install

Dev server

bash
mage serve

The standard development server does not generate the Pagefind search bundle.

Production build

Build the static site and its client-side search index:

bash
mage build

The generated site, including Pagefind assets under public/pagefind/, is written to public/.

Search preview

Build the site and search index, then serve the generated static output with Pagefind's preview server:

bash
mage serveSearch

Tool documentation

When adding or reorganizing first-party tool pages, follow the
tool documentation conventions.

Generating the Standard Library reference

The website generates the Standard Library reference from the published Ferret
Core API Reference selected by runtime.v2 in data/versions.yaml. The
selected release must include its sibling API Catalog, whose ordered categories
drive the reference navigation.

bash
mage generateStdlib

mage generate runs the same target. Normal development and production builds
generate the reference automatically; contributors do not download API files by
hand. See Standard Library reference generation
for the publication, validation, and generated-file contracts.

Published Ferret tool releases update data/versions.yaml through a
website-owned pull-request workflow. See Release version updates
for the dispatch contract, manual retry procedure, and GitHub App setup.

Deployment

GitHub Actions builds, tests, and deploys the site to GitHub Pages when changes
are pushed to main. Pull requests targeting main run the same checks without
deploying, and maintainers can also start the workflow manually from main in
the Actions tab.

The Pages source must be set to GitHub Actions in the repository settings.
The custom domain remains ferretlang.org; configure it and HTTPS in the Pages
settings rather than relying on the generated CNAME file alone.

File: content/blog/ferret-module-registry/index.md

Ferret now has a module Registry.

Until now, modules could be published as Go packages, but there was no shared place to discover them, compare versions, check compatibility, or find their documentation. You needed to know that a module existed and where its source lived before you could use it.

The [Ferret Registry]({{< ref "/registry" >}}) gives modules a common home. It is the public catalog that people browse and that the Ferret CLI reads when searching for or installing modules.

Barn is the Git-backed source repository and publication pipeline behind the Registry. Contributors submit registration records to Barn for validation and human review; Barn then generates the public Registry from the accepted records.

The Registry is available now as part of the Ferret v2 alpha. The workflow is intentionally practical: find a module, install a compatible release into a Go application, or publish your own module through a Barn pull request.

One place for modules

A Registry entry brings together the information needed to understand and install a module:

  • its canonical owner/name identity
  • available versions and their Ferret compatibility
  • the FQL namespace exposed at runtime
  • the public source repository and exact release commit
  • the installable Go package path
  • versioned module documentation

These identifiers are related, but they are not interchangeable. For the Archive module, they look like this:

Concept Value
Registry identity montferret/archive
FQL namespace ARCHIVE
Go package path github.com/MontFerret/contrib/modules/archive
Runtime registration ferret.WithModules(archive.New())

The Registry identity is how users search for and install the module. The namespace is what FQL scripts call. The package path is what a Go application compiles. Keeping these identities explicit lets a module use the names that make sense at each boundary without relying on naming conventions to guess the others.

Find a module

The Registry is the easiest way to browse the catalog from the web. Each module page shows its description, namespace, versions, compatibility, source, and published documentation.

The CLI provides the same discovery workflow from a terminal. Search by identity or description:

{{< terminal command="true" >}}
ferret mod search archive
{{< /terminal >}}

Then inspect a module before adding it to an application:

{{< terminal command="true" >}}
ferret mod info montferret/archive
{{< /terminal >}}

mod info shows the available and selected versions, runtime namespace, Ferret compatibility, repository, commit, and documentation URL. This makes the Registry useful before installation, not only as a package lookup service.

Install a module into an application

Run ferret mod install from the Go application that hosts Ferret:

{{< terminal command="true" >}}
ferret mod install montferret/archive
{{< /terminal >}}

Without an explicit version, the CLI selects the newest registered release compatible with the Ferret version already used by the project. It does not silently move the application to a different Ferret release to make a module fit.

You can also request a specific release:

{{< terminal command="true" >}}
ferret mod install montferret/[email protected]
{{< /terminal >}}

An exact release still has to declare compatibility with the project's Ferret version.

The installer resolves the Registry record, adds the published Go package, updates the application’s ferret.New(...) composition, and verifies that the affected package builds before applying the staged changes. If the application is missing a safe Ferret dependency or composition helper, interactive use can show and request approval for those prerequisite changes.

For project prerequisites, non-interactive setup, and configuration after installation, see [Install a module]({{< ref "/docs/modules/install" >}}).

Publish a module

Publishing starts from a tagged Go module in a public Git repository. The Registry does not accept uploaded package archives. Instead, Ferret resolves the public tag, inspects the files at its exact commit, and prepares the immutable Barn records that identify the release.

The example below publishes ziflex/kvplugin at version 1.0.0.

Complete the release metadata

Keep ferret.yaml, go.mod, and README.md together at the module root. A complete standalone manifest looks like this:

yaml
$schema: https://schemas.ferretlang.org/module/v1.json
name: ziflex/kvplugin
namespace: KV
version: 1.0.0
license: Apache-2.0
documentation: https://github.com/ziflex/ferret-kvplugin#readme
  url: https://github.com/ziflex/ferret-kvplugin
compatibility:
  ferret: ">=2.0.0-alpha.44 <3.0.0"
keywords:
  - cache
categories:
  - data

Module Manifest v1 is strict. Unknown fields, duplicate keys, malformed versions, mixed-case Registry identities, invalid URLs, and invalid SPDX license expressions are rejected.

The adjacent go.mod supplies the installable package path. The README.md supplies the versioned documentation Barn will snapshot and publish as Markdown and sanitized HTML. A module README.md typically includes Overview, Installation, Quick Start, and API Reference sections.

compatibility.ferret is optional in the schema, but Registry releases should declare it. The installer needs a usable compatibility range to select a release for a host application.

Tag and push the release

Commit the manifest, module implementation, tests, go.mod, and README.md before creating the release tag. For a standalone module, the default tag is v<version>:

{{< terminal command="true" >}}
git tag v1.0.0
git push origin v1.0.0
{{< /terminal >}}

For a monorepo module, the default tag is <repository.directory>/v<version>. Pass --tag to each publication command when a repository uses a different convention. The complete [publication guide]({{< ref "/docs/modules/publish" >}}) covers both layouts.

Validate the release

After the tag is public, validate the complete release without authenticating or changing anything on GitHub:

{{< terminal command="true" >}}
ferret mod publish --dry-run
{{< /terminal >}}

The command checks the local manifest against the current Registry, resolves the public tag to its exact commit, and verifies the README.md, go.mod, module identity, version, and package path. Nothing is submitted, and no module code is checked out or executed.

Authenticate with GitHub

Before submitting, set GH_TOKEN or GITHUB_TOKEN. If neither is set, Ferret reads the current github.com token by running:

{{< terminal command="true" >}}
gh auth token --hostname github.com
{{< /terminal >}}

Authenticate the GitHub CLI when needed:

{{< terminal command="true" >}}
gh auth login --hostname github.com
{{< /terminal >}}

The credential must be able to write to your personal Barn fork and open a pull request against MontFerret/barn.

Submit the publication

Run the default command from the module root:

{{< terminal command="true" >}}
ferret mod publish
{{< /terminal >}}

After validation, the CLI uses the GitHub API to create or reuse your personal Barn fork, create a focused publication branch, and open a pull request against MontFerret/barn. A module's first release prepares its module and version records; later releases prepare only the new version record. You do not need a local Barn checkout or knowledge of its record layout.

To inspect the records without submitting them, print a deterministic, versioned JSON document:

{{< terminal command="true" >}}
ferret mod publish --print
{{< /terminal >}}

--dry-run and --print cannot be combined, and neither mode submits records. Retries are safe: an already-published version exits successfully, while an exact open pull request or publication branch is reused. Ferret refuses to overwrite an immutable Registry record or a divergent branch. If a stale publication branch in your fork contains different changes, delete it before retrying.

Publication model

The Registry's publication process is deliberately Git-backed and review-based.

Specs owns the strict portable contracts for module manifests and Registry records. Barn owns repository inspection, cross-document checks, publication history, and generation of the public catalog. The CLI consumes the resulting versioned artifacts and applies compatibility rules when installing a module.

Before submitting, authenticate with GitHub using GH_TOKEN, GITHUB_TOKEN, or the GitHub CLI. See the publication guide for authentication and permission details.

Available during the v2 alpha

The Registry and ferret mod workflow are available now with the Ferret v2 alpha CLI. They are useful today, but the wider v2 ecosystem is still moving toward beta, so feedback on module authoring, discovery, compatibility, and publication is especially useful.

You can:

  • [browse the Registry]({{< ref "/registry" >}})
  • [install a module]({{< ref "/docs/modules/install" >}})
  • [initialize a module project]({{< ref "/docs/modules/develop" >}})
  • [build a complete module with the Ferret SDK]({{< ref "/docs/guides/writing-plugins" >}})
  • [publish a module through Barn]({{< ref "/docs/modules/publish" >}})
  • [review the ferret mod command reference]({{< ref "/docs/tools/cli/mod" >}})

If you have a module that would be useful outside your own application, the Registry now gives it a path from a public Git tag to something other Ferret users can find, inspect, and install.

File: content/blog/ferret-v0.8.1/index.md

With all big features, come small bugs. :)

With this new v0.8.1 release we are providing hot fixes for some of these small ones.

Broken CLICK function

In 0.7 CLICK and CLICK_ALL functions were doing a check before clicking and was returning a boolean value indicating whether a target element was clicked.
In 0.8 this logic got broken due to internal refactorings and the check was not performed which led to broken scripts using this returned value.

Scroll jumps

In 0.8, we introduct new interaction behavior with forms. Everytime you want to INPUT or CLICK Ferret would scroll into a target element before the operation. Unfortunately, we didn't add any checks whether the element is in a viewport, which caused weird page jumps.

Not that bad so far! If you find any new issues, please report.

File: content/blog/ferret-v0.8.2/index.md

Another portion of hot fixes has arrived!

v0.8.2 release brings some critical fixes.

Scrolling position is not centered

In 0.8, we introduced a new behavior for all user interactions. Before any actions like clicking ot typing, we scroll into a target element making it visible in a viewport. But the logic behind it, positioned elements on the top of the viewport, which led to some issues on some websites due to their markup - the elements were still invisible because they were under other elements.

Unable to set custom logger fields

This is a small fix/improvement for internal logger. There was no way to set a specific id for an execution. It makes it difficult to track script execution in embedded scenarios when you have a request id.
Now you can add as many fields as you want to the logger, thanks to Zerologger.

Invalid parent used for INNER_TEXT, INNER_HTML and friends

This is another issue that was triggered by internal refactoring. A new mechanism for retrieving inner HTML and text by selectors used document for all element look ups instead of a given parent element.

Unable to set custom headers

There was an error occuring whenever you pass values to headers object.

As always, if you find any new issues, please report.

File: content/blog/ferret-v0.8/index.md

Hooray, Ferret v0.8 has been released!

It's been a while since the last release, but we worked hard to bring new and better Ferret. This release has many new exciting features, but unfortunately, there are also some breaking changes.

You can find the full changelog here.

Let's go!

What's added

iframe

Ferret finally supports iframe elements.
When a page gets loaded, Ferret finds all available elements and provieds an access to them via the .frames property of a page object.

Here's an example of how to find a target frame:

{{< editor height="250px" readonly="true" >}}
LET page = DOCUMENT("https://www.w3schools.com/html/html_iframe.asp", {
driver: "cdp"
})

LET content = (
FOR f IN page.frames
FILTER f.URL == "https://www.w3schools.com/html/default.asp"
RETURN f.head.innerHTML
)

RETURN FIRST(content)
{{</ editor >}}

Alternatively, you can filter them out by url or access to a target iframe by index, if you know it's position.

Due to CORS security policies, you still may have issues with iframes if it points to another domain.

Namespaces

With this release, we introduce a new language feature - namespaces.

Namespaces allow library authors (and us) to isolate functions into dedicated sub sections.

Here is an example:

{{< code lang="fql" height="120px" >}}
LET blob = DOWNLOAD("https://raw.githubusercontent.com/MontFerret/ferret/master/assets/logo.png")

RETURN IO::FS::WRITE("logo.png", blob)
{{</ code >}}

To namespace a function, use the new namespace method. The namespace method is chainable:

{{< code lang="golang" height="380px" >}}
package main

import (
"context"
"encoding/json"
"fmt"
"os"
"strings"

text
"github.com/MontFerret/ferret/pkg/compiler"
"github.com/MontFerret/ferret/pkg/runtime/core"
"github.com/MontFerret/ferret/pkg/runtime/values"
"github.com/MontFerret/ferret/pkg/runtime/values/types"

)

func main() {
c := compiler.New()
c.Namespace("IO").Namespace("FS").RegisterFunction("Read", fs.Read)
}
{{</ code >}}

text
In future releases we will put HTML related functions into HTML:: namespaces.

XPath

A good web scraping tool needs XPath support, and Ferret finally has it!
Ferret provides simple interface to XPath engine for both drivers - CDP and HTTP.
It automatically detects the output value type and deserializes them accordingly.

{{< code lang="fql" height="90px" >}}
RETURN XPATH(page, "//div[contains(@class, 'form-group')]")
{{</ code >}}

{{< code lang="fql" height="90px" >}}
RETURN XPATH(page, "count(//div)")
{{</ code >}}

These two queries will return 2 different types:

  1. Returns an array of serialized elements (their inner HTML)
  2. Returns a number indicating how many "div" elements are on the page.

Regular expression operator

This release provides a shorthand for using regexp assertions:

{{< code lang="fql" height="90px" >}}
LET result = "foo" =~ "^f[o].$" // returns "true"
{{</ code >}}

{{< code lang="fql" height="90px" >}}
LET result = "foo" !~ "[a-z]+bar$" // returns "true"
{{</ code >}}

New functions to manipulate DOM

There are some cases when you might need to change the existing DOM. To help with that, we added the INNER_HTML_SET and INNER_TEXT_SET functions.

{{< code lang="fql" height="185px" >}}
// Using document and selector
INNER_HTML_SET(doc, "body", "Hello")
INNER_TEXT_SET(doc, "body", "Hello")

// Or an element directly
INNER_HTML_SET(doc.body, "Hello")
INNER_TEXT_SET(doc.body, "Hello")
{{</ code >}}

Viewport settings

In this release, you can override default values of a viewport in headless mode.

{{< code lang="fql" height="185px" >}}
LET doc = DOCUMENT(@url, {
driver: 'cdp',
viewport: {
width: 1920,
height: 1080
}
})
{{</ code >}}

Better emulation of user interaction

This is a big change in how Ferret handles page interactions.

Now Ferret interacts with pages in a more advanced way - your script can scrolls down or up to an element, moves the mouse, focuses and types... with random delays. Just like a real person!

Other

There are many other many small changes here and there, like adding FOCUS, ESCAPE_HTML, UNESCAPE_HTML and DECODE_URI_COMPONENT functions; improving performance; and changing internal design of some parts of the system.

What's broken

We try to maintain backwards compatibility, but some of the new features required serious design changes that lead to breaking compatibility with previous versions. As we approach to release v1.0, the API is becoming more stable and will require fewer dramatic changes.

text
Most of the breaking changes will affect only embedded solutions, use of HTML drivers in particular. No changes in the syntax, so no scripts need to change.

Virtual DOM structure

Work on iframe support required us to redesign the structure of the virtual DOM by introducing top level entity called HTMLPage:

{{< code lang="golang" height="440px" >}}
type HTMLPage interface {
core.Value
core.Iterable
core.Getter
core.Setter
collections.Measurable
io.Closer

text
IsClosed() values.Boolean
GetURL() values.String
GetMainFrame() HTMLDocument
GetFrames(ctx context.Context) (*values.Array, error)
GetFrame(ctx context.Context, idx values.Int) (core.Value, error)
GetCookies(ctx context.Context) (*values.Array, error)
SetCookies(ctx context.Context, cookies ...HTTPCookie) error
DeleteCookies(ctx context.Context, cookies ...HTTPCookie) error
PrintToPDF(ctx context.Context, params PDFParams) (values.Binary, error)
CaptureScreenshot(ctx context.Context, params ScreenshotParams) (values.Binary, error)
WaitForNavigation(ctx context.Context) error
Navigate(ctx context.Context, url values.String) error
NavigateBack(ctx context.Context, skip values.Int) (values.Boolean, error)
NavigateForward(ctx context.Context, skip values.Int) (values.Boolean, error)

}
{{</ code >}}

Previously, HTMLDocument contained the open page, but iframe nodes introduce the need to have multiple documents representing each node. This led to a new entity in the structure.

Driver API

Because of the changes in Virtual DOM structure, the driver API has been changed as well in order to be reasonable.

Driver.LoadDocument and LoadDocumentParams are renamed to Driver.Open and Params.

{{< code lang="golang" height="150px" >}}
type Driver interface {
io.Closer
Name() string
Open(ctx context.Context, params Params) (HTMLPage, error)
}
{{</ code >}}

Other

In the context of API stabilization and consistency, there are some other minor changes in vDOM elements like extra returned value (usually an error) or Get prefix in some methods.

File: content/blog/ferret-v0.9/index.md

Hello fellow miners, Ferret v0.9 has been released!

In this release, we mostly focused on bug fixes and filling gaps in user interaction functionality.

Let's see what we've got.

What's added

Clearing input values

Since early versions, Ferret has INPUT HTML function that allowed to type any value to input HTML element.
This works by appending the function parameter to any pre-existing value. While it works fine in many cases, there are some scenarios when a target input field has a preset value which needs to be erased. With this release, Ferret has a new methodINPUT_CLEAR`` that erases any data from a target input.

{{< code lang="fql" height="150px" >}}
LET doc = DOCUMENT(@url, {
driver: 'cdp'
})

RETURN INPUT_CLEAR(doc, '#my-input')
{{</ code >}}

Double and more click

Single, double, or more. Now it's possible to specify the number of clicks to CLICK function.

{{< code lang="fql" height="150px" >}}
LET doc = DOCUMENT(@url, {
driver: 'cdp'
})

RETURN CLICK(doc, '#like', 2)
{{</ code >}}

Unfocus

BLUR function removes focus from an active input field.

{{< code lang="fql" height="200px" >}}
LET doc = DOCUMENT(@url, {
driver: 'cdp'
})

FOCUS(doc, '#input')
BLUR(doc, '#input')

RETURN NONE
{{</ code >}}

Default headers and cookies

With this release, you can set default headers and/or cookies to HTML drivers.
One of the possible use cases is to create drivers with different custom names and pre-defined headers/cookies:

{{< code lang="golang" height="730px" >}}
cdp.NewDriver(
cdp.WithCustomName("cdp_headers"),
cdp.WithHeader("Single_header", []string{"single_header_value"}),
cdp.WithHeaders(drivers.HTTPHeaders{
"Multi_set_header": []string{"multi_set_header_value"},
"Multi_set_header2": []string{"multi_set_header2_value"},
}),
)

cdp.NewDriver(
cdp.WithCustomName("cdp_cookies"),
cdp.WithCookie(drivers.HTTPCookie{
Name: "single_cookie",
Value: "single_cookie_value",
Path: "/",
MaxAge: 0,
Secure: false,
HTTPOnly: false,
SameSite: 0,
}),
cdp.WithCookies([]drivers.HTTPCookie{
{
Name: "multi_set_cookie",
Value: "multi_set_cookie_value",
Path: "/",
MaxAge: 0,
Secure: false,
HTTPOnly: false,
SameSite: 0,
},
{
Name: "multi_set_cookie2",
Value: "multi_set_cookie2_value",
Path: "/",
MaxAge: 0,
Secure: false,
HTTPOnly: false,
SameSite: 0,
},
}),
)
{{</ code >}}

Params in dot notation

In previous versions of Ferret, it was possible to pass complex types as parameters, but it was impossible to read data from it using dot notation. Now it is possible. But even more - you can use params as dot notation segments:

{{< code lang="fql" height="90px" >}}
RETURN @one.@two.@three
{{</ code >}}

String literals

In addition to defining string literals with quotes (', ") and backtick (`), you can now define string literals with the forwardtick (´).

What's fixed

Open tabs on error

Ferret didn't close Chrome/Chromium tab if an error occurs during a page load.

CLICK

CLICK could not be used with both an element and selector.

Dot notation after a function call

Parser could not properly handle scenarios where dot notation was used right after a function call:

{{< code lang="fql" height="110px" >}}
LET items = [{name: "foo"}]
RETURN FIRST(items).name
{{</ code >}}

What's changed

Internal cleanup and refactoring

We've made some internal cleanups and minor refactoring.
There might be minor breaking changes for those who use HTML drivers directly.

2nd returned value

The following methods have an error as asecond returned value now:

  • HTMLNode.GetChildNodes
  • HTMLNode.GetChildNode
  • HTMLNode.QuerySelector
  • HTMLNode.QuerySelectorAll
  • HTMLNode.CountBySelector
  • HTMLElement.GetValue
  • HTMLElement.GetAttributes
  • HTMLElement.GetAttribute

Re-arranged methods

The following methods have been moved from HTMLDocument to HTMLElement interface:

  • SelectBySelector
  • FocusBySelector
  • HoverBySelector

Removed methods

The following method(s) have been removed from HTMLDocument interface:

  • MoveMouseBySelector

File: content/blog/ferret-v0.10.0/index.md

Hello fellow miners, Ferret v0.10.0 has been finally released!

It's been long and busy 6 months since the last release and many of you might thought that the project got abandoned.
We assure you it's not! But sometimes life happens and slows you down.

But we are back and ready to rock!

This release has many great features and improvements thank to all our contributors.
Let's take a look at the most important ones. The full changelog you can find here.

What's added

I/O functions

You asked - we added! Finally, I/O functions have arrived.
In the beginning of the project, we were concerned about security threats that access to filesystem poses, but seeing how demanded this functionality is we agreed to add it.
Now, Ferret has new I/O namespaces, with the following functions:

  • IO::FS::READ
  • IO::FS::WRITE
  • IO::NET::HTTP:GET
  • IO::NET::HTTP:POST
  • IO::NET::HTTP:PUT
  • IO::NET::HTTP:DELETE
  • IO::NET::HTTP:REQUEST

Enjoy!

Loading HTML string into memory

Another popular feature request - possibility to load a prefetched HTML string into Ferret.
With this release, you can finally do it with HTML_PARSE function.

{{< code lang="fql" height="180px" >}}
LET file = IO::FS::READ(@myfile)

LET doc = HTML_PARSE(TO_STRING(file), {
driver: 'cdp' // or 'http'
})

RETURN INNER_TEXT(doc, 'title')
{{</ code >}}

Parameter value availability check

In this release, we do pre-runtime check whether all values are provided for parameters used by a script.
It was very frustrating when your script was failing in the middle of its execution just because you forgot to provide a value for one of the paramaters.

What's changed

Case insensitivity

Finally, FQL keywords are case insensitive!

That's how Google Search query looks like in lower case now:

{{< code lang="fql" height="330px" >}}
let google = document("https://www.google.com/", {
driver: "cdp",
})

input(google, 'input[name="q"]', "ferret")
click(google, 'input[name="btnK"]')

wait_navigation(google)

for result in elements(google, '.g')
filter trim(result.attributes.class) == 'g'
return {
title: inner_text(result, 'h3'),
url: inner_text(result, 'cite')
}
{{</ code >}}

Improved CDP driver performance

CDP driver performance has been drastically improved that brings CPU usage down from 60% for 3 pages to <1%.

What's fixed

Redirects

It was a long lasting problem that Ferret could not correctly handle redirects that could occur during a page navigation that was leading to timeouts or corrupt page state.

Now it has finally been fixed!
WAIT_NAVIGATION function has 2nd optional parameter that can be an object with the following fields:

{{< code lang="golang" height="135px" >}}
type Parameters struct {
TargetURL string
Timeout time.Duration
}
{{</ code >}}

TargetURL is a regexp string that can be used to give Ferret a hint what the destination url is:

{{< code lang="fql" height="220px" >}}
LET doc = DOCUMENT("http://waos.ovh/redirect.html", {
driver: 'cdp'
})

CLICK(doc, '.click')

WAIT_NAVIGATION(doc, { targetURL: "redirect2.html" })

RETURN ELEMENT(doc, '.title')
{{</ code >}}

File: content/blog/ferret-v0.11.0/index.md

Hello fellow miners,

Today, we are living in unprecedented times. There are almost 5 million people infected by COVID-19 and thousands die every day in the world.

During this time, the highest priority is to stay safe and healthy, take care of family.

Ferret team has been locked down in their homes with their families as everyone around the globe, but still has managed to make new Ferret v0.11.0 release!

Thanks to all our contributors for their help - we have done a lot of cool features.

So, let’s take a look!

What's added

USE statement

There is no need to write a full function path anymore! Just add USE PATH::TO::PACKAGE at the top of your query.

{{< code lang="fql" height="190px" >}}
USE IO::FS

LET favicon = DOWNLOAD('https://www.google.com/favicon.ico')

// 'WRITE' the same as 'IO::FS::WRITE'.
// RETURN IO::FS::WRITE('google.favicon.ico', favicon) also valid.
RETURN WRITE('google.favicon.ico', favicon)
{{</ code >}}

PATH functions

New functions for working with file paths.

{{< code lang="fql" height="90px" >}}
LET filename = 'main.go'

RETURN PATH::EXT(filename) == '.go'
{{</ code >}}

What's changed

Scroll options

Scroll document's window and elements according to MDN

{{< code lang="fql" height="220px" >}}
LET doc = DOCUMENT(@url, { driver: 'cdp' })

SCROLL_BOTTOM(doc, {
behavior: 'auto',
block: 'end',
inline: 'end'
})

RETURN 1
{{</ code >}}

What's fixed

RANDOM_TOKEN

RANDOM_TOKEN return random string now! (even on Windows)

IO::FS::WRITE

Now files are created with read access automatically.

Around the Ferret

OSS-Fuzz platform

We have tried to connect Ferret with OSS-Fuzz platform. But Google has rejected Ferret :(

File: content/blog/ferret-v0.11.1/index.md

Hello fellow miners,

Ferret v0.11.1 has been released with a hotfix that solves an error in some functions that do screen positioning using scrolling.

File: content/blog/ferret-v0.12.0/index.md

Hello fellow miners,

We are happy to announce a new Ferret release - Ferret v0.12.0.

This release has a new useful module in the standard library - testing!
Let's dig in. The full changelog you can find here.

What's added

Assertion library

A new T namespace has been added to the standard library that provides some helpful methods for making assertions.
The functions can be used to test UI, validate data and ensure the correctnes of a script.

{{< editor lang="fql" height="190px" readonly="true" >}}

LET doc = DOCUMENT('https://github.com/', { driver: "cdp" })

T::TRUE(ELEMENT_EXISTS(doc, "#user[login]"))
T::TRUE(ELEMENT_EXISTS(doc, "#user[email]"))
T::TRUE(ELEMENT_EXISTS(doc, "#user[password]"))

RETURN NONE

{{</ editor >}}

Read about available functions here.

New FRAMES function

A new helper function that finds HTML frames by a given property selector.

{{< editor lang="fql" height="190px" readonly="true" >}}
LET page = DOCUMENT('https://mockery.ferretlang.org/scenarios/iframes/', {
driver: 'cdp'
})

LET frame = FRAMES(page, "url", "https://mockery.ferretlang.org/")

RETURN FIRST(frame).url
{{</ editor >}}

iFrame navigation handling

Finally, we can control navigation of nested iframes! Whenever your page has a nested iframe that performs some nested navigation, you can tell Ferret to wait for its completion.

{{< editor lang="fql" height="580px" readonly="true" >}}
LET page = DOCUMENT('https://mockery.ferretlang.org/scenarios/iframes/', {
driver: 'cdp'
})

LET innerPage = FIRST(FRAMES(page, "url", "https://www.mockery.ferretlang.org/"))

T::NOT::NONE(innerPage)

CLICK(innerPage, "#navbar-burger")
WAIT_CLASS(innerPage, "#primary-nav", "is-active")
CLICK(innerPage, "#blog")

WAIT_NAVIGATION(page, {
frame: innerPage
})

// We need to get a new document that we have navigated to
LET innerPage2 = FIRST(FRAMES(page, "url", "https://www.ferretlang.org/blog/"))

WAIT_ELEMENT(innerPage2, ".blog")

LET item = ELEMENT(innerPage2, ".blog-post a")
LET targetURL = item.attributes.href

CLICK(item)

WAIT_NAVIGATION(page, {
frame: innerPage2
})

// We need to get a new document that we have navigated to
LET innerPage3 = FIRST(FRAMES(page, "url", targetURL))

RETURN INNER_TEXT(innerPage3, ".content")
{{</ editor >}}

What's changed

Removed property caching and tracking

This could be a breaking change, so, please, test your scripts.
In previous versions, the CDP driver used aggressive caching and changes tracking on properties like .class, .attributes and etc., in order to redcue amount of calls to a browser, which led to high CPU usage when a runtime held a big amount of elements in memory.
To tackle the problem, the caching and tracking were removed and thus, every time element's property is accessed, a latest value gets retrived from a browser, without any further caching. If something needs to be cahced, it's user's responsibility to chace it in local variable.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#browser-automation Browse browser-automation catalog → #chrome-devtools-protocol Browse chrome-devtools-protocol catalog → #data-automation Browse data-automation catalog → #data-extraction Browse data-extraction catalog → #dsl Browse dsl catalog → #go Browse go catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →