## 1. Project Overview & Quickstart (MontFerret/ferret) ## File: README.md # Ferret --- ## Explore Ferret v2 Ferret v2 is currently in alpha. You can try the new syntax in the playground and read more about the design behind the new runtime and language capabilities: - [Try Ferret v2 in the Playground](https://ferretlang.org/try/) - [On the Road to Ferret v2](https://ferretlang.org/blog/ferret-v2-announcement/) - [Inside Ferret v2: The New Execution Model](https://ferretlang.org/blog/ferret-v2-execution-model/) - [Inside Ferret v2: New Language Capabilities](https://ferretlang.org/blog/ferret-v2-new-syntax/) --- > **Notice:** This branch contains the upcoming **Ferret v2**. For the stable v1 release, please visit [Ferret v1](https://github.com/MontFerret/ferret/tree/v1). --- ## What is it? Ferret is a declarative-first, expression-oriented embedded language and runtime for data automation. FQL combines querying, transformation, synchronization, and structured results with host-defined values and capabilities. Applications can embed the runtime and decide exactly which functions, modules, data, and external operations a program can use. The language keeps a declarative core and adds domain-oriented orchestration plus constrained mutable state for automation that cannot be expressed as a pure data transformation. It is deliberately focused rather than a general-purpose scripting language. ### Features - Purpose-built declarative language for querying, transforming, synchronizing, and automating structured data - Embeddable Go runtime with reusable compiled plans and isolated execution sessions - Capability-based host values for exposing application objects, resources, and external systems directly to FQL - Unified query model for browsers, APIs, databases, documents, and custom data sources - Extensible runtime through namespaced functions, modules, hooks, and custom value types - Event-driven synchronization and dispatch for interacting with asynchronous and stateful resources - Managed resource lifecycle for files, connections, cursors, streams, and other host resources - Bytecode VM and portable programs for efficient repeated execution and precompiled artifacts ## Getting started ```bash go get github.com/MontFerret/ferret/v2@latest ``` There are currently two ways to start with Ferret v2: - Native v2 API - recommended for new projects - `compat` module - recommended as a first migration step for existing v1 integrations ### New projects Use the native v2 API built around the following flow: ``` Engine -> compile query -> create session -> run ``` ```go package main import ( "context" "fmt" "log" "github.com/MontFerret/ferret/v2/pkg/engine" ) func main() { ctx := context.Background() eng, err := engine.New() if err != nil { log.Fatal(err) } defer eng.Close() plan, err := eng.Compile(`return 1 + 1`) if err != nil { log.Fatal(err) } session, err := plan.NewSession() if err != nil { log.Fatal(err) } defer session.Close() result, err := session.Run(ctx) if err != nil { log.Fatal(err) } fmt.Println(result.Content) } ``` ### Migration from v1 Ferret v2 introduces a new architecture and public API, so existing Go applications should migrate in two stages: first to the v2 compatibility API, then incrementally to the native v2 API. Run the [`ferret migrate`](https://github.com/MontFerret/cli#migrating-embedded-ferret-applications) command from anywhere inside the application's Go module: ```bash ferret migrate --dry-run # List the files that would change ferret migrate --print # Print a unified diff without changing files ferret migrate # Apply the migration ``` The command rewrites the documented v1 imports to their v2 compatibility packages, updates `go.mod` and `go.sum` as required by those rewrites, and formats changed Go files. Generated, vendored, and nested-module files are left untouched. Unsupported v1 imports are reported as manual follow-up; if the project vendors dependencies, run `go mod vendor` after applying the migration. This is only the mechanical compatibility stage. The command does not convert application logic to the native v2 API or migrate drivers and other unsupported v1 packages. Running it again on an already-migrated compatibility project is a no-op and does not upgrade the Ferret v2 dependency merely because the CLI is newer. After applying the migration, address any reported manual follow-up, build and test the application, and then migrate to the native v2 API over time. The compatibility layer is a migration aid, not the long-term preferred API; new projects should use the native v2 packages directly. ### Alpha status Ferret v2 is currently in active development. Alpha releases are intended for early adopters, experimentation, and feedback. Some APIs and language features may still change before the stable v2 release. ## Maintainers - [Versioned Ferret Core API Reference](docs/maintainers/core-api-reference.md) ## Support Ferret Ferret is supported by organizations and community members who help fund its continued development. [View all supporters](https://ferretlang.org/sponsor/). --- ## File: docs/maintainers/core-api-reference.md # Ferret Core API Artifacts Ferret publishes a versioned API Reference and sibling API Catalog with the canonical identity `montferret/core`. This identity describes optional built-in functionality provided by Ferret itself. It is not an installable Registry module and must not be added to a module manifest as a dependency. ## Authority and scope The function definitions produced by `stdlib.Full()` are the authority for the published API. Generation enumerates that in-process registry, including every fixed arity, variadic definition, overload, root function, and nested namespace. Unregistered Go declarations are not published. The generator then loads `pkg/stdlib` source with `go/packages` and resolves each registered function to its declaration. The literal testing assertion catalog is resolved statically: each entry supplies its public name, statically named descriptor, and explicit negation support. Descriptor `args.min` and `args.max` bounds define the fixed overloads for both `t` and, when enabled, `t::not`. Any unresolved declaration, dynamic or malformed catalog entry, documentation error, arity contradiction, or runtime/source mismatch fails generation without writing a partial artifact. Host-function qualified names are case-insensitive in FQL and have one canonical lowercase presentation in the registry and generated API Reference. This includes every namespace segment; casing compatibility is represented by lookup rather than duplicate aliases. The API Reference, API Catalog, and discovery-index wire contracts belong to [`github.com/MontFerret/specs`](https://github.com/MontFerret/specs). Ferret pins the released Specs version in the independent generator and publisher modules, which validate every completed reference and index. Ferret's root module does not carry those tooling dependencies. Ferret does not depend on Barn for generation or publication. ## Documentation authoring Registered declarations use the strict structured format parsed by `api.ParseDocumentation`: ```go // split divides a string at each separator. // @param value {String} Source string. // @param separator {String} Separator string. // @return {String[]} Split values. // @throws {TypeError} The supplied values cannot be converted. ``` The rules are intentionally strict: - Every generated signature has non-empty prose and exactly one `@return`. - `@param` uses `@param name {Type} Description`; JSDoc optional-name brackets and dash separators are invalid. - Fixed declarations document exactly their registered parameter count. - Variadic declarations document at least one logical parameter and are emitted only as variadic signatures. - Overload helpers carry their own complete documentation. Fixed overloads are ordered by arity before any variadic signature. - Parameter and `@throws` order is retained as authored. - Parameters are flat. Describe nested map fields in the parent parameter's description instead of using names such as `params.mode`. - Assertion descriptor prose is namespace-neutral because the same descriptor documents both positive and `t::not` overloads. The descriptor documents its maximum argument list; each fixed overload receives the corresponding prefix. Run the focused authoring and parity checks after changing stdlib registration or documentation: ```sh GOWORK=off go -C tools/apiref test ./internal/analyzer ``` ## Local generation Run the generator from the repository root with an unprefixed canonical SemVer: ```sh GOWORK=off go -C tools/apiref run . \ -version 2.0.0-alpha.45 \ -o /tmp/montferret-core-api.json \ -catalog /tmp/montferret-core-catalog.json ``` The command emits diagnostics only on stderr and writes deterministic, two-space-indented JSON with one trailing newline. `api.json` remains the canonical callable API. `catalog.json` contains presentation categories for global and namespaced functions. Each catalog member identifies a function by its namespace and name; the empty namespace identifies a global function. Categories such as `math`, `io`, and `testing` are presentation concepts, not callable Ferret namespaces. The generator contains no deployment domain. ## Release publication The `Publish Ferret Core API Artifacts` workflow runs independently from dependent-release notifications. It accepts `release.published` events and manual dispatches, verifies that the selected canonical v2 tag belongs to a published GitHub release, checks out that exact tag, strips the leading `v`, and generates both artifacts. The existing `gh-pages` branch is updated through the repository-local `scripts/publish-core-api.sh` wrapper. Publication validates the existing index and every referenced artifact before mutation, preserves unrelated Pages files, atomically installs one version directory containing `api.json` and `catalog.json`, recomputes the unchanged API index, commits once, and performs a normal non-force push. Version directories, index entries, and hrefs are immutable and cannot be republished even when bytes match. Existing API-only version directories remain valid legacy state and are never backfilled. A stale push fails; published history is never rewritten. All `gh-pages` writers use the repository-wide `gh-pages-writer` concurrency key with cancellation disabled. Unit and integration benchmarks still execute in parallel, but their two results are persisted downstream by one serialized job. Publication starts with the first release that contains this generator. Older v2 alpha tags are not backfilled. A prerelease-only index omits `latest`; once a stable release exists, `latest` identifies the greatest stable SemVer. The public documents are: - `https://ferretlang.org/ferret/index.json` - `https://ferretlang.org/ferret/versions//api.json` - `https://ferretlang.org/ferret/versions//catalog.json` These URLs share the existing Ferret Pages root, so benchmark history and other site files remain alongside the API Reference without being regenerated. ## 2. Official Technical Reference & Guides (MontFerret/MontFerret.github.io) ## File: README.md # Ferret Website Built with Hugo ## Prerequisites - [Go](https://golang.org/doc/install) - [Hugo](https://gohugo.io/getting-started/installing/) - [Node.js](https://nodejs.org/en/download/) - [Mage](https://magefile.org) for building the website. ## Getting Started ### Installing dependencies ```bash go mod tidy && mage install ``` ### Dev server ```bash mage serve ``` The standard development server does not generate the Pagefind search bundle. ### Production build Build the static site and its client-side search index: ```bash mage build ``` The generated site, including Pagefind assets under `public/pagefind/`, is written to `public/`. ### Search preview Build the site and search index, then serve the generated static output with Pagefind's preview server: ```bash mage serveSearch ``` ### Tool documentation When adding or reorganizing first-party tool pages, follow the [tool documentation conventions](docs/development/tool-documentation.md). ### Generating the Standard Library reference The website generates the Standard Library reference from the published Ferret Core API Reference selected by `runtime.v2` in `data/versions.yaml`. The selected release must include its sibling API Catalog, whose ordered categories drive the reference navigation. ```bash mage generateStdlib ``` `mage generate` runs the same target. Normal development and production builds generate the reference automatically; contributors do not download API files by hand. See [Standard Library reference generation](docs/maintainers/standard-library-reference.md) for the publication, validation, and generated-file contracts. Published Ferret tool releases update `data/versions.yaml` through a website-owned pull-request workflow. See [Release version updates](docs/maintainers/release-version-updates.md) for the dispatch contract, manual retry procedure, and GitHub App setup. ### Deployment GitHub Actions builds, tests, and deploys the site to GitHub Pages when changes are pushed to `main`. Pull requests targeting `main` run the same checks without deploying, and maintainers can also start the workflow manually from `main` in the Actions tab. The Pages source must be set to **GitHub Actions** in the repository settings. The custom domain remains `ferretlang.org`; configure it and HTTPS in the Pages settings rather than relying on the generated `CNAME` file alone. --- ## File: content/blog/ferret-module-registry/index.md --- title: "Introducing the Ferret Module Registry" subtitle: "Discover, install, and publish Ferret modules" draft: false author: "Tim Voronov" authorLink: "https://github.com/ziflex" date: "2026-08-10" --- Ferret now has a module Registry. Until now, modules could be published as Go packages, but there was no shared place to discover them, compare versions, check compatibility, or find their documentation. You needed to know that a module existed and where its source lived before you could use it. The [Ferret Registry]({{< ref "/registry" >}}) gives modules a common home. It is the public catalog that people browse and that the Ferret CLI reads when searching for or installing modules. [Barn](https://github.com/MontFerret/barn) is the Git-backed source repository and publication pipeline behind the Registry. Contributors submit registration records to Barn for validation and human review; Barn then generates the public Registry from the accepted records. The Registry is available now as part of the Ferret v2 alpha. The workflow is intentionally practical: find a module, install a compatible release into a Go application, or publish your own module through a Barn pull request. ## One place for modules A Registry entry brings together the information needed to understand and install a module: - its canonical `owner/name` identity - available versions and their Ferret compatibility - the FQL namespace exposed at runtime - the public source repository and exact release commit - the installable Go package path - versioned module documentation These identifiers are related, but they are not interchangeable. For the Archive module, they look like this: | Concept | Value | | --- | --- | | Registry identity | `montferret/archive` | | FQL namespace | `ARCHIVE` | | Go package path | `github.com/MontFerret/contrib/modules/archive` | | Runtime registration | `ferret.WithModules(archive.New())` | The Registry identity is how users search for and install the module. The namespace is what FQL scripts call. The package path is what a Go application compiles. Keeping these identities explicit lets a module use the names that make sense at each boundary without relying on naming conventions to guess the others. ## Find a module The Registry is the easiest way to browse the catalog from the web. Each module page shows its description, namespace, versions, compatibility, source, and published documentation. The CLI provides the same discovery workflow from a terminal. Search by identity or description: {{< terminal command="true" >}} ferret mod search archive {{< /terminal >}} Then inspect a module before adding it to an application: {{< terminal command="true" >}} ferret mod info montferret/archive {{< /terminal >}} `mod info` shows the available and selected versions, runtime namespace, Ferret compatibility, repository, commit, and documentation URL. This makes the Registry useful before installation, not only as a package lookup service. ## Install a module into an application Run `ferret mod install` from the Go application that hosts Ferret: {{< terminal command="true" >}} ferret mod install montferret/archive {{< /terminal >}} Without an explicit version, the CLI selects the newest registered release compatible with the Ferret version already used by the project. It does not silently move the application to a different Ferret release to make a module fit. You can also request a specific release: {{< terminal command="true" >}} ferret mod install montferret/archive@1.0.0-rc.3 {{< /terminal >}} An exact release still has to declare compatibility with the project's Ferret version. The installer resolves the Registry record, adds the published Go package, updates the application’s `ferret.New(...)` composition, and verifies that the affected package builds before applying the staged changes. If the application is missing a safe Ferret dependency or composition helper, interactive use can show and request approval for those prerequisite changes. For project prerequisites, non-interactive setup, and configuration after installation, see [Install a module]({{< ref "/docs/modules/install" >}}). ## Publish a module Publishing starts from a tagged Go module in a public Git repository. The Registry does not accept uploaded package archives. Instead, Ferret resolves the public tag, inspects the files at its exact commit, and prepares the immutable Barn records that identify the release. The example below publishes `ziflex/kvplugin` at version `1.0.0`. ### Complete the release metadata Keep `ferret.yaml`, `go.mod`, and `README.md` together at the module root. A complete standalone manifest looks like this: ```yaml $schema: https://schemas.ferretlang.org/module/v1.json name: ziflex/kvplugin namespace: KV version: 1.0.0 license: Apache-2.0 documentation: https://github.com/ziflex/ferret-kvplugin#readme url: https://github.com/ziflex/ferret-kvplugin compatibility: ferret: ">=2.0.0-alpha.44 <3.0.0" keywords: - cache categories: - data ``` Module Manifest v1 is strict. Unknown fields, duplicate keys, malformed versions, mixed-case Registry identities, invalid URLs, and invalid SPDX license expressions are rejected. The adjacent `go.mod` supplies the installable package path. The `README.md` supplies the versioned documentation Barn will snapshot and publish as `Markdown` and sanitized `HTML`. A module `README.md` typically includes `Overview`, `Installation`, `Quick Start`, and `API Reference` sections. `compatibility.ferret` is optional in the schema, but Registry releases should declare it. The installer needs a usable compatibility range to select a release for a host application. ### Tag and push the release Commit the manifest, module implementation, tests, `go.mod`, and `README.md` before creating the release tag. For a standalone module, the default tag is `v`: {{< terminal command="true" >}} git tag v1.0.0 git push origin v1.0.0 {{< /terminal >}} For a monorepo module, the default tag is `/v`. Pass `--tag` to each publication command when a repository uses a different convention. The complete [publication guide]({{< ref "/docs/modules/publish" >}}) covers both layouts. ### Validate the release After the tag is public, validate the complete release without authenticating or changing anything on GitHub: {{< terminal command="true" >}} ferret mod publish --dry-run {{< /terminal >}} The command checks the local manifest against the current Registry, resolves the public tag to its exact commit, and verifies the README.md, go.mod, module identity, version, and package path. Nothing is submitted, and no module code is checked out or executed. ### Authenticate with GitHub Before submitting, set `GH_TOKEN` or `GITHUB_TOKEN`. If neither is set, Ferret reads the current `github.com` token by running: {{< terminal command="true" >}} gh auth token --hostname github.com {{< /terminal >}} Authenticate the GitHub CLI when needed: {{< terminal command="true" >}} gh auth login --hostname github.com {{< /terminal >}} The credential must be able to write to your personal Barn fork and open a pull request against `MontFerret/barn`. ### Submit the publication Run the default command from the module root: {{< terminal command="true" >}} ferret mod publish {{< /terminal >}} After validation, the CLI uses the GitHub API to create or reuse your personal Barn fork, create a focused publication branch, and open a pull request against `MontFerret/barn`. A module's first release prepares its module and version records; later releases prepare only the new version record. You do not need a local Barn checkout or knowledge of its record layout. To inspect the records without submitting them, print a deterministic, versioned JSON document: {{< terminal command="true" >}} ferret mod publish --print {{< /terminal >}} `--dry-run` and `--print` cannot be combined, and neither mode submits records. Retries are safe: an already-published version exits successfully, while an exact open pull request or publication branch is reused. Ferret refuses to overwrite an immutable Registry record or a divergent branch. If a stale publication branch in your fork contains different changes, delete it before retrying. ## Publication model The Registry's publication process is deliberately Git-backed and review-based. Specs owns the strict portable contracts for module manifests and Registry records. Barn owns repository inspection, cross-document checks, publication history, and generation of the public catalog. The CLI consumes the resulting versioned artifacts and applies compatibility rules when installing a module. Before submitting, authenticate with GitHub using `GH_TOKEN`, `GITHUB_TOKEN`, or the GitHub CLI. See the publication guide for authentication and permission details. ## Available during the v2 alpha The Registry and `ferret mod` workflow are available now with the Ferret v2 alpha CLI. They are useful today, but the wider v2 ecosystem is still moving toward beta, so feedback on module authoring, discovery, compatibility, and publication is especially useful. You can: - [browse the Registry]({{< ref "/registry" >}}) - [install a module]({{< ref "/docs/modules/install" >}}) - [initialize a module project]({{< ref "/docs/modules/develop" >}}) - [build a complete module with the Ferret SDK]({{< ref "/docs/guides/writing-plugins" >}}) - [publish a module through Barn]({{< ref "/docs/modules/publish" >}}) - [review the `ferret mod` command reference]({{< ref "/docs/tools/cli/mod" >}}) If you have a module that would be useful outside your own application, the Registry now gives it a path from a public Git tag to something other Ferret users can find, inspect, and install. --- ## File: content/blog/ferret-v0.8.1/index.md --- title: "Ferret v0.8.1" subtitle: "Hot fixes" draft: false author: "Tim Voronov" authorLink: "https://www.twitter.com/ziflex" date: "2019-07-26" --- With all big features, come small bugs. :) With this new **[v0.8.1](https://github.com/MontFerret/ferret/releases/tag/v0.8.1)** release we are providing hot fixes for some of these small ones. ## Broken CLICK function In 0.7 ``CLICK`` and ``CLICK_ALL`` functions were doing a check before clicking and was returning a boolean value indicating whether a target element was clicked. In 0.8 this logic got broken due to internal refactorings and the check was not performed which led to broken scripts using this returned value. ## Scroll jumps In 0.8, we introduct new interaction behavior with forms. Everytime you want to ``INPUT`` or ``CLICK`` Ferret would scroll into a target element before the operation. Unfortunately, we didn't add any checks *whether* the element is in a viewport, which caused weird page jumps. Not that bad so far! If you find any new issues, please **[report](https://github.com/MontFerret/ferret/issues)**. --- ## File: content/blog/ferret-v0.8.2/index.md --- title: "Ferret v0.8.2" subtitle: "Hot fixes" draft: false author: "Tim Voronov" authorLink: "https://www.twitter.com/ziflex" date: "2019-08-05" --- Another portion of hot fixes has arrived! **[v0.8.2](https://github.com/MontFerret/ferret/releases/tag/v0.8.2)** release brings some critical fixes. ## Scrolling position is not centered In 0.8, we introduced a new behavior for all user interactions. Before any actions like clicking ot typing, we scroll into a target element making it visible in a viewport. But the logic behind it, positioned elements on the top of the viewport, which led to some issues on some websites due to their markup - the elements were still invisible because they were under other elements. ## Unable to set custom logger fields This is a small fix/improvement for internal logger. There was no way to set a specific id for an execution. It makes it difficult to track script execution in embedded scenarios when you have a request id. Now you can add as many fields as you want to the logger, thanks to [Zerologger](https://github.com/rs/zerolog). ## Invalid parent used for INNER_TEXT, INNER_HTML and friends This is another issue that was triggered by internal refactoring. A new mechanism for retrieving inner HTML and text by selectors used document for all element look ups instead of a given parent element. ## Unable to set custom headers There was an error occuring whenever you pass values to ``headers`` object. As always, if you find any new issues, please **[report](https://github.com/MontFerret/ferret/issues)**. --- ## File: content/blog/ferret-v0.8/index.md --- title: "Ferret v0.8" subtitle: "More features, better API" draft: false author: "Tim Voronov" authorLink: "https://www.twitter.com/ziflex" date: "2019-07-23" --- Hooray, **[Ferret v0.8](https://github.com/MontFerret/ferret/releases/tag/v0.8.0)** has been released! It's been a while since the last release, but we worked hard to bring new and better Ferret. This release has many new exciting features, but unfortunately, there are also some breaking changes. You can find the full changelog **[here](https://github.com/MontFerret/ferret/blob/master/CHANGELOG.md#080)**. Let's go! # What's added ## iframe Ferret *finally* supports ``iframe`` elements. When a page gets loaded, Ferret finds all available elements and provieds an access to them via the ``.frames`` property of a page object. Here's an example of how to find a target frame: {{< editor height="250px" readonly="true" >}} LET page = DOCUMENT("https://www.w3schools.com/html/html_iframe.asp", { driver: "cdp" }) LET content = ( FOR f IN page.frames FILTER f.URL == "https://www.w3schools.com/html/default.asp" RETURN f.head.innerHTML ) RETURN FIRST(content) {{}} Alternatively, you can filter them out by url or access to a target iframe by index, if you know it's position. Due to CORS security policies, you still may have issues with iframes if it points to another domain. ## Namespaces With this release, we introduce a new language feature - **namespaces**. Namespaces allow library authors (and us) to isolate functions into dedicated sub sections. Here is an example: {{< code lang="fql" height="120px" >}} LET blob = DOWNLOAD("https://raw.githubusercontent.com/MontFerret/ferret/master/assets/logo.png") RETURN IO::FS::WRITE("logo.png", blob) {{}} To namespace a function, use the new ``namespace`` method. The ``namespace`` method is chainable: {{< code lang="golang" height="380px" >}} package main import ( "context" "encoding/json" "fmt" "os" "strings" "github.com/MontFerret/ferret/pkg/compiler" "github.com/MontFerret/ferret/pkg/runtime/core" "github.com/MontFerret/ferret/pkg/runtime/values" "github.com/MontFerret/ferret/pkg/runtime/values/types" ) func main() { c := compiler.New() c.Namespace("IO").Namespace("FS").RegisterFunction("Read", fs.Read) } {{}} In future releases we will put HTML related functions into HTML:: namespaces. ## XPath A good web scraping tool needs XPath support, and Ferret finally has it! Ferret provides simple interface to XPath engine for both drivers - CDP and HTTP. It automatically detects the output value type and deserializes them accordingly. {{< code lang="fql" height="90px" >}} RETURN XPATH(page, "//div[contains(@class, 'form-group')]") {{}} {{< code lang="fql" height="90px" >}} RETURN XPATH(page, "count(//div)") {{}} These two queries will return 2 different types: 1. Returns an array of serialized elements (their inner HTML) 2. Returns a number indicating how many "div" elements are on the page. ## Regular expression operator This release provides a shorthand for using regexp assertions: {{< code lang="fql" height="90px" >}} LET result = "foo" =~ "^f[o].$" // returns "true" {{}} {{< code lang="fql" height="90px" >}} LET result = "foo" !~ "[a-z]+bar$" // returns "true" {{}} ## New functions to manipulate DOM There are some cases when you might need to change the existing DOM. To help with that, we added the ``INNER_HTML_SET`` and ``INNER_TEXT_SET`` functions. {{< code lang="fql" height="185px" >}} // Using document and selector INNER_HTML_SET(doc, "body", "Hello") INNER_TEXT_SET(doc, "body", "Hello") // Or an element directly INNER_HTML_SET(doc.body, "Hello") INNER_TEXT_SET(doc.body, "Hello") {{}} ## Viewport settings In this release, you can override default values of a viewport in headless mode. {{< code lang="fql" height="185px" >}} LET doc = DOCUMENT(@url, { driver: 'cdp', viewport: { width: 1920, height: 1080 } }) {{}} ## Better emulation of user interaction This is a big change in how Ferret handles page interactions. Now Ferret interacts with pages in a more advanced way - your script can scrolls down or up to an element, moves the mouse, focuses and types... with random delays. Just like a real person! ## Other There are many other many small changes here and there, like adding ``FOCUS``, ``ESCAPE_HTML``, ``UNESCAPE_HTML`` and ``DECODE_URI_COMPONENT`` functions; improving performance; and changing internal design of some parts of the system. # What's broken We try to maintain backwards compatibility, but some of the new features required serious design changes that lead to breaking compatibility with previous versions. As we approach to release v1.0, the API is becoming more stable and will require fewer dramatic changes. Most of the breaking changes will affect only embedded solutions, use of HTML drivers in particular. No changes in the syntax, so no scripts need to change. ## Virtual DOM structure Work on ``iframe`` support required us to redesign the structure of the virtual DOM by introducing top level entity called ``HTMLPage``: {{< code lang="golang" height="440px" >}} type HTMLPage interface { core.Value core.Iterable core.Getter core.Setter collections.Measurable io.Closer IsClosed() values.Boolean GetURL() values.String GetMainFrame() HTMLDocument GetFrames(ctx context.Context) (*values.Array, error) GetFrame(ctx context.Context, idx values.Int) (core.Value, error) GetCookies(ctx context.Context) (*values.Array, error) SetCookies(ctx context.Context, cookies ...HTTPCookie) error DeleteCookies(ctx context.Context, cookies ...HTTPCookie) error PrintToPDF(ctx context.Context, params PDFParams) (values.Binary, error) CaptureScreenshot(ctx context.Context, params ScreenshotParams) (values.Binary, error) WaitForNavigation(ctx context.Context) error Navigate(ctx context.Context, url values.String) error NavigateBack(ctx context.Context, skip values.Int) (values.Boolean, error) NavigateForward(ctx context.Context, skip values.Int) (values.Boolean, error) } {{}} Previously, ``HTMLDocument`` contained the open page, but ``iframe`` nodes introduce the need to have multiple documents representing each node. This led to a new entity in the structure. ## Driver API Because of the changes in Virtual DOM structure, the driver API has been changed as well in order to be reasonable. ``Driver.LoadDocument`` and ``LoadDocumentParams`` are renamed to ``Driver.Open`` and ``Params``. {{< code lang="golang" height="150px" >}} type Driver interface { io.Closer Name() string Open(ctx context.Context, params Params) (HTMLPage, error) } {{}} ## Other In the context of API stabilization and consistency, there are some other minor changes in vDOM elements like extra returned value (usually an error) or ``Get`` prefix in some methods. --- ## File: content/blog/ferret-v0.9/index.md --- title: "Ferret v0.9" subtitle: "Moving forward" draft: false author: "Tim Voronov" authorLink: "https://www.twitter.com/ziflex" date: "2019-09-08" --- Hello fellow miners, **[Ferret v0.9](https://github.com/MontFerret/ferret/releases/tag/v0.9.0)** has been released! In this release, we mostly focused on bug fixes and filling gaps in user interaction functionality. Let's see what we've got. # What's added ## Clearing input values Since early versions, Ferret has ``INPUT`` HTML function that allowed to type any value to input HTML element. This works by appending the function parameter to any pre-existing `` value. While it works fine in many cases, there are some scenarios when a target input field has a preset value which needs to be erased. With this release, Ferret has a new method ``INPUT_CLEAR`` that erases any data from a target input. {{< code lang="fql" height="150px" >}} LET doc = DOCUMENT(@url, { driver: 'cdp' }) RETURN INPUT_CLEAR(doc, '#my-input') {{}} ## Double and more click Single, double, or more. Now it's possible to specify the number of clicks to ``CLICK`` function. {{< code lang="fql" height="150px" >}} LET doc = DOCUMENT(@url, { driver: 'cdp' }) RETURN CLICK(doc, '#like', 2) {{}} ## Unfocus ``BLUR`` function removes focus from an active input field. {{< code lang="fql" height="200px" >}} LET doc = DOCUMENT(@url, { driver: 'cdp' }) FOCUS(doc, '#input') BLUR(doc, '#input') RETURN NONE {{}} ## Default headers and cookies With this release, you can set default headers and/or cookies to HTML drivers. One of the possible use cases is to create drivers with different custom names and pre-defined headers/cookies: {{< code lang="golang" height="730px" >}} cdp.NewDriver( cdp.WithCustomName("cdp_headers"), cdp.WithHeader("Single_header", []string{"single_header_value"}), cdp.WithHeaders(drivers.HTTPHeaders{ "Multi_set_header": []string{"multi_set_header_value"}, "Multi_set_header2": []string{"multi_set_header2_value"}, }), ) cdp.NewDriver( cdp.WithCustomName("cdp_cookies"), cdp.WithCookie(drivers.HTTPCookie{ Name: "single_cookie", Value: "single_cookie_value", Path: "/", MaxAge: 0, Secure: false, HTTPOnly: false, SameSite: 0, }), cdp.WithCookies([]drivers.HTTPCookie{ { Name: "multi_set_cookie", Value: "multi_set_cookie_value", Path: "/", MaxAge: 0, Secure: false, HTTPOnly: false, SameSite: 0, }, { Name: "multi_set_cookie2", Value: "multi_set_cookie2_value", Path: "/", MaxAge: 0, Secure: false, HTTPOnly: false, SameSite: 0, }, }), ) {{}} ## Params in dot notation In previous versions of Ferret, it was possible to pass complex types as parameters, but it was impossible to read data from it using dot notation. Now it is possible. But even more - you can use params as dot notation segments: {{< code lang="fql" height="90px" >}} RETURN @one.@two.@three {{}} ## String literals In addition to defining string literals with quotes (', ") and backtick (\`), you can now define string literals with the forwardtick (´). # What's fixed ## Open tabs on error Ferret didn't close Chrome/Chromium tab if an error occurs during a page load. ## CLICK ``CLICK`` could not be used with both an element and selector. ## Dot notation after a function call Parser could not properly handle scenarios where dot notation was used right after a function call: {{< code lang="fql" height="110px" >}} LET items = [{name: "foo"}] RETURN FIRST(items).name {{}} # What's changed ## Internal cleanup and refactoring We've made some internal cleanups and minor refactoring. There might be minor breaking changes for those who use HTML drivers directly. #### 2nd returned value The following methods have an error as asecond returned value now: - ``HTMLNode.GetChildNodes`` - ``HTMLNode.GetChildNode`` - ``HTMLNode.QuerySelector`` - ``HTMLNode.QuerySelectorAll`` - ``HTMLNode.CountBySelector`` - ``HTMLElement.GetValue`` - ``HTMLElement.GetAttributes`` - ``HTMLElement.GetAttribute`` #### Re-arranged methods The following methods have been moved from ``HTMLDocument`` to ``HTMLElement`` interface: - ``SelectBySelector`` - ``FocusBySelector`` - ``HoverBySelector`` #### Removed methods The following method(s) have been removed from ``HTMLDocument`` interface: - ``MoveMouseBySelector`` --- ## File: content/blog/ferret-v0.10.0/index.md --- title: "Ferret v0.10.0" subtitle: "Comeback" draft: false author: "Tim Voronov" authorLink: "https://www.twitter.com/ziflex" date: "2020-02-21" --- Hello fellow miners, **[Ferret v0.10.0](https://github.com/MontFerret/ferret/releases/tag/v0.10.0)** has been **finally** released! It's been long and busy 6 months since the last release and many of you might thought that the project got abandoned. We assure you it's not! But sometimes life happens and slows you down. But we are back and ready to rock! This release has many great features and improvements thank to all our contributors. Let's take a look at the most important ones. The full changelog you can find [here](https://github.com/MontFerret/ferret/blob/master/CHANGELOG.md#0100). # What's added ## I/O functions You asked - we added! Finally, I/O functions have arrived. In the beginning of the project, we were concerned about security threats that access to filesystem poses, but seeing how demanded this functionality is we agreed to add it. Now, Ferret has new I/O namespaces, with the following functions: - ``IO::FS::READ`` - ``IO::FS::WRITE`` - ``IO::NET::HTTP:GET`` - ``IO::NET::HTTP:POST`` - ``IO::NET::HTTP:PUT`` - ``IO::NET::HTTP:DELETE`` - ``IO::NET::HTTP:REQUEST`` Enjoy! ## Loading HTML string into memory Another popular feature request - possibility to load a prefetched HTML string into Ferret. With this release, you can finally do it with ``HTML_PARSE`` function. {{< code lang="fql" height="180px" >}} LET file = IO::FS::READ(@myfile) LET doc = HTML_PARSE(TO_STRING(file), { driver: 'cdp' // or 'http' }) RETURN INNER_TEXT(doc, 'title') {{}} ## Parameter value availability check In this release, we do pre-runtime check whether all values are provided for parameters used by a script. It was very frustrating when your script was failing in the middle of its execution just because you forgot to provide a value for one of the paramaters. # What's changed ## Case insensitivity Finally, FQL keywords are case insensitive! That's how Google Search query looks like in lower case now: {{< code lang="fql" height="330px" >}} let google = document("https://www.google.com/", { driver: "cdp", }) input(google, 'input[name="q"]', "ferret") click(google, 'input[name="btnK"]') wait_navigation(google) for result in elements(google, '.g') filter trim(result.attributes.class) == 'g' return { title: inner_text(result, 'h3'), url: inner_text(result, 'cite') } {{}} ## Improved CDP driver performance CDP driver performance has been drastically improved that brings CPU usage down from 60% for 3 pages to <1%. # What's fixed ## Redirects It was a long lasting problem that Ferret could not correctly handle redirects that could occur during a page navigation that was leading to timeouts or corrupt page state. Now it has finally been fixed! ```WAIT_NAVIGATION``` function has 2nd optional parameter that can be an object with the following fields: {{< code lang="golang" height="135px" >}} type Parameters struct { TargetURL string Timeout time.Duration } {{}} ``TargetURL`` is a regexp string that can be used to give Ferret a hint what the destination url is: {{< code lang="fql" height="220px" >}} LET doc = DOCUMENT("http://waos.ovh/redirect.html", { driver: 'cdp' }) CLICK(doc, '.click') WAIT_NAVIGATION(doc, { targetURL: "redirect2\.html" }) RETURN ELEMENT(doc, '.title') {{}} --- ## File: content/blog/ferret-v0.11.0/index.md --- title: "Ferret v0.11.0" subtitle: "More features" draft: false author: "Vladimir Fetisov" authorLink: "https://github.com/3timeslazy" date: "2020-05-21" --- Hello fellow miners, Today, we are living in unprecedented times. There are almost 5 million people infected by COVID-19 and thousands die every day in the world. During this time, the highest priority is to stay safe and healthy, take care of family. --- Ferret team has been locked down in their homes with their families as everyone around the globe, but **still** has managed to make new **[Ferret v0.11.0](https://github.com/MontFerret/ferret/releases/tag/v0.11.0)** release! Thanks to all our contributors for their help - we have done a lot of cool features. So, let’s take a look! --- # What's added ## USE statement There is no need to write a full function path anymore! Just add ``USE PATH::TO::PACKAGE`` at the top of your query. {{< code lang="fql" height="190px" >}} USE IO::FS LET favicon = DOWNLOAD('https://www.google.com/favicon.ico') // 'WRITE' the same as 'IO::FS::WRITE'. // RETURN IO::FS::WRITE('google.favicon.ico', favicon) also valid. RETURN WRITE('google.favicon.ico', favicon) {{}} ## PATH functions New functions for working with file paths. {{< code lang="fql" height="90px" >}} LET filename = 'main.go' RETURN PATH::EXT(filename) == '.go' {{}} --- # What's changed ## Scroll options Scroll document's window and elements according to **[MDN](https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollIntoView)** {{< code lang="fql" height="220px" >}} LET doc = DOCUMENT(@url, { driver: 'cdp' }) SCROLL_BOTTOM(doc, { behavior: 'auto', block: 'end', inline: 'end' }) RETURN 1 {{}} --- # What's fixed ## RANDOM_TOKEN ``RANDOM_TOKEN`` return random string now! (even on Windows) ## IO::FS::WRITE Now files are created with read access automatically. --- # Around the Ferret ## OSS-Fuzz platform We have tried to connect Ferret with [OSS-Fuzz](https://github.com/google/oss-fuzz) platform. But Google has [rejected](https://github.com/google/oss-fuzz/pull/3782) Ferret :( --- ## File: content/blog/ferret-v0.11.1/index.md --- title: "Ferret v0.11.1" subtitle: "Hot fix" draft: false author: "Vladimir Fetisov" authorLink: "https://github.com/3timeslazy" date: "2020-05-27" --- Hello fellow miners, **[Ferret v0.11.1](https://github.com/MontFerret/ferret/releases/tag/v0.11.1)** has been released with a hotfix that solves an error in some functions that do screen positioning using scrolling. --- ## File: content/blog/ferret-v0.12.0/index.md --- title: "Ferret v0.12.0" subtitle: "More stability" draft: false author: "Tim Voronov" authorLink: "https://github.com/ziflex" date: "2020-09-14" --- Hello fellow miners, We are happy to announce a new Ferret release - **[Ferret v0.12.0](https://github.com/MontFerret/ferret/releases/tag/v0.12.0)**. This release has a new useful module in the standard library - testing! Let's dig in. The full changelog you can find [here](https://github.com/MontFerret/ferret/blob/master/CHANGELOG.md#0120). --- # What's added ## Assertion library A new ``T`` namespace has been added to the standard library that provides some helpful methods for making assertions. The functions can be used to test UI, validate data and ensure the correctnes of a script. {{< editor lang="fql" height="190px" readonly="true" >}} LET doc = DOCUMENT('https://github.com/', { driver: "cdp" }) T::TRUE(ELEMENT_EXISTS(doc, "#user[login]")) T::TRUE(ELEMENT_EXISTS(doc, "#user[email]")) T::TRUE(ELEMENT_EXISTS(doc, "#user[password]")) RETURN NONE {{}} Read about available functions [here](https://www.ferretlang.org/docs/standard-library/testing). ## New FRAMES function A new helper function that finds HTML frames by a given property selector. {{< editor lang="fql" height="190px" readonly="true" >}} LET page = DOCUMENT('https://mockery.ferretlang.org/scenarios/iframes/', { driver: 'cdp' }) LET frame = FRAMES(page, "url", "https://mockery.ferretlang.org/") RETURN FIRST(frame).url {{}} ## iFrame navigation handling Finally, we can control navigation of nested iframes! Whenever your page has a nested iframe that performs some nested navigation, you can tell Ferret to wait for its completion. {{< editor lang="fql" height="580px" readonly="true" >}} LET page = DOCUMENT('https://mockery.ferretlang.org/scenarios/iframes/', { driver: 'cdp' }) LET innerPage = FIRST(FRAMES(page, "url", "https://www.mockery.ferretlang.org/")) T::NOT::NONE(innerPage) CLICK(innerPage, "#navbar-burger") WAIT_CLASS(innerPage, "#primary-nav", "is-active") CLICK(innerPage, "#blog") WAIT_NAVIGATION(page, { frame: innerPage }) // We need to get a new document that we have navigated to LET innerPage2 = FIRST(FRAMES(page, "url", "https://www.ferretlang.org/blog/")) WAIT_ELEMENT(innerPage2, ".blog") LET item = ELEMENT(innerPage2, ".blog-post a") LET targetURL = item.attributes.href CLICK(item) WAIT_NAVIGATION(page, { frame: innerPage2 }) // We need to get a new document that we have navigated to LET innerPage3 = FIRST(FRAMES(page, "url", targetURL)) RETURN INNER_TEXT(innerPage3, ".content") {{}} # What's changed ## Removed property caching and tracking This **could be** a breaking change, so, please, test your scripts. In previous versions, the CDP driver used aggressive caching and changes tracking on properties like ``.class``, ``.attributes`` and etc., in order to redcue amount of calls to a browser, which led to high CPU usage when a runtime held a big amount of elements in memory. To tackle the problem, the caching and tracking were removed and thus, every time element's property is accessed, a latest value gets retrived from a browser, without any further caching. If something needs to be cahced, it's user's responsibility to chace it in local variable.