# Technical Documentation: wader/fq > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/wader/fq) ### Overview (`README.md`) # fq Tool, language and decoders for working with binary formats. Basic usage is `fq . file`, `fq d file` or `fq 'some query' file ...`. For details see [the manual](https://wader.github.io/fq/), also available as `man fq` and as [AsciiDoc](doc/fq.1.adoc). ### Background fq is a tool, language, and decoders for working with binary formats and data. In most cases it behaves and feels similar to [jq](https://jqlang.github.io/jq/) and it also uses the same expression language. To get the most out of fq it's recommended to learn more about jq. It features a structural hex viewer, nested format decoding, slicing and concatenating binary data, bit-level decoding and an interactive REPL with auto-completion. ### Goals - Make binaries more accessible, queryable and sliceable. - Nested formats and bit-oriented decoding. - Quick and comfortable CLI tool. - Bits and bytes transformations. ### Hopes - Make it useful enough that people want to help improve it. - Inspire people to create similar tools. ### Supported formats [fq -rn -L doc 'include "formats"; formats_list']: sh-start aac_frame, adts, adts_frame, aiff, amf0, apev2, apple_bookmark, ar, asn1_ber, av1_ccr, av1_frame, av1_obu, avc_annexb, avc_au, avc_dcr, avc_nalu, avc_pps, avc_sei, avc_sps, avi, avro_ocf, bencode, bitcoin_blkdat, bitcoin_block, bitcoin_script, bitcoin_transaction, bits, bplist, bsd_loopback_frame, bson, bytes, bzip2, caff, cbor, csv, dns, dns_tcp, elf, ether8023_frame, exif, fairplay_spc, fit, flac, flac_frame, flac_metadatablock, flac_metadatablocks, flac_picture, flac_streaminfo, gif, gzip, heif, hevc_annexb, hevc_au, hevc_dcr, hevc_nalu, hevc_pps, hevc_sps, hevc_vps, html, icc_profile, icmp, icmpv6, id3v1, id3v11, id3v2, ipv4_packet, ipv6_packet, jp2c, jpeg, json, jsonl, leveldb_descriptor, leveldb_log, leveldb_table, luajit, macho, macho_fat, markdown, matroska, midi, moc3, mosaic, mp3, mp3_frame, mp3_frame_vbri, mp3_frame_xing, mp4, mpeg_asc, mpeg_es, mpeg_pes, mpeg_pes_packet, mpeg_spu, mpeg_ts, msgpack, negentropy, nes, ogg, ogg_page, opentimestamps, opus_packet, pcap, pcapng, pg_btree, pg_control, pg_heap, png, prores_frame, protobuf, protobuf_widevine, pssh_playready, rtmp, safetensors, sll2_packet, sll_packet, stl, tap, tar, tcp_segment, tiff, tls, toml, tzif, tzx, udp_datagram, vorbis_comment, vorbis_packet, vp8_frame, vp9_cfm, vp9_frame, vpx_ccr, wasm, wav, webp, xml, yaml, zip [#]: sh-end It can also work with some common text formats like URLs, hex, base64, PEM etc and for some serialization formats like XML, YAML, etc. it can transform both from and to jq values. For details see [the manual](https://wader.github.io/fq/). ## Presentations and media - [PBS Tidbit 8 of Y: Interview with jq Maintainer Mattias Wadman](https://pbs.bartificer.net/tidbit8) - English podcast episode about jq and some fq. - [Kodsnack 585 - Polymorfisk JSON](https://kodsnack.se/585/) - Swedish podcast episode about jq and fq - "fq - jq for binary formats" at [FOSDEM 2023](https://fosdem.org/2023/) - [video & slides](https://fosdem.org/2023/schedule/event/bintools_fq/) - "fq - jq for binary formats" at [No time to wait 6](https://mediaarea.net/NoTimeToWait6) - [video](https://www.youtube.com/watch?v=-Pwt5KL-xRs&t=1450s) - [slides](doc/presentations/nttw6/fq-nttw6-slides.pdf) - "fq - jq for binary formats" at [Binary Tools Summit 2022](https://binary-tools.net/summit.html) - [video](https://www.youtube.com/watch?v=GJOq_b0eb-s&list=PLTj8twuHdQz-JcX7k6eOwyVPDB8CyfZc8&index=1) - [slides](doc/presentations/bts2022/fq-bts2022-v1.pdf) ## Install Use one of the methods listed below or download a pre-built [release](https://github.com/wader/fq/releases) for macOS, Linux or Windows. Unarchive it and move the executable to `PATH` etc. On macOS if you don't install using one of the method below then you might have to manually allow the binary to run. This can be done by trying to run the binary, ignore the warning and then go into security preference and allow it. Same can be done with these commands: ```sh xattr -d com.apple.quarantine fq # for macOS version before Sequoia you might need this also spctl --add fq ``` ### Homebrew (macOS) ```sh brew install wader/tap/fq ``` ### MacPorts On macOS, `fq` can also be installed via [MacPorts](https://www.macports.org). More details [here](https://ports.macports.org/port/fq/). ```sh sudo port install fq ``` ### Windows `fq` can be installed via [scoop](https://scoop.sh/). ```powershell scoop install fq ``` ### Arch Linux `fq` can be installed from the [extra repository](https://archlinux.org/packages/extra/x86_64/fq/) using [pacman](https://wiki.archlinux.org/title/Pacman): ```sh pacman -S fq ``` You can also build and install the development (VCS) package using an [AUR helper](https://wiki.archlinux.org/index.php/AUR_helpers): ```sh paru -S fq-git ``` ### Gentoo `fq` can be installed from the Gentoo package repository using `emerge`: ```sh emerge dev-util/fq ``` ### Guix ```sh guix shell fq -- fq --help # in develompen shell guix install fq # into current profile ``` ### Nix ```sh nix-shell -p fq ``` ### FreeBSD Use the [fq](https://cgit.freebsd.org/ports/tree/misc/fq) port. ### Alpine Currently in edge testing but should work fine in stable also. ``` apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq ``` ### Build from source Make sure you have [go](https://go.dev) 1.22 or later installed. To install directly from git repository (no git clone needed): ```sh # build and install latest release go install github.com/wader/fq@latest # build and install latest master go install github.com/wader/fq@master # copy binary to $PATH if needed cp "$(go env GOPATH)/bin/fq" /usr/local/bin ``` To build, run and test from source: ```sh # build and run go run . # build and run with arguments go run . -d mp3 . file.mp3 # just build go build -o fq . # run all tests and build binary make test fq ``` ## Development and adding a new decoder See [dev.md](doc/dev.md) ## Thanks and related projects This project would not have been possible without [itchyny](https://github.com/itchyny)'s jq implementation [gojq](https://github.com/itchyny/gojq). I also want to thank [HexFiend](https://github.com/HexFiend/HexFiend) for inspiration and ideas and [stedolan](https://github.com/stedolan) for inventing the [jq](https://github.com/stedolan/jq) language. ### Similar or related works #### Tools - [HexFiend](https://github.com/HexFiend/HexFiend) - Hex editor for macOS with format template support. - [ImHex](https://github.com/WerWolv/ImHex) - A Hex Editor for Reverse Engineers. - [binspector](https://github.com/binspector/binspector) - Binary format analysis tool with query language and REPL. - [kaitai](https://kaitai.io) - Declarative binary format parsing. - [Wireshark](https://www.wireshark.org) - Decodes network traffic (tip: `tshark -T json`). - [MediaInfo](https://mediaarea.net/en/MediaInfo) - Analyze media files (tip `mediainfo --Output=JSON` and `mediainfo --Details=1`). - [GNU poke](https://www.jemarch.net/poke) - The extensible editor for structured binary data. - [ffmpeg/ffprobe](https://ffmpeg.org) - Powerful media libraries and tools. - [hexdump](https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/text-utils/hexdump.c) - Hex viewer tool. - [hex](https://git.janouch.name/p/hex) - Interactive hex viewer with format support via lua. - [hachoir](https://github.com/vstinner/hachoir) - General python library for working binary data. - [scapy](https://scapy.net) - Decode/Encode formats, focus on network protocols. #### Projects and Standards - [Let's Solve the File Format Problem](http://fileformats.archiveteam.org). - [PRONOM](https://www.nationalarchives.gov.uk/PRONOM/) file format registry. - [Sustainability of Digital Formats](https://www.loc.gov/preservation/digital/formats/) at Library of Congress. - [Data Format Description Language (DFDL)](https://en.wikipedia.org/wiki/Data_Format_Description_Language). ## TODO and ideas See [TODO.md](doc/TODO.md) ## License `fq` is distributed under the terms of the MIT License. See the [LICENSE](LICENSE) file for license details. Licenses of direct dependencies: - Forked version of gojq - https://github.com/itchyny/gojq/blob/main/LICENSE (MIT) - github.com/ergochat/readline - https://github.com/ergochat/readline/blob/master/LICENSE (MIT) - github.com/BurntSushi/toml - https://github.com/BurntSushi/toml/blob/master/COPYING (MIT) - github.com/creasty/defaults - https://github.com/creasty/defaults/blob/master/LICENSE (MIT) - github.com/gomarkdown/markdown - https://github.com/gomarkdown/markdown/blob/master/LICENSE.txt (BSD) - github.com/gopacket/gopacket - https://github.com/gopacket/gopacket/blob/master/LICENSE (BSD) - github.com/mitchellh/copystructure - https://github.com/mitchellh/copystructure/blob/master/LICENSE (MIT) - github.com/go-viper/mapstructure/v2 - https://github.com/go-viper/mapstructure/blob/main/LICENSE (MIT) - github.com/pmezard/go-difflib - https://github.com/pmezard/go-difflib/blob/master/LICENSE (BSD) - golang/snappy - https://github.com/golang/snappy/blob/master/LICENSE (BSD) - golang/x/* - https://github.com/golang/text/blob/master/LICENSE (BSD) - github.com/goccy/go-yaml - https://github.com/goccy/go-yaml/blob/master/LICENSE (MIT) - Parts of go crypto/tls and github.com/zmap/zcrypto - https://github.com/zmap/zcrypto/blob/master/LICENSE (Apache) --- ### Overview (`format/fit/README.md`) ### Generated files - format/fit/mappers/messages_generated.go - format/fit/mappers/types_generated.go ### How to generate them if needed 1. Download the Fit SDK from: https://developer.garmin.com/fit/download/ 2. Install NodeJS and NPM 3. Go to the `format/fit/testdata/generator` folder. 4. Run `npm install` if it's your first time 5. Run `node index.js t /PathToSDK/Profile.xlsx > ../../mappers/types_generated.go` 6. Run `node index.js m /PathToSDK/Profile.xlsx > ../../mappers/messages_generated.go` 8. Correct formating and spelling of farenheit->fahrenheit and bondary->boundary in generated files to please Go linter --- ### Overview (`format/matroska/README.md`) Generated code is based on ebml_matroska.xml from http://ietf-wg-cellar.github.io/matroska-specification/ It's licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0). See http://ietf-wg-cellar.github.io/matroska-specification/LICENSE.html --- ### Overview (`format/mosaic/README.md`) Generated code is based from https://gitlab.softwareheritage.org/swh/devel/swh-mosaic/-/raw/main/schema.ebml.xml - modified such that enum-restricted values are typed "string" instead of UTF8. It is licensed under the GNU GENERAL PUBLIC LICENSE Version 3, see https://gitlab.softwareheritage.org/swh/devel/swh-mosaic/-/blob/main/LICENSE --- ### Overview (`format/tls/README.md`) ## Dev notes TLS deflate compression seems to actually be zlib, so zlib header + deflate. Also each record is compressed with a flush (trailing 0x00 0x00 0xff 0xff) so that they can be uncompressed individually. https://lekensteyn.nl/files/wireshark-ssl-tls-decryption-secrets-sharkfest18eu.pdf ``` tshark -x -V -o tls.keylog_file:file.keylog -r file.pcap ``` Wireshark gui has TLS debug option to write key/iv etc ``` tcpdump -i en0 -w file.pcap SSLKEYLOGFILE=file.keylog /path/to/sslkey-able/curl --http1.1 -tlsv1.2 --tls-max 1.2 -v https://host/path ``` TLS 1.3 dumps https://gitlab.com/wireshark/wireshark/-/issues/12779 --- ### Overview (`doc/presentations/bts2022/README.md`) fq presentation from Binary Tools Summit 2022 https://binary-tools.net/ [fq-bts2022-v1.pdf](fq-bts2022-v1.pdf) Will update with link to recording when available. Was done at the time of ~fq 0.0.5, things might have changed since. How to build: ``` go install golang.org/x/tools/cmd/present present -notes -content doc/presentations/bts2022 -base ~/go/pkg/mod/golang.org/x/tools@v0.1.9/cmd/present ``` ``` ./usage.sh | ansisvg > usage.svg ``` Export to PDF via browser. --- ### Overview (`format/asn1/testdata/README.md`) tc* files from http://www.strozhevsky.com/free_docs/TEST_SUITE.zip Files were created using: for i in tc*.ber; do echo "\$ fq -d asn1_ber v $i" > $i.fqtest ; done rename 's/transformed_//' transformed_tc* laymans_guide_examples.json extracted from https://luca.ntop.org/Teaching/Appunti/asn1.html From https://lapo.it/asn1js/ released under ISC license: sig-p256-der.p7m sig-p256-ber.p7m sig-rsa1024-sha1.p7s letsencrypt-x3.cer ed25519.cer --- ### Overview (`format/flac/testdata/README.md`) More test files can by found at https://github.com/ietf-wg-cellar/flac-test-files/ at the time of writing the decoder passes tests 01-64. --- ### Overview (`format/midi/testdata/README.md`) # NOTES ## MIDI files The test and example MIDI files are located in the _testdata/midi_ folder. 1. _reference.mid_ Reference MIDI file for testing/development only (it is not a valid MIDI file), with two tracks: - _Track 0_: _empty_ track with only an _end-of-track_ event - _Track 1_: _instrument_ track with sequential examples of all decoded MIDI events 2. _format-0.mid_ Basic MIDI format 0 test file. Contains a single track with only a _track name_ and _end-of-track_ events. 3. _format-1.mid_ Basic MIDI format 1 test file. Contains two tracks, each with only a _track name_ and _end-of-track_ events. 4. _format-2.mid_ Basic MIDI format 2 test file. Contains two tracks, each with only a _track name_ and _end-of-track_ events. 5. _smpte-timecode.mid_ MIDI format 0 test file with an SMPTE timecode for the divisions field. 6. _empty.mid_ Empty MIDI file to verify MIDI decoder handles empty files. 7. _key_signatures.mid_ Test file with all supported MIDI key signatures. 8. _notes.mid_ Test file with all supported MIDI notes. 9. _unknown-chunks.mid_ Test file with 'alien' chunks interleaved with the _MTrk_ track chunks. 10. _invalid-MThd-length.mid_ Test file with invalid _MThd_ chunk length. 11. _invalid-MTrk-length.mid_ Test file with invalid _MTrk_ chunk length. 12. _twinkle.mid_ Sample valid MIDI file for the example queries in the help. ## MIDI event files MIDI files with a single event for development and debugging are located in the _testdata/events_ folder. ### Meta events 1. _sequence-number.mid_ ``` 00 ff 00 02 00 17 delta: 0 sequence number: 23 ``` 2. _text.mid_ ``` 00 ff 01 0d 54 68 69 73 20 61 6e 64 20 54 68 61 74 delta: 0 text: This and That ``` 3. _copyright.mid_ ``` 00 ff 02 04 54 68 65 6d delta: 0 copyright: Them ``` 4. _track_name.mid_ ``` 00 ff 03 0f 41 63 6f 75 73 74 69 63 20 47 75 69 74 61 72 delta: 0 track name: Acoustic Guitar ``` 5. _instrument_name.mid_ ``` 00 ff 04 0a 44 69 64 67 65 72 69 64 6f 6f delta: 0 instrument: Didgeridoo ``` 6. _lyric.mid_ ``` 00 ff 05 08 4c 61 2d 6c 61 2d 6c 61 delta: 0 lyric: La-la-la ``` 7. _marker.mid_ ``` 00 ff 06 0f 48 65 72 65 20 42 65 20 44 72 61 67 6f 6e 73 delta: 0 marker: Here Be Dragons ``` 8. _cuepoint.mid_ ``` 00 ff 07 0c 4d 6f 72 65 20 63 6f 77 62 65 6c 6c delta: 0 cue: More cowbell ``` 9. _program_name.mid_ ``` 00 ff 08 06 45 73 63 61 70 65 delta: 0 program: Escape ``` 10. _device_name.mid_ ``` 00 ff 09 08 54 68 65 54 68 69 6e 67 delta: 00 device: TheThing ``` 11. _midi-channel-prefix.mid_ ``` 00 ff 20 01 0d delta: 00 MIDI channel prefix: 13 ``` 12. _midi-port.mid_ ``` 00 ff 21 01 70 delta: 00 MIDI port: 112 ``` 13. _tempo.mid_ ``` 00 ff 51 03 07 a1 20 delta: 0 tempo: 500000 ``` 14. _smpte-offset.mid_ ``` 00 ff 54 05 4d 2d 3b 07 27 delta: 0 framerate: 25 hour: 13 minute: 45 second: 59 frames: 7 fractions: 39 ``` 15. _time-signature.mid_ ``` 00 ff 58 04 04 02 18 08 delta: 0 numerator: 4 denominator: 4 ticks_per_click: 24 thirty_seconds_per_quarter: 8 ``` 16. _key-signature.mid_ ``` 00 ff 59 02 00 01 delta: 0 key: A minor ``` 17. _end-of-track.mid_ ``` 00 ff 2f 00 delta: 0 ``` 18. _sequencer-specific-event_ ``` 00 ff 7f 06 00 00 3b 3a 4c 5e delta: 0 manufacturer: 00 00 3b (Mark Of The Unicorn (MOTU)) data: 3a 4c 5e ``` ### MIDI events 1. _note-off.mid_ ``` 00 81 70 60 40 71 48 delta: 0 delta: 64 channel: 1 note: 112 (E8) note: 113 (F8) velocity: 96 velocity: 72 ``` 2. _note-on.mid_ ``` 00 92 30 48 40 32 48 delta: 0 delta: 64 channel: 0 note: 48 (C3) note: 50 (D3) velocity: 72 velocity: 72 ``` 3. _polyphonic-pressure.mid_ ``` 00 a0 64 40 48 delta: 0 delta: 64 channel: 0 pressure: 100 pressure: 72 ``` 4. _controller.mid_ ``` 00 b0 20 21 40 20 22 delta: 0 delta: 64 channel: 0 controller: 32 controller: 32 value: 33 value: 34 ``` 5. _program-change.mid_ ``` 00 c0 19 40 20 delta: 0 delta: 64 channel: 0 program: 25 program: 32 ``` 6. _channel-pressure.mid_ ``` 00 d0 07 40 48 delta: 0 delta: 64 channel: 0 pressure: 7 pressure: 72 ``` 7. _pitch-bend.mid_ ``` 00 e5 40 00 20 60 00 40 20 00 delta: 0 delta: 32 delta: 64 channel: 5 bend: 0 bend: 4096 bend: -4096 ``` ### System Exclusive events 1. _sysex-message.mid_ ``` 00 f0 05 7e 00 09 01 f7 delta: 0 manufacturer: 7e (Non-RealTime Extensions) data: 00 09 01 ``` 2. _sysex-continuation.mid_ ``` 00 f0 03 43 01 23 00 f7 06 45 67 89 ab cd ef 00 f7 04 01 23 45 f7 delta: 0 delta: 0 delta: 0 manufacturer: 43 (Yamaha) data: 45 67 89 ab cd ef data: 01 23 45 data: 01 23 ``` 3. _sysex-escape.mid_ ``` 00 f7 02 f3 01 delta: 0 data: f3 01 ``` --- ### Overview (`format/rtmp/testdata/README.md`) Some files based on pcap:s from https://wiki.wireshark.org/SampleCaptures ffmpeg_*_stream files were generated using: ```sh ffmpeg -hide_banner -f lavfi -i sine=frequency=500:sample_rate=44100 -acodec aac -ac 2 -f flv -t 200ms -metadata streamName="test stream" -listen 1 rtmp://0.0.0.0:1935/test_stream ffmpeg -i rtmp://localhost:1935/test_stream -f null - tcpdump -w rtmp.pcap -i lo0 fq '.tcp_connections[1].client_stream | tobytes' rtmp.pcap > ffmpeg_client_stream fq '.tcp_connections[1].server_stream | tobytes' rtmp.pcap > ffmpeg_server_stream ``` --- ### Overview (`format/tap/testdata/README.md`) ### basic_prog1.tap The `basic_prog1.tap` test file was created directory from the FUSE emulator. Inside the emulated ZX Spectrum a BASIC program was created: ``` 10 PRINT "fq is the best!" 20 GOTO 10 ``` and saved to tape: ``` SAVE "fqTestProg", LINE 10 ``` Then from FUSE select the menu item `Media > Tape > Save As..`. Any BASIC, machine code, screen image, or other data, can be saved directly using the `SAVE` command. Further instructions can be found here: https://worldofspectrum.org/ZXBasicManual/zxmanchap20.html --- ### Overview (`format/tls/testdata/README.md`) dump.pcapng dump-broken.pcapng was created by Peter Wu and comes from https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=9144. dump.pcapng contains 73 tls connections with differens cipher suites. split.jq was used to split it into one pcap per connection named after cipher suit used. dump-broken.pcapng is a broken SSL v3, uses extensions. dump-broken.pcapng.keylog not used yet. --- ### Overview (`format/tls/tlsdecrypt/README.md`) This code is an adaptation of golang crypto/tls and https://github.com/zmap/zcrypto Copyright 2009 The Go Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file. https://github.com/zmap/zcrypto/blob/master/LICENSE --- ### Overview (`format/tzx/testdata/README.md`) ### basic_prog1.tzx The `basic_prog1.tzx` test file was created directory from the FUSE emulator. Inside the emulated ZX Spectrum a BASIC program was created: ``` 10 PRINT "fq is the best!" 20 GOTO 10 ``` and saved to tape: ``` SAVE "fqTestProg", LINE 10 ``` Then from FUSE select the menu item `Media > Tape > Save As..`. Any BASIC, machine code, screen image, or other data, can be saved directly using the `SAVE` command. Further instructions can be found here: https://worldofspectrum.org/ZXBasicManual/zxmanchap20.html #### Archive Info The FUSE emulator is not able to add the tape metadata. As this tape block is very simple, it was added manually using a Hex editor. --- ### Doc TODO (`doc/TODO.md`) ### Known bugs to fix - `fq -n '"aabbccdd" | hex | tobytes[1:] | decode("bytes") | tobytes'` create binary `aabbcc` should be `bbccdd`. I think decode (raw in this case) is confused by root value buffer. - Buffers/string duality is confusing, most string functions should be wrapped to understand binary. - REPL cancel seems to sometimes exit a sub-REPl without properly cleanup options. - Value errors, can only be accessed with `._error`. - Framed (add unknown gaps) decode should be on struct level not format? - `tovalue({bits_format: "base64"})` only affect root value. - Auto complete of non-global variables is broken. `scope` is broken for variables. - `echo '{} {} {}' | jq` vs `echo '{} {} {}' | fq` works differently. fq currently decodes one root format and might add unknown gap fields etc. Maybe should work differently for `json` format? - `format/0` overlap with jq builtin `format/1`. What to rename it to? `decode_format`? - repl expression returning a value that produced lots of output can't be interrupted. This is because ctrl-c currently only interrupts the eval interpreter, outputted value is printed (`display`) by parent interpreter. - Rework cli/repl user interrupt (context cancel via ctrl-c), see comment in Interp.Main - Optimize `Interp.Options` calls, now called per display. Cache per eval? needs to handle nested evals. - `[{start: ...: end: ...}]` syntax a bit broken. - REPL completion might have side effcts. Make interp.Function type know and wrap somehow? input, inputs, open, ... - Rework group arguments so that `{is_probe:true}` is not needed. Look up group name and see if it has an argument somehow? ### TODO and ideas #### CLI - Reset color at prompt? context cancel #### CLI and REPL - ctxstack index cancel wrong order, should just skip? - Pager for long output. Configurable? `$PAGER`? only explicit with some kind of syntax? `.. | less` but how? - `dump` cancel output of large root value, ex: `.frames`. Problem is dump is done by parent repl. - Error position "^" pointer? - Configurable history file/name? - Auto complete $variables - Auto complete keys that need escaping, now just filtered out - Auto complete add "." just one and is object - Use JQ_COLORS but extended to allow name= also? #### Language - Cleanup/Make binary buffers make sense. - gojq uses golang `int` for slice indexes, might be issue for non-64bit cpus #### Functions - buffer truncate, left/right pad? - `toimage`? can be done in CLI with "\x1b]1337" but maybe something for a UI? - `toplot`? - `dump` should handle binary, make column code more generic? share with `hexdump`? (bindump also?) - `dump` colorize/notify row range discontinuity - `hexdump` etc should handle binary non byte aligned data - Cleanup rework cipher functions, `ctr(aes("key"), "iv")` or `cipher(ctr("iv"), aes("key))`? - `open` when to close file? - Safe mode interpreter? - Allow/deny `open` in autocomplete - `open` leak, file and ctxreadseeker - Summary tree with format specific summaries for each format, sample count etc etc? - List all unique paths in some compact form? ### Tests - update tests does not preserve comment order for readlines - empty file test - CLI tests, raw write, colors? - Interactive tests #### Documentation - `help("topic")`? - Generate from source - `-n`, `inputs/0` and `input/0` behavior. Same as jq. - Mention `empty.something`? - Use https://github.com/fadado/JBOL/blob/master/doc/JQ-Distilled.md notation - Manual with interactive examples #### Decode - Use interfaces to save memory, "Value V" interface so can have U, Str, etc implementations? - Array of "decorations" sym, display format? - Store original filename somewhere? description for now - Nicer "synthetic" values? now zero length - Cleanup and rethink nested buffers (zip, muxed like ogg) - Endian bitfield helper (elf etc) - Cleanup checksums, should just be fields and add warning if mismatch? - Decoder in jq - Use jq array/object syntax and pass around decode context, collect fields and build tree - Somehow control/limit nested decoding, depth/exclude/include? `probe({depth:1})` etc? per format skip options? - Can't use range while decoding, not calculated yet - Keep track of encoding for values, u16le, utf8, varint etc - Option to ignore range checks, decode until read error instead. Ex: mp4 with truncated mdat. #### Formats - `asn1_ber` `asn1_der`, `asn1_cer` decoder - `flatbuffer` decoder - `capnproto` decoder - Pass argument to format - Value decoder in jq `u(32)`, `u32`? - Warnings and errors - `mp4` sample counts - `flac` truncated picture, mix sample rate, bitdepth etc? - `protobuf` schema? - `matroska` crc - `mp4` styp segment test - Document maturity/completeness - Add `dsf` format - Make `json` format more normal? is a bit a of a special case now - exif in mp4 (heif/heic):
`. as $r | grep("iloc") | parent.items[] | select(.id == (first($r | grep("exif";"i")) | parent.id)).extends[0] as $e | $r | tobytes[$e.offset+10:$e.offset+$e.length] | exif` #### Scripts - Probe tool with common field names - MIME codec encode/decoder "avc1.PPCCLL" etc https://tools.ietf.org/html/rfc6381#section-3.3 - Validate scripts for mp4, matroska #### gojq - Common errors with gojq? re-implemented now - `0b` -> `1.7976931348623157e+308` something fishy with bin/hex/... literals change - Do something similar to `builtin.go` in gojq to speedup a bit - remove `scopedump`? #### Big things - fq play website? - UI, web interface? tree interface, multiple repl windows? nicer way of showing overlapping fiends in hex etc? - jupyter notebook integration - FUSE interface - Lazy decode, should work on known sizes? could also save memory be re-decode? --- ### Doc Demo (`doc/demo.txt`) # Overview of mp3 file $ fq . file.mp3 │00 01 02 03 04 05 06 07 08 09│0123456789│.{}: file.mp3 (mp3) 0x000│49 44 33 04 00 00 00 00 15 39│ID3......9│ headers[0:1]: * │until 0xac2.7 (2755) │ │ 0xabe│ ff fb 40 c0 00│ ..@..│ frames[0:3]: 0xac8│00 00 00 00 00 00 00 00 00 00│..........│ * │until 0xd19.7 (end) (599) │ │ │ │ │ footers[0:0]: # Show header of first ID3v2 tag inside mp3 file $ fq '.headers[0].header' file.mp3 │00 01 02 03 04 05 06 07 08 09│0123456789│.headers[0].header{}: 0x0│49 44 33 │ID3 │ magic: "ID3" (valid) 0x0│ 04 │ . │ version: 4 (valid) 0x0│ 00 │ . │ revision: 0 0x0│ 00 │ . │ flags{}: 0x0│ 00 00 15 39│ ...9│ size: 2745 # Show encoder software used $ fq -Vr '.frames[0].tag.encoder' file.mp3 Lavc58.13 # Decode at two offsets as mp3_frame and show bitrate $ fq -d bytes '.[0xb79,0xc49:] | mp3_frame.header.bitrate' file.mp3 │00 01 02 03 04 05 06 07 08 09│0123456789│ 0xb72│ 50│ P│.header.bitrate: 64000 (5) │00 01 02 03 04 05 06 07 08 09│0123456789│ 0xc44│ 52 │ R │.header.bitrate: 64000 (5) # Extract PNG file $ fq '.headers[].frames[] | select(.id == "APIC")?.picture | tobits' file.mp3 >file.png $ file file.png file.png: PNG image data, 320 x 240, 8-bit/color RGB, non-interlaced # Grep for PNG header, extract resolution and output as YAML $ fq -r 'grep_by(.type == "IHDR") | {res: {width, height}} | to_yaml' file.mp3 res: height: 240 width: 320 # Add query parameter to URL $ echo 'http://host?a=b' | fq -Rr 'from_url | .query.b = "a b c" | to_url' http://host?a=b&b=a+b+c # Extract JSON and base64 encoded query parameter p $ echo 'https://host?p=eyJhIjoiaGVsbG8ifQ%3D%3D' | fq -R 'from_url.query.p | from_base64 | fromjson' { "a": "hello" } --- ### Doc Dev (`doc/dev.md`) ## Implement a decoder ### Steps to add new decoder - Create a directory `format/` - Copy some similar decoder, `format/bson/bson.go` is quite small, to `format//.go` - Cleanup and fill in the register struct, rename `format.BSON` and add it to `format/format.go` and don't forget to change the string constant. - Add an import to `format/all/all.go` ### Some general tips - Main goal is to produce a tree structure that is user-friendly and easy to work with. Prefer a nice and easy tree structure over nice decoder implementation. - Use same names, symbols, constant number bases etc as in specification. But maybe in lowercase to be jq/JSON-ish. - Decode only ranges you know what they are. If possible let "parent" decide what to do with unknown gaps bits by using `*Decode*Len/Range/Limit` functions. fq will also automatically add "gap" fields if it finds gaps. - If you have decode helpers functions that decode a bunch of fields etc it is usually nice to make it only decode fields, not seek or add it's own "containing" struct. That way the function will be easier to reuse and only do one thing. Ex the helper `func decodeHeader(d *decode.D)` can then be use as `d.FieldStruct("header", decodeHeader)`, `d.SeekRel(1234, decodeHeader)` or `d.SeekRel(1234, func(d *decode.D) { d.FieldStruct("header", decodeHeader) }` - Try to not decode too much as one value. A length encoded int could be two fields, but maybe a length prefixed string should be one. Flags can be struct with bit-fields. - Map as many value as possible to symbolic values. - Endian is inherited inside one format decoder, defaults to big endian for new format decoder - Make sure zero length or no frames/packets etc fails decoding - If format is in the probe group make sure to validate input to make it non-ambiguous with other decoders - Try keep decoder code "declarative" if possible - Split into multiple sub formats if possible. Makes it possible to use them separately. - Validate/Assert - Error/Fatal/panic - Can new formats be added to other formats? - Does the new format include existing formats? ### Checklist - Commits: - Use commit messages with a context prefix to make it easier to find and understand, ex:
`mp3: Validate sync correctly` - Tests: - If possible use a pair of `testdata/file` and `testdata/file.fqtest` where `file.fqtest` is `$ fq dv file` or `$ fq 'dv,torepr' file` if there is `torepr` support. - If `dv` produces a lof of output maybe use `dv({array_truncate: 50})` etc - Run `go test ./format -run TestFormats/` to test expected output. - Run `go test ./format -run TestFormats/ -update` to update current output as expected output. - If you have format specific documentation: - Put it in `format/*/.md` and use `//go:embed .md`/`interp.RegisterFS(..)` to embed/register it. - Use simple markdown, just sections (depth starts at 3, `### Section`), paragraphs, lists and links. - No heading section is needs with format name, will be added by `make doc` and fq cli help system. - Add a `testdata/_help.fqtest` with just `$ fq -h ` to test CLI help. - If in doubt look at `mp4.md`/`mp4.go` etc. - Run `make README.md doc/fq.1.adoc doc/fq.1` to update md files. - Run linter `make lint` - Run fuzzer `make fuzz GROUP=`, see usage in Makefile ### Decoder API `*decode.D` reader methods use this name convention: `?(?>|Fn>)(...[, scalar.Mapper...]) ` - If it starts with `Field` a field will be added and first argument will be name of field. If not it will just read. - `??>|?Fn>` a reader or a reader function - `?` If prefixed with `Try` function return error instead of panic on error. - `?>` Read bits using some decoder. - `U16` unsigned 16 bit integer. - `UTF8` UTF8 with byte length as argument. - `Fn>` read using a `func(d *decode.D) ` function. - This can be used to implement own custom readers. All `Field` functions takes a var args of `scalar.Mapper`:s that will be applied after reading. `` are these types: | `` | Go type | jq type | | -------- | ------- | ------- | | U | uint64 | number | | S | int64 | number | | F | float64 | number | | Str | string | string | | Bool | bool | boolean | | Nil | nil | null | TODO: there are some more (BitBuf etc, should be renamed) To add a struct or array use `d.FieldStruct(...)` and `d.FieldArray(...)`. TODO: nested formats, buffers, own decoders, scalar mappers TODO: seeking, framed/limited/range decode For example this decoder: ```go // read 4 byte UTF8 string and add it as "magic", return a string d.FieldUTF8("magic", 4) // create a new struct and add it as "headers", returns a *decode.D d.FieldStruct("headers", func(d *decode.D) { // read 8 bit unsigned integer, map it and add it as "type", returns a uint64 d.FieldU8("type", scalar.UintMapSymStr{ 1: "start", // ... }) }) ``` will produce something like this: ```go *decode.Value{ Parent: nil, V: *decode.Compound{ IsArray: false, // is struct Children: []*decode.Value{ *decode.Value{ Name: "magic", V: scalar.Str{ Actual: "abcd", // read and set by UTF8 reader }, Range: ranges.Range{Start: 0, Len: 32}, }, *decode.Value{ Parent: &... // ref parent *decode.Value>, Name: "headers", V: *decode.Compound{ IsArray: false, // is struct Children: []*decode.Value{ *decode.Value{ Name: "type", V: scalar.Uint{ Actual: uint64(1), // read and set by U8 reader Sym: "start", // set by UintMapSymStr scalar.Mapper }, Range: ranges.Range{Start: 32, Len: 8}, }, }, }, Range: ranges.Range{Start: 32, Len: 8}, }, }, }, Range: ranges.Range{Start: 0, Len: 40}, } ``` and will look like this in jq/JSON: ```json { "magic": "abcd", "headers": { "type": "start" } } ``` #### `*decode.D` type This is the main type used during decoding. It keeps track of: - A current array or struct [`*decode.Value`](#decodevalue-type) where fields will be added. - Current bit reader - Current default endian - Decode options New [`*decode.D`](#decoded-type) are created during decoding when `d.FieldStruct` etc is used. It is also a kitchen sink of all kind functions for reading various standard number and string encodings etc. Decoder authors do not have to create them. #### `*decode.Value` type Is what [`*decode.D`](#decoded-type) produces and it used to represent the decoded structure. Can be array, struct, number, string etc. It is the underlying type used by `interp.DecodeValue` that implements `gojq.JQValue` to expose it as various jq types, which in turn is used to produce JSON. It stores: - Parent [`*decode.Value`](#decodevalue-type) unless it's a root. - A decoded value, a [`scalar.S`](#scalars-type) or [`*decode.Compound`](#decodecompound-type) (struct or array) - Name in parent struct or array. If parent is a struct the name is unique. - Index in parent array. Not used if parent is a struct. - A bit range. Also struct and array have a range that is the min/max range of its children. - A bit reader where the bit range can be read from. Decoder authors will probably not have to create them. #### `scalar.S` type Keeps track of - Actual value. Decoded value represented using a go type like `uint64`, `string` etc. For example a value reader by a utf8 or utf16 reader both will ends up as a `string`. - Symbolic value. Optional symbolic representation of the actual value. For example a `scalar.UintMapSymStr` would map an actual `uint64` to a symbolic `string`. - String description of the value. - Number representation The `scalar` package has `scalar.Mapper` implementations for all types to map actual to whole [`scalar.S`](#scalars-type) value `scalar.ToScalar` or to just to set symbolic value `scalar.ToSym`. There is also mappers to just set values or to change number representations `scalar.Hex`/`scalar.SymHex` etc. Decoder authors will probably not have to create them. But you might implement your own `scalar.Mapper` to modify them. #### `*decode.Compound` type Used to store struct or array of [`*decode.Value`](#decodevalue-type). Decoder authors do not have to create them. ## Development tips I usually use `-d ` and `dv` while developing, that way you will get a decode tree even if it fails. `dv` gives verbose output and also includes stacktrace. ```sh go run . -d dv file ``` If the format is inside some other format it can be handy to first extract the bits and run the decode directly. For example if working a `aac_frame` decoder issue: ```sh fq '.tracks[0].samples[1234] | tobytes' file.mp4 > aac_frame_1234 fq -d aac_frame dv aac_frame_1234 ``` Sometimes nested decoding fails then maybe a good way is to change the parent decoder to use `d.RawLen()` etc instead of `d.FormatLen()` etc temporary to extract the bits. Hopefully there will be some option to do this in the future. When researching or investigating something I can recommend to use `watchexec`, `modd` etc to make things more comfortable. Also using vscode/delve for debugging should work fine once launch `args` are setup etc. ``` watchexec "go run . -d aac_frame dv aac_frame" ``` Some different ways to run tests: ```sh # run all tests make test # run all go tests go test ./... # run all tests for one format go test -run TestFormats/mp4 ./format/ # update all expected outputs for tests go test ./pkg/interp ./format -update # update actual output for specific tests go run ./format -run TestFormats/elf -update # color diff DIFF_COLOR=1 go test ... ``` To lint source use: ``` make lint ``` Generate documentation. Requires [FFmpeg](https://github.com/FFmpeg/FFmpeg) and [Graphviz](https://gitlab.com/graphviz/graphviz): ```sh make doc ``` TODO: `make fuzz` ## Debug Split debug and normal output even when using repl: Write `log` package output and stderr to a file that can be `tail -f`:ed in another terminal: ```sh LOGFILE=/tmp/log go run . ... 2>>/tmp/log ``` gojq execution debug: ```sh GOJQ_DEBUG=1 go run -tags debug . ... ``` Memory and CPU profile (will open a browser): ```sh make memprof ARGS=". file" make cpuprof ARGS=". test.mp3" ``` ## From start to decoded value ``` main:main() cli.Main(default registry) interp.New(registry, std os interp implementation) interp.(*Interp).Main() interp.jq _main/0: args.jq _args_parse/2 populate filenames for input/0 interp.jq inputs/0 foreach valid input/0 output interp.jq open funcs.go _open interp.jq decode funcs.go _decode decode.go Decode(...) ... interp.jq eval expr funcs.go _eval interp.jq display funcs.go _display for interp.(decodeValueBase).Display() dump.go print tree empty output ``` ## bitio and other io packages ``` *os.File, *bytes.Buffer ^ ctxreadseeker.Reader defers blocking io operations to a goroutine to make them cancellable ^ progressreadseeker.Reader approximates how much of a file has been read ^ aheadreadseeker.Reader does readahead caching ^ | (io.ReadSeeker interface) | bitio.IOBitReader (implements bitio.Bit* interfaces) SectionBitReader MultiBitReader ``` ## jq oddities ``` jq -n '[1,2,3,4] | .[null:], .[null:2], .[2:null], .[:null]' ``` ## Setup docker desktop with golang windows container ```sh git clone https://github.com/StefanScherer/windows-docker-machine.git cd windows-docker-machine vagrant up 2016-box cd ../fq docker --context 2016-box run --rm -ti -v "C:${PWD//\//\\}:C:${PWD//\//\\}" -w "$PWD" golang:1.18-windowsservercore-ltsc2016 ``` ## Implementation details - fq uses a gojq fork that can be found at https://github.com/wader/gojq/tree/fq (the "fq" branch) - cli readline uses raw mode to blocks ctrl-c to become a SIGINT ## Dependencies and source origins - [gojq](https://github.com/itchyny/gojq) fork that can be found at https://github.com/wader/gojq/tree/fq
Issues and PR:s related to fq:
[#43](https://github.com/itchyny/gojq/issues/43) Support for functions written in go when used as a library
[#46](https://github.com/itchyny/gojq/pull/46) Support custom internal functions
[#56](https://github.com/itchyny/gojq/issues/56) String format query with no operator using %#v or %#+v panics [#65](https://github.com/itchyny/gojq/issues/65) Try-catch with custom function
[#67](https://github.com/itchyny/gojq/pull/67) Add custom iterator function support which enables implementing a REPL in jq
[#81](https://github.com/itchyny/gojq/issues/81) path/1 behavior and path expression question
[#86](https://github.com/itchyny/gojq/issues/86) ER: basic TCO [#109](https://github.com/itchyny/gojq/issues/109) jq halt_error behavior difference
[#113](https://github.com/itchyny/gojq/issues/113) error/0 and error/1 behavior difference
[#117](https://github.com/itchyny/gojq/issues/117) Negative number modulus *big.Int behaves differently to int
[#118](https://github.com/itchyny/gojq/issues/118) Regression introduced by "remove fork analysis from tail call optimization (ref #86)"
[#122](https://github.com/itchyny/gojq/issues/122) Slow performance for large error values that ends up using typeErrorPreview()
[#125](https://github.com/itchyny/gojq/pull/125) improve performance of join by make it internal
[#141](https://github.com/itchyny/gojq/issues/141) Empty array flatten regression since "improve flatten performance by reducing copy" - [gopacket](https://github.com/gopacket/gopacket) for TCP and IPv4 reassembly - [mapstructure](https://github.com/go-viper/mapstructure) for convenient JSON/map conversion - [go-difflib](https://github.com/pmezard/go-difflib) for diff tests - [golang.org/x/text](https://pkg.go.dev/golang.org/x/text) for text encoding conversions - [float16.go](https://android.googlesource.com/platform/tools/gpu/+/gradle_2.0.0/binary/float16.go) to convert bits into 16-bit floats ## Release process Run and follow instructions: ``` make release VERSION=1.2.3 ``` Commits since release ``` git log --no-decorate --no-merges --oneline v0.0.4..wader/master | sort -t " " -k 2 | sed 's/\(.*\)/* \1/' ``` --- ### Doc Display Decode Value (`doc/display_decode_value.txt`) $ fq 'first(grep_by(.type == "elst"))' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd5c| 00| .| size: 28 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd66| 00 | . | version: 0 0xd66| 00 00| ..| flags: 0 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 0xd70| 00 00 00 28 00| ...(.| entries[0:1]: 0xd7a|00 00 00 00 01 00 00 |....... | --- ### Doc Display Decode Value D (`doc/display_decode_value_d.txt`) $ fq 'first(grep_by(.type == "elst")) | d' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd5c| 00| .| size: 28 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd66| 00 | . | version: 0 0xd66| 00 00| ..| flags: 0 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 | | | entries[0:1]: | | | [0]{}: entry 0xd70| 00 00 00 28 | ...( | segment_duration: 40 0xd70| 00| .| media_time: 0 0xd7a|00 00 00 |... | 0xd7a| 00 01 00 00 | .... | media_rate: 1 --- ### Doc Display Decode Value Dv (`doc/display_decode_value_dv.txt`) $ fq 'first(grep_by(.type == "elst")) | dv' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd65-0xd81 (28) 0xd5c| 00| .| size: 28 0xd65-0xd69 (4) 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd69-0xd6d (4) 0xd66| 00 | . | version: 0 0xd6d-0xd6e (1) 0xd66| 00 00| ..| flags: 0 0xd6e-0xd71 (3) 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 0xd71-0xd75 (4) | | | entries[0:1]: 0xd75-0xd81 (12) | | | [0]{}: entry 0xd75-0xd81 (12) 0xd70| 00 00 00 28 | ...( | segment_duration: 40 0xd75-0xd79 (4) 0xd70| 00| .| media_time: 0 0xd79-0xd7d (4) 0xd7a|00 00 00 |... | 0xd7a| 00 01 00 00 | .... | media_rate: 1 0xd7d-0xd81 (4) --- ### Doc Display Decode Value Dv2 (`doc/display_decode_value_dv2.txt`) $ fq '.frames[1].header | dv' file.mp3 |00 01 02 03 04 05 06 07 08 09|0123456789|.frames[1].header{}: 0xb79-0xb7d (4) 0xb72| ff fb | .. | sync: 0b11111111111 (valid) 0xb79-0xb7a.3 (1.3) 0xb72| fb | . | mpeg_version: "1" (3) (MPEG Version 1) 0xb7a.3-0xb7a.5 (0.2) 0xb72| fb | . | layer: 3 (1) (MPEG Layer 3) 0xb7a.5-0xb7a.7 (0.2) | | | sample_count: 1152 0xb72| fb | . | protection_absent: true (No CRC) 0xb7a.7-0xb7b (0.1) 0xb72| 50| P| bitrate: 64000 (5) 0xb7b-0xb7b.4 (0.4) 0xb72| 50| P| sample_rate: 44100 (0) 0xb7b.4-0xb7b.6 (0.2) 0xb72| 50| P| padding: "not_padded" (0b0) 0xb7b.6-0xb7b.7 (0.1) 0xb72| 50| P| private: 0 0xb7b.7-0xb7c (0.1) 0xb7c|c4 |. | channels: "mono" (0b11) 0xb7c-0xb7c.2 (0.2) 0xb7c|c4 |. | channel_mode: "none" (0b0) 0xb7c.2-0xb7c.4 (0.2) 0xb7c|c4 |. | copyright: 0 0xb7c.4-0xb7c.5 (0.1) 0xb7c|c4 |. | original: 1 0xb7c.5-0xb7c.6 (0.1) 0xb7c|c4 |. | emphasis: "none" (0b0) 0xb7c.6-0xb7d (0.2) --- ### Doc Display Json (`doc/display_json.txt`) $ fq -n '"hello"' "hello" $ fq -n '"hello" | d' "hello" --- ### Doc Fq.1 (`doc/fq.1.adoc`) = fq(1) Mattias Wadman v0.18.0 :doctype: manpage :manmanual: FQ :mansource: FQ :toc: left :toclevels: 3 :sectanchors: :reproducible: :description: fq - jq for binary formats. Tool, language and decoders for working with binary formats == Name fq - tool, language and decoders for working with binary formats == Synopsis `*fq [_OPTIONS_] [--] [_EXPRESSION_] [_FILE_...]*` == Description **fq** is a tool, language, and set of decoders for working with binary formats and data. In most cases it behaves and feels similar to https://jqlang.github.io/jq/[jq] and it also uses the same expression language. To get the most out of fq it's recommended to learn more about jq. It features a structural hex viewer, nested format decoding, slicing and concatenating binary data, bit-level decoding and an interactive REPL with auto-completion. [source,shell] ---- # Evaluate "d" (display) for file.mp4 $ fq d file.mp4 # structural hex output # Evaluate "da" (display all) for file.mp4 $ fq da file.mp4 # structural hex output # JSON for file.mp4 $ fq -V . file.mp4 # JSON output # Evaluate ".boxes[0].type" for all *.mp4 files # -Vr to output value without quotes $ fq -Vr '.boxes[0].type' *.mp4 ftyp # Evaluate "1+2" without reading any input $ fq -n 1+2 3 ---- For more usage examples see the examples section at the end of the documentation. == Options `*--arg NAME VALUE*`:: Set $NAME to string VALUE `*--argdecode NAME PATH*`:: Set $NAME to decode of PATH `*--argjson NAME JSON*`:: Set $NAME to JSON `*--args*`:: Consume remaining arguments as positional strings `*--color-output*`, `*-C*`:: Force color output `*--compact-output*`, `*-c*`:: Use compact output `*--decode*`, `*-d NAME*`:: Decode format or group (probe) `*--from-file*`, `*-f PATH*`:: Read EXPRESSION from file `*--help*`, `*-h [TOPIC]*`:: Show help for TOPIC (ex: -h formats, -h mp4) `*--include-path*`, `*-L PATH*`:: Add PATH to include search paths `*--join-output*`, `*-j*`:: No newline after each output `*--jsonargs*`:: Consume remaining arguments as positional JSON `*--monochrome-output*`, `*-M*`:: Force monochrome output `*--null-input*`, `*-n*`:: Null input (use input and inputs to read) `*--option*`, `*-o NAME=VALUE/@PATH*`:: Set option (ex: -o color=true, see --help options) `*-o addrbase=number*`::: Number base for addresses `*-o array_truncate=number*`::: Array display length to truncate `*-o bits_format=string*`::: Raw bits representation `*-o bits_format=base64*`:::: Base64 string. `*-o bits_format=byte_array*`:::: Array of bytes (zero bit padded if size is not byte aligned). `*-o bits_format=hex*`:::: Hex string. `*-o bits_format=md5*`:::: MD5 hex string (zero bit padded). `*-o bits_format=snippet*`:::: Truncated Base64 string prefixed with bit length. `*-o bits_format=string*`:::: String with raw bytes (zero bit padded if size is not byte aligned). The string is binary safe internally in fq but bytes not representable as UTF-8 will be lost if turned into JSON (default). `*-o bits_format=truncate*`:::: Truncated string. `*-o byte_colors=ranges=string,...*`::: Byte value colorization `*-o color=true|false*`::: Use color `*-o colors=key=value,...*`::: Color scheme `*-o compact=true|false*`::: Use compact JSON `*-o completion_timeout=number*`::: Seconds to wait for completion results `*-o depth=number*`::: Display tree depth limit `*-o display_bytes=number*`::: Display bytes limit `*-o force=true|false*`::: Force decode `*-o join_string=string*`::: String used to join outputs `*-o line_bytes=number*`::: Number of bytes per display line `*-o raw_string=true|false*`::: Raw string output `*-o sizebase=number*`::: Number base for sizes `*-o skip_gaps=true|false*`::: Skip gaps when representing decode value (arrays) as JSON `*-o string_truncate=number*`::: String display length truncate `*-o unicode=true|false*`::: Use unicode `*-o verbose=true|false*`::: Verbose display `*-o width=number*`::: Terminal width `*--raw-file NAME PATH*`:: Set $NAME to string content of file `*--raw-input*`, `*-R*`:: Read raw input strings (don't decode) `*--raw-output*`, `*-r*`:: Raw string output (without quotes) `*--raw-output0*`:: NUL (zero) byte after each output `*--repl*`, `*-i*`:: Interactive REPL `*--slurp*`, `*-s*`:: Slurp all inputs into an array or string (-Rs) `*--unicode-output*`, `*-U*`:: Force unicode output `*--value-output*`, `*-V*`:: Output JSON value (-Vr for raw string) `*--version*`, `*-v*`:: Show version == Configuration === Init files To add your own functions you can use `init.jq` that is read from: macOS:: `$HOME/Library/Application Support/fq/init.jq` or + `$HOME/.config/fq/init.jq` Linux and BSD:: `$HOME/.config/fq/init.jq` Windows:: `%AppData%\fq\init.jq` === Environment `*NO_COLOR*`:: Non-empty string disables color output. `*CLIUNICODE*`:: Enables use of unicode output characters. `*COMPLETION_TIMEOUT*`:: REPL completion timeout in seconds. `*NO_DECODE_PROGRESS*`:: Disables decode progress indicator. == Exit status `*0*` Success + `*2*` Argument error or input IO error + `*3*` Expression compile error + `*4*` Decode error + `*5*` Expression error == Expression === Syntax See *jq*(1) for syntax details. But here are some common beginner gotchas: - Functions that take no arguments are called using `name` instead of `name()`. - Arguments are separated by `;` instead of `,`. Comma is used to concatenate output streams. To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a single argument `1, 2`, a filter that outputs `1` and then `2`, to `f`. - Expressions can return or "output" zero or more values. This is how iteration etc is done, `1, 2` outputs `1` then `2`. - Similar to shell pipelines, implicit input and output are used and piped together using `|`. `.` is used to refer to current input. Ex `1 | . + 2` outputs `3`, `1, 2 | . + 2` outputs `3` and `4`. - In the jq manual and other jq related documentation you might see `name/2`, this means the function `name` takes two arguments (arity). === Additional features fq uses an extended variant of the jq language with a few extra features: * Arbitrary-precision integers and arithmetic. * Supports raw strings using back-ticks. String interpolation and codepoint escapes are not processed. + ---- `hello \("world")\ud83c\udf0d` ---- results in the string `hello \("world")\ud83c\udf0d` or as JSON `"hello \\(\"world\")\\ud83c\\udf0d"`. In contrast + ---- "hello \("world")\ud83c\udf0d" ---- results in the string `hello world🌍` or as JSON `"hello world🌍"`. + * Supports more number bases in integer literals. ** `0xabcd` (Hexadecimal). ** `0o125715` (Octal). ** `0b1010101111001101` (Binary). ** Grouping using underscore `0xab_cd`. * Binary and decode value types, see below. * Try include using an ending question mark `include "file?";` that doesn't fail if file is missing or has errors. * Some values can act as an object with keys even when they are arrays, numbers etc. * There can be keys hidden from `keys` and `[]`. * Some values are readonly and can't be updated or will convert to JSON on update. * Mixing `--args` and `--jsonargs` does not behave the same as in jq. === Additional functions `*band*`, `*bor*`, `*bxor*`, `*bsl*`, `*bsr*`, `*bnot*`:: Bitwise operations as functions. Work the same as jq's math functions. Functions that take one argument use input, `1 | bnot`, and functions with more arguments ignore the input and use formal arguments `bsl(1; 3)`. `*chunk($size)*`:: Split array or string into `$size` length chunks. Last chunk might be shorter. `*count*`, `*count_by(f)*`:: Like `group` but outputs array of `[value, count]` pairs. `*delta*`, `*delta_by(f)*`:: Array with differences between consecutive elements. `delta` is the same as `delta_by(.b - .a)`. `*diff($a; $b)*`:: Produce a diff between `$a` and `$b`. Differences are represented as an object `{a: , b: }`. `*expr_to_path*`:: Converts from a string `".key[1]"` to a path value `["key", 1]`. `*grep_by(f)*`:: Recursively select using a filter and ignore any errors. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`. This is the same as doing `.. | select(f)?`. `*group*`:: Group values, same as `group_by(.)`. `*path_to_expr*`:: Converts a path value `["key", 1]` to a string `".key[1]"`. `*paste*`:: Read string from stdin until ^D. Useful for pasting text. Ex: `paste | from_pem | asn1_ber | repl` read from stdin then decode and start a new sub-REPL with result. `*streaks*`, `*streaks_by(f)*`:: Like `group` but groups streaks based on condition. `*repl*`, `*repl($opts)*`:: Nested REPL. Must be last in a pipeline. `repl` can "slurp" outputs, ex: `1, 2, 3 | repl`, and supports options, ex: `[1,2,3] | repl({compact: true})`. `*slurp("")*`:: Slurp outputs and saves them to `$name`. Must be last in the pipeline. Will be available as a global array `$name`. Ex `1,2,3 | slurp("a")`, `$a[]` same as `spew("a")`. `*spew*`, `*spew("")*`:: Outputs all or a specific slurp. Ex: `spew("a")`. `*println*`, `*print*`:: Print string or compact JSON to stdout with and without new line. `*printerrln*`, `*printerr*`:: Print string or compact JSON to stderr with and without new line. `*hexdump*`, `*hexdump($opts)*`, `*hd*`, `*hd($opts)*`:: Display binary as a hex dump. `*format*`:: Format name for a decode value. Ex: `first(grep_by(format == "jpeg"))`. === Decode value A decode value is the type returned from decoding a format and used to represent values produced by a decoder. It can be seen as representing any standard jq type but with some additional properties attached. Each decode value has these properties: * Bit range in the input. Can be used as a binary using `tobytes`, `tobytesrange`, `tobits` and `tobitsrange`. * If scalar type, an actual value: ** This is the decoded representation of the bits, a number, string, bool etc. ** Can be accessed using `toactual`. * If scalar type, an optional symbolic value: ** Is usually a mapping of the actual to symbolic value, ex: map number to a string value. ** Can be accessed using `tosym`. * An optional description: ** Can be accessed using `todescription`. * `parent` is the parent decode value * `parents` is all the parent decode values * `topath` is the jq path for the decode value * `torepr` converts decode value to its representation if possible The value of a decode value is the symbolic value if available and otherwise the actual value. To explicitly access the value use `tovalue`. In most expressions this is not needed as it will be done automatically. === Decode value functions `*root*`:: Root decode value for decode value. `*buffer_root*`:: Root decode value of sub buffer for decode value. `*format_root*`:: Root decode value of nested format for decode value. `*parent*`:: Parent decode value for decode value. `*parents*`:: Outputs all parent decode values from decode value. `*topath*`:: Path for decode value. Use `path_to_expr` to get a string representation. `*tovalue*`, `*tovalue($opts)*`:: Symbolic, if available, or actual value for decode value. `*toactual*`, `*toactual($opts)*`:: Actual value for decode value. `*tosym*`, `*tosym($opts)*`:: Symbolic value for decode value. `*todescription*`:: Description for decode value. `*torepr*`:: Converts decode value into what it represents. For example converts msgpack decode value into a value representing its JSON representation. `*decode*`, `*decode($name)*`, `*decode($name; $opts)*`:: Decode binary as a format. `$name` is format name, `$opts` is format options. Ex: `... | decode("mp4")`. Note that all decoders are available as functions. Ex: `... | mp4` is the same as `... | decode("mp4").` `tobytes`, `tobytesrange`, `tobits` and `tobitsrange` on a decode value will return the raw source bits as a binary. === Binary Binary type is used to store raw bits or bytes. Raw bits will act as zero bits padded strings in standard jq expressions. Use `tobits` and `tobytes` to create them from decode value, string, number or binary array. `tobytes` will if needed zero pad most significant bits to be byte aligned. There is also `tobitsrange` and `tobytesrange` which do the same thing but will preserve source range when displayed. - `"string" | tobytes` produces a binary with UTF8 bytes. - `1234 | tobits` produces a binary with the unsigned big-endian integer 1234 with enough bits to represent the number. Use `tobytes` to get the same but with enough bytes to represent the number. This is different to how numbers work inside binary arrays where they are limited to 0-255. - `["abc", 123, ...] | tobytes` produces a binary from a binary array. See <> below. - `.[index]` accesses bit or byte at index `index`. Index is in units. - `[0x12, 0x34, 0x56] | tobytes[1]` is `0x34` - `[0x12, 0x34, 0x56] | tobits[3]` is `1` - `.[start:]`, `.[start:end]` or `.[:end]` is normal jq slice syntax and will slice the binary from `start` to `end`. `start` and `end` are in units. - `[0x12, 0x34, 0x56] | tobytes[1:2]` will be a binary with the byte `0x34` - `[0x12, 0x34, 0x56] | tobits[4:12]` will be a binary with the byte `0x23` - `[0x12, 0x34, 0x56] | tobits[4:20]` will be a binary with the bytes `0x23`, `0x45` - `[0x12, 0x34, 0x56] | tobits[4:20] | tobytes[1:]` will be a binary with the byte `0x45` - Both `.[index]` and `.[start:end]` support negative indices to index from end. - `explode` outputs an array with all bytes or bits as integers. === Binary functions `*grep($v)*`, `*grep($v; $flags)*`, `*vgrep($v)*`, `*vgrep($v; $flags)*`, `*bgrep($v)*`, `*bgrep($v; $flags)*`:: Recursively match `$v`. `$v` is a scalar to match, where a string is treated as a regexp. A binary will match exact bytes. `$flags` arguments are regexp flags with additional flag "b" that will treat each byte in the input binary as a code point. This makes it possible to match exact bytes. `*fgrep($v)*`, `*fgrep($v; $flags)*`:: Recursively match field name in a decode value. `*tobits*`:: Transform input to binary with bit as unit and don't preserve source range. `*tobitsrange*`:: Transform input to binary with bit as unit and preserve source range. `*tobytes*`:: Transform input to binary with byte as unit and don't preserve source range. `*tobytesrange*`:: Transform input to binary with byte as unit and preserve source range. `*open*`:: Open file for reading. Returns a binary. Ex: `"file" | open | decode`, `"file" | open | tostring`. The standard jq functions `test`, `match`, `capture`, `scan` and `split` also work on binary values, matching against the raw bytes. === Binary array [[binary_array]] Binary array is a value "shape" and not a new type. It's an array of numbers, strings, binaries or other binary arrays. They can be used as input to `tobits`, `tobytes` or other functions that accept a binary as input. - Number is a byte with value 0-255 - String as UTF8 bytes - Binary as is - Binary array used recursively Binary arrays are similar to and inspired by https://www.erlang.org/doc/man/erlang.html#type-iolist[Erlang iolist]. Some examples: - `[0, 123, 255] | tobytes` will be binary with 3 bytes 0, 123 and 255. - `[0, [123, 255]] | tobytes` same as above. - `[0, 1, 1, 0, 0, 1, 1, 0 | tobits] | tobytes` will be binary with 1 byte, 0x66. - `[(.a | tobytes[-10:]), 255, (.b | tobits[:10])] | tobytes` the concatenation of the last 10 bytes of `.a`, byte of value 255 and the first 10 bits of `.b`. === Differences to jq See https://github.com/itchyny/gojq#difference-to-jq[gojq's differences to jq]. === Naming inconsistencies jq's naming convention is a bit inconsistent. Some standard library functions are named `tojson` while others `from_entries`. fq follows this tradition but tries to use `snake_case` unless there is a good reason. Here are all the non-snake_case functions added by fq. Most of them deal with decode and binary values which are new "primitive" types: - `toactual` - `tobits` - `tobitsrange` - `tobytes` - `tobytesrange` - `todescription` - `topath` - `torepr` - `tosym` - `tovalue` == Display output `display` or `d` is the main function for displaying values and is also the function that will be used if no other output function is explicitly used. If its input is a decode value it will output a dump and tree structure or otherwise it will output as JSON. Below are some usage examples: The first and second examples do the same thing, inputting `"hello"` to `display`. [source,console] ---- $ fq -n '"hello"' "hello" $ fq -n '"hello" | d' "hello" ---- In the next few examples we select out the first "edit list" box in an mp4 file and display it in various ways. By default, display will only show the root level: [source,console] ---- $ fq 'first(grep_by(.type == "elst"))' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd5c| 00| .| size: 28 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd66| 00 | . | version: 0 0xd66| 00 00| ..| flags: 0 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 0xd70| 00 00 00 28 00| ...(.| entries[0:1]: 0xd7a|00 00 00 00 01 00 00 |....... | ---- First row shows a ruler with byte offset into the line and jq path for the value. The columns are: - Start address for the line. For example we see that `size` starts at `0xd5c` (row) + `0x09` (column) = `0xd65`. - Hex representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it. - ASCII representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it. - Tree structure of decoded value, symbolic value and description. Notation: - `{}` value is an object that might have nested values. - `[start:end]` value is an array with index starting at `start` and ending at `end` (exclusive). With `display` or `d` it will recursively show the whole tree: [source,console] ---- $ fq 'first(grep_by(.type == "elst")) | d' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd5c| 00| .| size: 28 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd66| 00 | . | version: 0 0xd66| 00 00| ..| flags: 0 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 | | | entries[0:1]: | | | [0]{}: entry 0xd70| 00 00 00 28 | ...( | segment_duration: 40 0xd70| 00| .| media_time: 0 0xd7a|00 00 00 |... | 0xd7a| 00 01 00 00 | .... | media_rate: 1 ---- Same but verbose `dv`: [source,console] ---- $ fq 'first(grep_by(.type == "elst")) | dv' file.mp4 |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd65-0xd81 (28) 0xd5c| 00| .| size: 28 0xd65-0xd69 (4) 0xd66|00 00 1c |... | 0xd66| 65 6c 73 74 | elst | type: "elst" (An edit list) 0xd69-0xd6d (4) 0xd66| 00 | . | version: 0 0xd6d-0xd6e (1) 0xd66| 00 00| ..| flags: 0 0xd6e-0xd71 (3) 0xd70|00 |. | 0xd70| 00 00 00 01 | .... | entry_count: 1 0xd71-0xd75 (4) | | | entries[0:1]: 0xd75-0xd81 (12) | | | [0]{}: entry 0xd75-0xd81 (12) 0xd70| 00 00 00 28 | ...( | segment_duration: 40 0xd75-0xd79 (4) 0xd70| 00| .| media_time: 0 0xd79-0xd7d (4) 0xd7a|00 00 00 |... | 0xd7a| 00 01 00 00 | .... | media_rate: 1 0xd7d-0xd81 (4) ---- In verbose mode bit ranges and array element names are shown. Bit ranges use `[.]-[.]` as notation where `.` is left out if byte aligned. For example `type` starts at byte `0xd69` bit `0` (`.0` is left out) and ends at `0xd6d` bit `0` (exclusive) and has a size of `4` bytes. This verbosely displays the header of the second frame in an mp3 file which has a bunch of non-byte-aligned fields: [source,console] ---- $ fq '.frames[1].header | dv' file.mp3 |00 01 02 03 04 05 06 07 08 09|0123456789|.frames[1].header{}: 0xb79-0xb7d (4) 0xb72| ff fb | .. | sync: 0b11111111111 (valid) 0xb79-0xb7a.3 (1.3) 0xb72| fb | . | mpeg_version: "1" (3) (MPEG Version 1) 0xb7a.3-0xb7a.5 (0.2) 0xb72| fb | . | layer: 3 (1) (MPEG Layer 3) 0xb7a.5-0xb7a.7 (0.2) | | | sample_count: 1152 0xb72| fb | . | protection_absent: true (No CRC) 0xb7a.7-0xb7b (0.1) 0xb72| 50| P| bitrate: 64000 (5) 0xb7b-0xb7b.4 (0.4) 0xb72| 50| P| sample_rate: 44100 (0) 0xb7b.4-0xb7b.6 (0.2) 0xb72| 50| P| padding: "not_padded" (0b0) 0xb7b.6-0xb7b.7 (0.1) 0xb72| 50| P| private: 0 0xb7b.7-0xb7c (0.1) 0xb7c|c4 |. | channels: "mono" (0b11) 0xb7c-0xb7c.2 (0.2) 0xb7c|c4 |. | channel_mode: "none" (0b0) 0xb7c.2-0xb7c.4 (0.2) 0xb7c|c4 |. | copyright: 0 0xb7c.4-0xb7c.5 (0.1) 0xb7c|c4 |. | original: 1 0xb7c.5-0xb7c.6 (0.1) 0xb7c|c4 |. | emphasis: "none" (0b0) 0xb7c.6-0xb7d (0.2) ---- Here the `sync` pattern starts at `0xb79` (bit `0`) and ends at `0xb7a.3` (exclusive) and has a size of `1` byte and `3` bits, `11` bits in total (`8+3`). There are also some other `display` aliases: - `da` is `display({array_truncate: 0, string_truncate: 0})` don't truncate arrays and strings. - `dd` is `display({array_truncate: 0, string_truncate: 0, display_bytes: 0})` don't truncate arrays and strings, show all raw bytes. - `dv` is `display({array_truncate: 0, string_truncate: 0, verbose: true})` don't truncate arrays and strings and display verbosely. - `ddv` is `display({array_truncate: 0, string_truncate: 0, display_bytes: 0, verbose: true})` don't truncate arrays and strings, show all raw bytes and display verbosely. == Formats By default fq will try to automatically determine input format. In some cases this might fail or is not possible, then a format can be specified using `-d NAME`. It's possible sometimes to force decode and get a partial or broken result using `-o force=true`. [source,shell] ---- # decode as msgpack $ fq -d msgpack d file # force decode as msgpack $ fq -d msgpack -o force=true d file # see msgpack format help $ fq -h msgpack # list supported formats $ fq -h formats ---- === Format options Some formats have their own options that can be set using `-o`. For example the `mp4` format has a `decode_samples` option that controls if individual samples should be decoded. To disable it one can do `fq -o decode_samples=false . file.mp4`. See format list for options. === Format functions In addition to using `-d` all format decoders are also available as normal jq functions. Each format provides multiple functions: ``:: Decode and return a decode value even on error. Ex: `... | mp4` `($options)`:: Same as above with format options. Ex: `... | mp4({decode_samples: false})` `from_`:: Decode or throw on error. Ex: `... | from_mp4` `from_($options)`:: Same as above with format options. Ex: `... | from_mp4({decode_samples: false})` Example usage: [source,shell] ---- # decode jpeg found inside some other format $ fq '.some[].query | jpeg' file # decode jpeg at byte range 100-200 $ fq -d bytes '.[100:200] | jpeg' file ---- === Supported formats `*aac_frame*`:: Advanced Audio Coding frame + -- .Options `*-o object_type=1*`::: Audio object type -- + `*adts*`:: Audio Data Transport Stream `*adts_frame*`:: Audio Data Transport Stream frame `*aiff*`:: Audio Interchange File Format `*amf0*`:: Action Message Format 0 `*apev2*`:: APEv2 metadata tag `*apple_bookmark*`:: Apple BookmarkData + -- Apple's `bookmarkData` format is used to encode information that can be resolved into a `URL` object for a file even if the user moves or renames it. Can also contain security scoping information for App Sandbox support. These `bookmarkData` blobs are often found encoded in data fields of Binary Property Lists. Notable examples include: - `com.apple.finder.plist` - contains an `FXRecentFolders` value, which is an array of ten objects, each of which consists of a `name` and `file-bookmark` field, which is a `bookmarkData` object for each recently accessed folder location. - `com.apple.LSSharedFileList.RecentApplications.sfl2` - `sfl2` files are actually `plist` files of the `NSKeyedArchiver` format. They can be parsed the same as `plist` files, but they have a more complicated tree-like structure than would typically be found, which can make locating and retrieving specific values difficult, even once it has been converted to a JSON representation. For more information about these types of files, see Sarah Edwards' excellent research on the subject (link in references). //- The `grep_by` function can be used to recursively descend through the decoded tree, probing for and selecting any `bookmark` blobs, then converting them to readable JSON with `torepr`: [source,shell] ---- fq 'grep_by(.type=="data" and .value[0:4] == "book") | .value | apple_bookmark | torepr' ---- Authors::: - David McDonald https://github.com/dgmcdona[@dgmcdona] https://twitter.com/river_rat_504[@river_rat_504] //- References::: - https://developer.apple.com/documentation/foundation/url/2143023-bookmarkdata - https://mac-alias.readthedocs.io/en/latest/bookmark_fmt.html - https://www.mac4n6.com/blog/2016/1/1/manual-analysis-of-nskeyedarchiver-formatted-plist-files-a-review-of-the-new-os-x-1011-recent-items - https://michaellynn.github.io/2015/10/24/apples-bookmarkdata-exposed/ //- -- + `*ar*`:: Unix archive `*asn1_ber*`:: ASN1 BER (basic encoding rules, also CER and DER) + -- Supports decoding BER, CER and DER (X.690). - Currently no extra validation is done for CER and DER. - Does not support specifying a schema. - Supports `torepr` but without schema all sequences and sets will be arrays. //- Can be used to decode certificates etc::: [source,shell] ---- $ fq -d bytes 'from_pem | asn1_ber | d' cert.pem ---- Can decode nested values::: [source,shell] ---- $ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber ---- Manual schema::: [source,shell] ---- $ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der ---- References::: - https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf - https://en.wikipedia.org/wiki/X.690 - https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/ - https://lapo.it/asn1js/ //- -- + `*av1_ccr*`:: AV1 Codec Configuration Record `*av1_frame*`:: AV1 frame `*av1_obu*`:: AV1 Open Bitstream Unit `*avc_annexb*`:: H.264/AVC Annex B `*avc_au*`:: H.264/AVC Access Unit + -- .Options `*-o bottom_field_pic_order_in_frame_present_flag=false*`::: No description `*-o cpb_cnt=0*`::: No description `*-o cpb_removal_delay_length=0*`::: No description `*-o delta_pic_order_always_zero_flag=false*`::: No description `*-o dpb_output_delay_length=0*`::: No description `*-o frame_mbs_only_flag=true*`::: No description `*-o initial_cpb_removal_delay_length=0*`::: No description `*-o length_size=0*`::: Length value size `*-o log2max_frame_num=4*`::: No description `*-o log2max_pic_order_cnt_lsb=4*`::: No description `*-o nal_hrd_parameters_present=false*`::: No description `*-o pic_order_cnt_type=0*`::: No description `*-o redundant_pic_cnt_present_flag=false*`::: No description `*-o separate_colour_plane_flag=false*`::: No description `*-o time_offset_length=0*`::: No description `*-o vcl_hrd_parameters_present=false*`::: No description -- + `*avc_dcr*`:: H.264/AVC Decoder Configuration Record `*avc_nalu*`:: H.264/AVC Network Access Layer Unit + -- .Options `*-o bottom_field_pic_order_in_frame_present_flag=false*`::: No description `*-o cpb_cnt=0*`::: No description `*-o cpb_removal_delay_length=0*`::: No description `*-o delta_pic_order_always_zero_flag=false*`::: No description `*-o dpb_output_delay_length=0*`::: No description `*-o frame_mbs_only_flag=true*`::: No description `*-o initial_cpb_removal_delay_length=0*`::: No description `*-o log2max_frame_num=4*`::: No description `*-o log2max_pic_order_cnt_lsb=4*`::: No description `*-o nal_hrd_parameters_present=false*`::: No description `*-o pic_order_cnt_type=0*`::: No description `*-o redundant_pic_cnt_present_flag=false*`::: No description `*-o separate_colour_plane_flag=false*`::: No description `*-o time_offset_length=0*`::: No description `*-o vcl_hrd_parameters_present=false*`::: No description -- + `*avc_pps*`:: H.264/AVC Picture Parameter Set `*avc_sei*`:: H.264/AVC Supplemental Enhancement Information + -- .Options `*-o cpb_cnt=0*`::: No description `*-o cpb_removal_delay_length=0*`::: No description `*-o delta_pic_order_always_zero_flag=false*`::: No description `*-o dpb_output_delay_length=0*`::: No description `*-o frame_mbs_only_flag=true*`::: No description `*-o initial_cpb_removal_delay_length=0*`::: No description `*-o log2max_frame_num=4*`::: No description `*-o log2max_pic_order_cnt_lsb=4*`::: No description `*-o nal_hrd_parameters_present=false*`::: No description `*-o pic_order_cnt_type=0*`::: No description `*-o separate_colour_plane_flag=false*`::: No description `*-o time_offset_length=0*`::: No description `*-o vcl_hrd_parameters_present=false*`::: No description -- + `*avc_sps*`:: H.264/AVC Sequence Parameter Set `*avi*`:: Audio Video Interleaved + -- Samples::: AVI has many redundant ways to index samples so currently `.streams[].samples` will only include samples using the most "modern" method found in the file. That is in order of stream super index, movi ix index then idx1 index. Extract samples for stream 1::: [source,shell] ---- $ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3 ---- Show stream summary::: [source,shell] ---- $ fq -o decode_samples=false '[.chunks[0] | grep_by(.id=="LIST" and .type=="strl") | grep_by(.id=="strh") as {$type} | grep_by(.id=="strf") as {$format_tag, $compression} | {$type,$format_tag,$compression}]' *.avi ---- Speed up decoding by disabling sample and extended chunks decoding::: If you're not interested in sample details or extended chunks you can speed up decoding by using: [source,shell] ---- $ fq -o decode_samples=false -o decode_extended_chunks=false d file.avi ---- References::: - https://learn.microsoft.com/en-us/windows/win32/directshow/avi-riff-file-reference[AVI RIFF File Reference] - http://www.jmcgowan.com/odmlff2.pdf[OpenDML AVI File Format Extensions] //- .Options `*-o decode_extended_chunks=true*`::: Decode extended chunks `*-o decode_samples=true*`::: Decode samples -- + `*avro_ocf*`:: Avro object container file + -- Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification. Capable of handling null, deflate, and snappy codecs for data compression. Limitations: - Schema does not support self-referential types, only built-in types. - Decimal logical types are not supported for decoding, will just be treated as their primitive type //- References::: - https://avro.apache.org/docs/current/spec.html#Object+Container+Files //- Authors::: - Xentripetal xentripetal@fastmail.com https://github.com/xentripetal[@xentripetal] //- -- + `*bencode*`:: BitTorrent bencoding + -- Convert represented value to JSON::: [source,shell] ---- $ fq -d bencode torepr file.torrent ---- References::: - https://wiki.theory.org/BitTorrentSpecification#Bencoding //- -- + `*bitcoin_blkdat*`:: Bitcoin blk.dat `*bitcoin_block*`:: Bitcoin block + -- .Options `*-o has_header=false*`::: Has blkdat header -- + `*bitcoin_script*`:: Bitcoin script `*bitcoin_transaction*`:: Bitcoin transaction `*bits*`:: Raw bits + -- Decode to a slice and indexable binary of bits. Slice and decode bit range::: [source,shell] ---- $ echo 'some {"a":1} json' | fq -d bits '.[40:-48] | fromjson' { "a": 1 } ---- Index bits::: [source,shell] ---- $ echo 'hello' | fq -d bits '.[4]' 1 $ echo 'hello' | fq -c -d bits '[.[range(8)]]' [0,1,1,0,1,0,0,0] ---- -- + `*bplist*`:: Apple Binary Property List + -- Show full decoding::: [source,shell] ---- $ fq d Info.plist ---- Timestamps::: Timestamps in Apple Binary Property Lists are encoded as Cocoa Core Data timestamps, where the raw value is the floating point number of seconds since January 1, 2001. By default, `fq` will render the raw floating point value. In order to get the raw value or the string description, use the `tovalue` or `todescription` functions: [source,shell] ---- $ fq 'torepr.SomeTimeStamp | tovalue' Info.plist 685135328 $ fq 'torepr.SomeTimeStamp | todescription' Info.plist "2022-09-17T19:22:08Z" ---- Get JSON representation::: `bplist` files can be converted to a JSON representation using the `torepr` filter: [source,shell] ---- $ fq torepr com.apple.UIAutomation.plist { "UIAutomationEnabled": true } ---- Decoding NSKeyedArchiver serialized objects::: A common way that Swift and Objective-C libraries on macOS serialize objects is through the NSKeyedArchiver API, which flattens objects into a list of elements and class descriptions that are reconstructed into an object graph using CFUID elements in the property list. `fq` includes a function, `from_ns_keyed_archiver`, which will rebuild this object graph into a friendly representation. If no parameters are supplied, it will assume that there is a CFUID located at `."$top".root` that specifies the root from which decoding should occur. If this is not present, an error will be produced, asking the user to specify a root object in the `.$objects` list from which to decode. The following examples show how this might be used (in this case, within the `fq` REPL): [source,shell] ---- # Assume $top.root is present bplist> from_ns_keyed_archiver # Specify optional root bplist> from_ns_keyed_archiver(1) ---- Authors::: - David McDonald https://github.com/dgmcdona[@dgmcdona] //- References::: - http://fileformats.archiveteam.org/wiki/Property_List/Binary - https://medium.com/@karaiskc/understanding-apples-binary-property-list-format-281e6da00dbd - https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c //- -- + `*bsd_loopback_frame*`:: BSD loopback frame `*bson*`:: Binary JSON + -- Limitations::: - The decimal128 type is not supported for decoding, will just be treated as binary //- Convert represented value to JSON::: [source,shell] ---- $ fq -d bson torepr file.bson ---- Filter represented value::: [source,shell] ---- $ fq -d bson 'torepr | select(.name=="bob")' file.bson ---- Authors::: - Mattias Wadman mattias.wadman@gmail.com, original author - Matt Dale https://github.com/matthewdale[@matthewdale], additional types and bug fixes //- References::: - https://bsonspec.org/spec.html //- -- + `*bytes*`:: Raw bytes + -- Decode to a slice and indexable binary of bytes. Slice out byte ranges::: [source,shell] ---- $ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes $ echo -n 'hello' | fq -d bytes '[.[-2:], .[0:2]] | tobytes' > first_last_2_bytes_swapped ---- Slice and decode byte range::: [source,shell] ---- $ echo 'some {"a":1} json' | fq -d bytes '.[5:-6] | fromjson' { "a": 1 } ---- Index bytes::: [source,shell] ---- $ echo 'hello' | fq -d bytes '.[1]' 101 ---- -- + `*bzip2*`:: bzip2 compression `*caff*`:: Live2D Cubism archive + -- Authors::: - https://github.com/ronsor[@ronsor] //- .Options `*-o uncompress=true*`::: Uncompress and probe files -- + `*cbor*`:: Concise Binary Object Representation + -- Convert represented value to JSON::: [source,shell] ---- $ fq -d cbor torepr file.cbor ---- References::: - https://en.wikipedia.org/wiki/CBOR - https://www.rfc-editor.org/rfc/rfc8949.html //- -- + `*csv*`:: Comma separated values + -- TSV to CSV::: [source,shell] ---- $ fq -d csv -o comma="\t" to_csv file.tsv ---- Convert rows to objects based on header row::: [source,shell] ---- $ fq -d csv '.[0] as $t | .[1:] | map(with_entries(.key = $t[.key]))' file.csv ---- .Options `*-o comma=","*`::: Separator character `*-o comment="#"*`::: Comment line character -- + `*dns*`:: DNS packet `*dns_tcp*`:: DNS packet (TCP) `*elf*`:: Executable and Linkable Format `*ether8023_frame*`:: Ethernet 802.3 frame `*exif*`:: Exchangeable Image File Format `*fairplay_spc*`:: FairPlay Server Playback Context `*fit*`:: Garmin Flexible and Interoperable Data Transfer + -- Limitations::: - Fields with subcomponents, such as "compressed_speed_distance" field on globalMessageNumber 20 is not represented correctly. The field is read as 3 separate bytes where the first 12 bits are speed and the last 12 bits are distance. - There are still lots of UNKNOWN fields due to gaps in Garmin's SDK Profile documentation. (Currently FIT SDK 21.126) - Compressed timestamp messages are not accumulated against last known full timestamp. //- Convert stream of data messages to JSON array::: [source,shell] ---- $ fq '[.data_records[] | select(.record_header.message_type == "data").data_message]' file.fit ---- Authors::: - Mikael Lofjärd mikael.lofjard@gmail.com, original author //- References::: - https://developer.garmin.com/fit/protocol/ - https://developer.garmin.com/fit/cookbook/decoding-activity-files/ //- -- + `*flac*`:: Free Lossless Audio Codec file `*flac_frame*`:: FLAC frame + -- .Options `*-o bits_per_sample=16*`::: Bits per sample `*-o sample_details=false*`::: Decode more sample details like residuals etc -- + `*flac_metadatablock*`:: FLAC metadatablock `*flac_metadatablocks*`:: FLAC metadatablocks `*flac_picture*`:: FLAC metadatablock picture `*flac_streaminfo*`:: FLAC streaminfo `*gif*`:: Graphics Interchange Format `*gzip*`:: gzip compression `*heif*`:: High Efficiency Image Format + -- .Options `*-o allow_truncated=false*`::: Allow box to be truncated -- + `*hevc_annexb*`:: H.265/HEVC Annex B `*hevc_au*`:: H.265/HEVC Access Unit + -- .Options `*-o length_size=4*`::: Length value size -- + `*hevc_dcr*`:: H.265/HEVC Decoder Configuration Record `*hevc_nalu*`:: H.265/HEVC Network Access Layer Unit `*hevc_pps*`:: H.265/HEVC Picture Parameter Set `*hevc_sps*`:: H.265/HEVC Sequence Parameter Set `*hevc_vps*`:: H.265/HEVC Video Parameter Set `*html*`:: HyperText Markup Language + -- HTML is decoded in HTML5 mode and will always include ``, `` and `` elements. See xml format for more examples and how to preserve element order and how to encode to xml. There is no `to_html` function, see `to_xml` instead. Element as object::: [source,shell] ---- # decode as object is the default $ echo 'text' | fq -d html { "html": { "body": { "a": { "#text": "text", "@href": "url" } }, "head": "" } } ---- Element as array::: [source,shell] ---- $ echo 'text' | fq -d html -o array=true [ "html", null, [ [ "head", null, [] ], [ "body", null, [ [ "a", { "#text": "text", "href": "url" }, [] ] ] ] ] ] # decode html files to a {file: "title", ...} object $ fq -n -d html '[inputs | {key: input_filename, value: .html.head.title?}] | from_entries' *.html # href:s in file $ fq -r -o array=true -d html '.. | select(.[0] == "a" and .[1].href)?.[1].href' file.html ---- .Options `*-o array=false*`::: Decode as nested arrays `*-o attribute_prefix="@"*`::: Prefix for attribute keys `*-o seq=false*`::: Use seq attribute to preserve element order -- + `*icc_profile*`:: International Color Consortium profile `*icmp*`:: Internet Control Message Protocol `*icmpv6*`:: Internet Control Message Protocol v6 `*id3v1*`:: ID3v1 metadata `*id3v11*`:: ID3v1.1 metadata `*id3v2*`:: ID3v2 metadata `*ipv4_packet*`:: Internet protocol v4 packet `*ipv6_packet*`:: Internet protocol v6 packet `*jp2c*`:: JPEG 2000 codestream `*jpeg*`:: Joint Photographic Experts Group file `*json*`:: JavaScript Object Notation `*jsonl*`:: JavaScript Object Notation Lines `*leveldb_descriptor*`:: LevelDB Descriptor + -- Limitations::: - fragmented non-"full" records are not merged and decoded further. //- Authors::: - https://github.com/mikez[@mikez], original author //- References::: - https://github.com/google/leveldb/blob/main/doc/impl.md#manifest - https://github.com/google/leveldb/blob/main/doc/log_format.md - https://github.com/google/leveldb/blob/main/db/version_edit.cc //- -- + `*leveldb_log*`:: LevelDB Log + -- Limitations::: - fragmented non-"full" records are not merged and decoded further. //- Authors::: - https://github.com/mikez[@mikez], original author //- References::: - https://github.com/google/leveldb/blob/main/doc/impl.md#log-files - https://github.com/google/leveldb/blob/main/doc/log_format.md - https://github.com/google/leveldb/blob/main/db/write_batch.cc //- -- + `*leveldb_table*`:: LevelDB Table + -- Limitations::: - no Meta Blocks (like "filter") are decoded yet. - Zstandard decompression is not implemented yet. //- Authors::: - https://github.com/mikez[@mikez], original author //- References::: - https://github.com/google/leveldb/blob/main/doc/table_format.md - https://github.com/google/leveldb/blob/main/doc/impl.md - https://github.com/google/leveldb/blob/main/doc/index.md //- -- + `*luajit*`:: LuaJIT 2.0 bytecode + -- Authors::: - https://github.com/dlatchx[@dlatchx] //- References::: - https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_bcdump.h - http://scm.zoomquiet.top/data/20131216145900/index.html //- -- + `*macho*`:: Mach-O macOS executable + -- Supports decoding vanilla and FAT Mach-O binaries. Select 64bit load segments::: [source,shell] ---- $ fq '.load_commands[] | select(.cmd=="segment_64")' file ---- References::: - https://github.com/aidansteele/osx-abi-macho-file-format-reference //- Authors::: - Sıddık AÇIL acils@itu.edu.tr https://github.com/Akaame[@Akaame] //- -- + `*macho_fat*`:: Fat Mach-O macOS executable (multi-architecture) `*markdown*`:: Markdown + -- Array with all level 1 and 2 headers::: [source,shell] ---- $ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' file.md ---- -- + `*matroska*`:: Matroska file + -- Lookup element using path::: [source,shell] ---- $ fq 'matroska_path(".Segment.Tracks[0]")' file.mkv ---- Get path to element::: [source,shell] ---- $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv ---- References::: - https://www.rfc-editor.org/info/rfc8794 - https://matroska.org/technical/specs/index.html - https://www.matroska.org/technical/basics.html - https://www.matroska.org/technical/codec_specs.html - https://wiki.xiph.org/MatroskaOpus //- .Options `*-o decode_samples=true*`::: Decode samples -- + `*midi*`:: Standard MIDI file + -- Notes::: * Only supports the MIDI 1.0 MIDI file specification. * Only supports _MThd_ and _MTrk_ chunks. * Does only basic validation on the MIDI data. //- Sample queries::: * Extract the track names from a MIDI file //- [source,shell] ---- fq -d midi '.. | select(.event=="track_name")? | "\(.track_name)"' midi/twinkle.mid ---- * Extract the tempo changes from a MIDI file //- [source,shell] ---- fq -d midi '.. | select(.event=="tempo")?.tempo' midi/twinkle.mid ---- * Extract the key changes from a MIDI file //- [source,shell] ---- fq -d midi '.. | select(.event=="key_signature")?.key_signature' midi/twinkle.mid ---- * Extract NoteOn events: //- [source,shell] ---- fq -d midi 'grep_by(.event=="note_on") | [.time.tick, .note_on.note] | join(" ")' midi/twinkle.mid ---- Authors::: - https://github.com/transcriptaze[transcriptaze] //- References::: * https://www.midi.org/specifications/item/the-midi-1-0-specification[The Complete MIDI 1.0 Detailed Specification] * https://midi.org/standard-midi-files[Standard MIDI Files] * http://midi.teragonaudio.com/tech/midifile.htm[Standard MIDI File (SMF) Format] * http://www.somascape.org/midi/tech/mfile.html[MIDI Files Specification] * https://www.recordingblogs.com/wiki/midi-smpte-offset-meta-message[MIDI SMPTE Offset meta message] * http://www.somascape.org/midi/tech/mfile.html#meta[Somascape MIDI Files Specification] //- -- + `*moc3*`:: MOC3 file + -- Authors::: - https://github.com/ronsor[@ronsor] //- -- + `*mosaic*`:: MOdular Storage of Archived and Indexed Contents + -- List indexes in a mosaic::: [source,shell] ---- $ fq '.mosaic.index[].idx_description' file.mosaic ---- List only small objects::: [source,shell] ---- $ fq '.mosaic.tile[].object[] | select(.size < 1000)' file.mosaic ---- Check if the reported size is correct::: [source,shell] ---- $ fq 'add(.mosaic.tile[].object[].size) == .mosaic.container_meta_data.objects_total_size.value' basic.mosaic ---- Command above only works if the file does not use compression: See if a file uses compression::: [source,shell] ---- $ fq '.mosaic.container_meta_data.compression_method' basic.mosaic ---- Authors::: - https://github.com/martinkirch/[@martinkirch] //- References::: - https://docs.softwareheritage.org/devel/swh-mosaic/ - https://gitlab.softwareheritage.org/swh/devel/swh-mosaic //- -- + `*mp3*`:: MP3 file + -- .Options `*-o max_sync_seek=32768*`::: Max byte distance to next sync `*-o max_unique_header_configs=5*`::: Max number of unique frame header configs allowed `*-o max_unknown=50*`::: Max percent (0-100) unknown bits -- + `*mp3_frame*`:: MPEG audio layer 3 frame `*mp3_frame_vbri*`:: MP3 frame Fraunhofer encoder variable bitrate tag `*mp3_frame_xing*`:: MP3 frame Xing/Info tag `*mp4*`:: ISOBMFF, QuickTime and similar + -- Speed up decoding by not decoding samples::: [source,shell] ---- # manually decode first sample as an aac_frame $ fq -o decode_samples=false '.tracks[0].samples[0] | aac_frame | d' file.mp4 ---- Entries for first edit list as values::: [source,shell] ---- $ fq 'first(grep_by(.type=="elst").entries) | tovalue' file.mp4 ---- Whole box tree as JSON (exclude mdat data and tracks)::: [source,shell] ---- $ fq 'del(.tracks) | grep_by(.type=="mdat").data = "" | tovalue' file.mp4 ---- Force decode a single box::: [source,shell] ---- $ fq -n '"AAAAHGVsc3QAAAAAAAAAAQAAADIAAAQAAAEAAA==" | from_base64 | mp4({force:true}) | d' ---- Lookup mp4 box using an mp4 box path::: [source,shell] ---- # | mp4_path($path) -> $ fq 'mp4_path(".moov.trak[1]")' file.mp4 ---- Get mp4 box path for a decode value box::: [source,shell] ---- # | mp4_path -> string $ fq 'grep_by(.type == "trak") | mp4_path' file.mp4 ---- References::: - https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format[ISO/IEC base media file format (MPEG-4 Part 12)] - https://developer.apple.com/standards/qtff-2001.pdf[Quicktime file format] //- .Options `*-o allow_truncated=false*`::: Allow box to be truncated `*-o decode_samples=true*`::: Decode track samples `*-o skip_samples=false*`::: Skip track samples -- + `*mpeg_asc*`:: MPEG-4 Audio Specific Config `*mpeg_es*`:: MPEG Elementary Stream `*mpeg_pes*`:: MPEG Packetized elementary stream `*mpeg_pes_packet*`:: MPEG Packetized elementary stream packet `*mpeg_spu*`:: Sub Picture Unit (DVD subtitle) `*mpeg_ts*`:: MPEG Transport Stream `*msgpack*`:: MessagePack + -- Convert represented value to JSON::: [source,shell] ---- $ fq -d msgpack torepr file.msgpack ---- References::: - https://github.com/msgpack/msgpack/blob/master/spec.md //- -- + `*negentropy*`:: Negentropy message + -- View a full Negentropy message::: [source,shell] ---- $ fq -d negentropy dd file ---- Or from hex::: [source,shell] ---- $ echo '6186b7abb47c0001108e4206828ee3bf34258465809a337c6c00019a68e37b177a50b3ae7164ccc628b962020114019c1381281c9e3849d5fbd514b7bb65ad0101e601fbf7451f5d22e7fa36ae3e910e9f5215020157014a1b26853e06e9c32eb41b1df4f9ab300201e6011840e273c84bb1344f1d4e15d9aa67920200016f12ee2340888653f10b0ec2d438ac9f0101840156d2d796f4dff004ab369b9bcfa4d81e020187013f1b3c8a019800d5764e2de6bdfd2785020114017caaf0acb5dfe249aa0f7f742402168a01018301e7b8c4decb1eae455ca5714281e3245302017a01409c22636b097362df125ddffb6d944302015b01f332208bee82acf8ed922853ee54057f020001fc3e51fdb0b92966e38017f7959903850101cc01428ce0c96d49f15b50143e4fb228cb9300000131712d30e5296a7a45d07bba452d61cd' | fq -R 'from_hex | negentropy | dd' ---- Check how many ranges the message has and how many of those are of 'fingerprint' mode::: [source,shell] ---- $ fq -d negentropy '.bounds | length as $total | map(select(.mode == "fingerprint")) | length | {$total, fingerprint: .}' message ---- Get all ids in all idlists::: [source,shell] ---- $ fq -d negentropy '.bounds | map(select(.mode == "idlist") | .idlist | .ids) | flatten' message ---- Authors::: - fiatjaf, https://fiatjaf.com //- References::: - https://github.com/hoytech/negentropy //- -- + `*nes*`:: iNES/NES 2.0 cartridge ROM format + -- Limitations::: - `prg_rom`, `chr_rom` and `trainer` fields may contain data that is just random junk from the memory chips, since they are of a fixed size. - The `nes_toasm` function outputs ALL opcodes, including the unofficial ones, which means that none of the regular assemblers can recompile it. - The `nes_tokitty` function works on tiles in `chr_rom` but only outputs a Kitty graphics compatible string. You need to manually `printf` that string to get Kitty (or another compatible terminal) to output the graphics. //- Decompile PRG ROM::: [source,shell] ---- $ fq -r '.prg_rom[] | nes_toasm' file.nes ---- Print out first CHR ROM tile in Kitty (or Konsole, wayst, WezTerm) at size 5::: [source,shell] ---- $ printf $(fq -r -d nes '.chr_rom[0] | nes_tokitty(5)' file.nes) ---- Print out all CHR ROM tiles in Kitty (with Bash) at size 5::: [source,shell] ---- $ for line in $(fq -r '.chr_rom[] | nes_tokitty(5)' file.nes);do printf "%b%s" "$line";done ---- Authors::: - Mikael Lofjärd mikael.lofjard@gmail.com, original author //- References::: - https://www.nesdev.org/wiki/INES - https://www.nesdev.org/wiki/NES_2.0 - https://www.nesdev.org/wiki/CPU - https://bugzmanov.github.io/nes_ebook/chapter_6_3.html //- -- + `*ogg*`:: OGG file `*ogg_page*`:: OGG page `*opentimestamps*`:: OpenTimestamps file + -- View a full OpenTimestamps file::: [source,shell] ---- $ fq dd file.ots ---- List the names of the Calendar servers used::: [source,shell] ---- $ fq '.operations | map(select(.attestation_type == "calendar") | .url)' file.ots ---- Check if there are Bitcoin attestations present::: [source,shell] ---- $ fq '.operations | map(select(.attestation_type == "bitcoin")) | length > 0' file.ots ---- Authors::: - fiatjaf, https://fiatjaf.com //- References::: - https://opentimestamps.org/ - https://github.com/opentimestamps/python-opentimestamps //- -- + `*opus_packet*`:: Opus packet `*pcap*`:: PCAP packet capture + -- Build object with number of (reassembled) TCP bytes sent to/from client IP::: [source,shell] ---- # for a pcapng file you would use .[0].tcp_connections for first section $ fq '.tcp_connections | group_by(.client.ip) | map({key: .[0].client.ip, value: map(.client.stream, .server.stream | tobytes.size) | add}) | from_entries' { "10.1.0.22": 15116, "10.99.12.136": 234, "10.99.12.150": 218 } ---- -- + `*pcapng*`:: PCAPNG packet capture `*pg_btree*`:: PostgreSQL btree index file + -- Btree index meta page::: [source,shell] ---- $ fq -d pg_btree -o flavour=postgres14 ".[0] | d" 16404 ---- Btree index page::: [source,shell] ---- $ fq -d pg_btree -o flavour=postgres14 ".[1]" 16404 ---- Authors::: - Pavel Safonov p.n.safonov@gmail.com https://github.com/pnsafonov[@pnsafonov] //- References::: - https://www.postgresql.org/docs/current/storage-page-layout.html //- .Options `*-o page=0*`::: First page number in file, default is 0 -- + `*pg_control*`:: PostgreSQL control file + -- Decode content of pg_control file::: [source,shell] ---- $ fq -d pg_control -o flavour=postgres14 d pg_control ---- Specific fields can be queried individually::: [source,shell] ---- $ fq -d pg_control -o flavour=postgres14 ".state, .check_point_copy.redo, .wal_level" pg_control ---- Authors::: - Pavel Safonov p.n.safonov@gmail.com https://github.com/pnsafonov[@pnsafonov] //- References::: - https://github.com/postgres/postgres/blob/REL_14_2/src/include/catalog/pg_control.h //- .Options `*-o flavour=""*`::: PostgreSQL flavour: postgres14, pgproee14, postgres10 -- + `*pg_heap*`:: PostgreSQL heap file + -- To see heap page's content::: [source,shell] ---- $ fq -d pg_heap -o flavour=postgres14 ".[0]" 16994 ---- To see page's header::: [source,shell] ---- $ fq -d pg_heap -o flavour=postgres14 ".[0].page_header" 16994 ---- First and last item pointers on first page::: [source,shell] ---- $ fq -d pg_heap -o flavour=postgres14 ".[0].pd_linp[0, -1]" 16994 ---- First and last tuple on first page::: [source,shell] ---- $ fq -d pg_heap -o flavour=postgres14 ".[0].tuples[0, -1]" 16994 ---- Authors::: - Pavel Safonov p.n.safonov@gmail.com https://github.com/pnsafonov[@pnsafonov] //- References::: - https://www.postgresql.org/docs/current/storage-page-layout.html //- .Options `*-o flavour="postgres14"*`::: PostgreSQL flavour: postgres14, pgproee14, postgres10 `*-o page=0*`::: First page number in file, default is 0 `*-o segment=0*`::: Segment file number (16790.1 is 1), default is 0 -- + `*png*`:: Portable Network Graphics file `*prores_frame*`:: Apple ProRes frame `*protobuf*`:: Protobuf + -- Can decode sub messages::: [source,shell] ---- $ fq -d protobuf '.fields[6].wire_value | protobuf | d' file ---- References::: - https://developers.google.com/protocol-buffers/docs/encoding //- -- + `*protobuf_widevine*`:: Widevine protobuf `*pssh_playready*`:: PlayReady PSSH `*rtmp*`:: Real-Time Messaging Protocol + -- Currently only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3). Show rtmp streams in PCAP file::: [source,shell] ---- fq '.tcp_connections[] | select(.server.port=="rtmp") | d' file.cap ---- References::: - https://rtmp.veriskope.com/docs/spec/ - https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf //- -- + `*safetensors*`:: SafeTensors `*sll2_packet*`:: Linux cooked capture encapsulation v2 `*sll_packet*`:: Linux cooked capture encapsulation `*stl*`:: Standard Tessellation Language + -- Decode binary STL (Stereolithography, Standard Tessellation Language) files. Current limitations::: * No support for ASCII STL files * No support for VisCAM and SolidView colors * No support for Materialise Magics colors //- -- + `*tap*`:: TAP tape format for ZX Spectrum computers + -- The TAP- (and BLK-) format is nearly a direct copy of the data that is stored in real tapes, as it is written by the ROM save routine of the ZX-Spectrum. A TAP file is simply one data block or a group of 2 or more data blocks, one followed after the other. The TAP file may be empty. You will often find this format embedded inside the TZX tape format. The default file extension is `.tap`. Processing JSON files::: When needing to process a generated JSON file it's recommended to convert the plain data bytes to an array by setting `bits_format=byte_array`: [source,shell] ---- fq -o bits_format=byte_array -d tap -V d /path/to/file.tap ---- Authors::: - Michael R. Cook work.mrc@pm.me, original author //- References::: - https://worldofspectrum.net/zx-modules/fileformats/tapformat.html //- -- + `*tar*`:: Tar archive `*tcp_segment*`:: Transmission control protocol segment `*tiff*`:: Tag Image File Format `*tls*`:: Transport layer security + -- Supports decoding of most standard records, messages and extensions. Can also decrypt most standard cipher suites in a PCAP with traffic in both directions if an NSS key log is provided. Decode and decrypt providing a PCAP and key log::: Write traffic to a PCAP file: [source,shell] ---- $ tcpdump -i -w traffic.pcap ---- Make sure your curl TLS backend supports `SSLKEYLOGFILE` and do: [source,shell] ---- $ SSLKEYLOGFILE=traffic.keylog curl --tls-max 1.2 https://host/path ---- Decode, decrypt and query. Uses `keylog=@` to read option value from keylog file: [source,shell] ---- # decode and show whole tree $ fq -o keylog=@traffic.keylog d traffic.pcap # write unencrypted server response to a file. # first .stream is the TCP stream, second .stream is TLS application data stream # # first TCP connections: $ fq -o keylog=@traffic.keylog '.tcp_connections[0].server.stream.stream | tobytes' traffic.pcap > data # first TLS connection: $ fq -o keylog=@traffic.keylog 'first(grep_by(.server.stream | format == "tls")).server.stream.stream | tobytes' > data ---- Supported cipher suites for decryption::: `TLS_DH_ANON_EXPORT_WITH_DES40_CBC_SHA`, `TLS_DH_ANON_EXPORT_WITH_RC4_40_MD5`, `TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA`, `TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA`, `TLS_DHE_DSS_WITH_AES_128_CBC_SHA`, `TLS_DHE_DSS_WITH_AES_128_CBC_SHA256`, `TLS_DHE_DSS_WITH_AES_128_GCM_SHA256`, `TLS_DHE_DSS_WITH_AES_256_CBC_SHA`, `TLS_DHE_DSS_WITH_AES_256_CBC_SHA256`, `TLS_DHE_DSS_WITH_AES_256_GCM_SHA384`, `TLS_DHE_DSS_WITH_DES_CBC_SHA`, `TLS_DHE_DSS_WITH_RC4_128_SHA`, `TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA`, `TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_DHE_RSA_WITH_AES_128_CBC_SHA`, `TLS_DHE_RSA_WITH_AES_128_CBC_SHA256`, `TLS_DHE_RSA_WITH_AES_128_GCM_SHA256`, `TLS_DHE_RSA_WITH_AES_256_CBC_SHA`, `TLS_DHE_RSA_WITH_AES_256_CBC_SHA256`, `TLS_DHE_RSA_WITH_AES_256_GCM_SHA384`, `TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_DHE_RSA_WITH_DES_CBC_SHA`, `TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA`, `TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA`, `TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384`, `TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDH_ECDSA_WITH_RC4_128_SHA`, `TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDH_RSA_WITH_AES_128_CBC_SHA`, `TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDH_RSA_WITH_AES_256_CBC_SHA`, `TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384`, `TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDH_RSA_WITH_RC4_128_SHA`, `TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384`, `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_ECDSA_WITH_RC4_128_SHA`, `TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384`, `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_RSA_WITH_RC4_128_SHA`, `TLS_PSK_WITH_AES_128_CBC_SHA`, `TLS_PSK_WITH_AES_256_CBC_SHA`, `TLS_PSK_WITH_RC4_128_SHA`, `TLS_RSA_EXPORT_WITH_DES40_CBC_SHA`, `TLS_RSA_EXPORT_WITH_RC4_40_MD5`, `TLS_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_RSA_WITH_AES_128_CBC_SHA`, `TLS_RSA_WITH_AES_128_CBC_SHA256`, `TLS_RSA_WITH_AES_128_GCM_SHA256`, `TLS_RSA_WITH_AES_256_CBC_SHA`, `TLS_RSA_WITH_AES_256_CBC_SHA256`, `TLS_RSA_WITH_AES_256_GCM_SHA384`, `TLS_RSA_WITH_DES_CBC_SHA`, `TLS_RSA_WITH_RC4_128_MD5`, `TLS_RSA_WITH_RC4_128_SHA` References::: - https://www.rfc-editor.org/rfc/rfc5246[RFC 5246: The Transport Layer Security (TLS) Protocol] - https://www.rfc-editor.org/rfc/rfc6101[RFC 6101: The Secure Sockets Layer (SSL) Protocol Version 3.0] //- .Options `*-o keylog=""*`::: NSS Key Log content -- + `*toml*`:: Tom's Obvious, Minimal Language `*tzif*`:: Time Zone Information Format + -- Get last transition time::: [source,shell] ---- fq '.v2plusdatablock.transition_times[-1] | tovalue' tziffile ---- Count leap second records::: [source,shell] ---- fq '.v2plusdatablock.leap_second_records | length' tziffile ---- Authors::: - Takashi Oguma https://github.com/bitbears-dev[@bitbears-dev] https://twitter.com/0xb17bea125[@0xb17bea125] //- References::: - https://datatracker.ietf.org/doc/html/rfc8536 //- -- + `*tzx*`:: TZX tape format for ZX Spectrum computers + -- `TZX` is a file format designed to preserve cassette tapes compatible with the ZX Spectrum computers, although some specialized versions of the format have been defined for other machines such as the Amstrad CPC and C64. The format was originally created by Tomaz Kac, who was the maintainer until `revision 1.13`, before passing it to Martijn v.d. Heide. For a brief period the company Ramsoft became the maintainers, and created revision `v1.20`. The default file extension is `.tzx`. Processing JSON files::: When needing to process a generated JSON file it's recommended to convert the plain data bytes to an array by setting `bits_format=byte_array`: [source,shell] ---- fq -o bits_format=byte_array -d tzx -V d /path/to/file.tzx ---- Authors::: - Michael R. Cook work.mrc@pm.me, original author //- References::: - https://worldofspectrum.net/TZXformat.html //- -- + `*udp_datagram*`:: User datagram protocol `*vorbis_comment*`:: Vorbis comment `*vorbis_packet*`:: Vorbis packet `*vp8_frame*`:: VP8 frame `*vp9_cfm*`:: VP9 Codec Feature Metadata `*vp9_frame*`:: VP9 frame `*vpx_ccr*`:: VPX Codec Configuration Record `*wasm*`:: WebAssembly Binary Format + -- Count opcode usage::: [source,shell] ---- $ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' file.wasm ---- List exports and imports::: [source,shell] ---- $ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' file.wasm ---- Authors::: - Takashi Oguma https://github.com/bitbears-dev[@bitbears-dev] https://twitter.com/0xb17bea125[@0xb17bea125] //- References::: - https://webassembly.github.io/spec/core/ //- -- + `*wav*`:: WAV file `*webp*`:: WebP image `*xml*`:: Extensible Markup Language + -- XML can be decoded and encoded into jq values in two ways, elements as object or array. The object variant might be easier to query for a specific value but array might be easier to use to generate xml or to query after all elements of some kind etc. Encoding is done using the `to_xml` function and it will figure out what variant is used based on the input value. It has two optional options `indent` and `attribute_prefix`. Elements as object::: Element can have different shapes depending on body text, attributes and children: - `text` is `{"a":{"#text":"text","@key":"value"}}`, has text (`#text`) and attributes (`@key`) - `text` is `{"a":"text"}` - `text` is `{"a":{"b":"text"}}` one child with only text and no attributes - `text` is `{"a":{"b":["","text"]}}` two children with same name end up in an array - `text` is `{"a":{"b":["",{"#text":"text","@key":"value"}]}}` //- If there is `#seq` attribute it encodes the child element order. Use `-o seq=true` to include sequence number when decoding, otherwise order might be lost. [source,shell] ---- # decode as object is the default $ echo 'bbbccc' | fq -d xml -o seq=true { "a": { "b": [ { "#seq": 0 }, { "#seq": 1, "#text": "bbb" } ], "c": { "#seq": 2, "#text": "ccc", "@attr": "value" } } } # access text of the element $ echo 'bbbccc' | fq '.a.c["#text"]' "ccc" # decode to object and encode to xml $ echo 'bbbccc' | fq -r -d xml -o seq=true 'to_xml({indent:2})' bbb ccc ---- Elements as array::: Elements are arrays of the shape `["name", null|{#text|attribute: "value"}, [, ...]]`. [source,shell] ---- # decode as array $ echo 'bbbccc' | fq -d xml -o array=true [ "a", null, [ [ "b", null, [] ], [ "b", { "#text": "bbb" }, [] ], [ "c", { "#text": "ccc", "attr": "value" }, [] ] ] ] # decode to array and encode to xml $ echo 'bbbccc' | fq -r -d xml -o array=true -o seq=true 'to_xml({indent:2})' bbb ccc # access text of the element, the object variant above is probably easier to use $ echo 'bbbccc' | fq -o array=true '.[2][2][1]["#text"]' "ccc" ---- References::: - https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html[xml.com's Converting Between XML and JSON] //- .Options `*-o array=false*`::: Decode as nested arrays `*-o attribute_prefix="@"*`::: Prefix for attribute keys `*-o seq=false*`::: Use seq attribute to preserve element order -- + `*yaml*`:: YAML Ain't Markup Language `*zip*`:: ZIP archive + -- Supports ZIP64. Timestamp and time zones::: The timestamp accessed via `.local_files[].last_modification` is encoded in ZIP files using https://learn.microsoft.com/en-us/windows/win32/api/oleauto/nf-oleauto-dosdatetimetovarianttime[MS-DOS representation] which lacks a known time zone. Probably the local time/date was used at creation. The `unix_guess` field in `last_modification` is a guess assuming the local time zone was UTC at creation. References::: - https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT - https://opensource.apple.com/source/zip/zip-6/unzip/unzip/proginfo/extra.fld - https://formats.kaitai.io/dos_datetime/ - https://learn.microsoft.com/en-us/windows/win32/api/oleauto/nf-oleauto-dosdatetimetovarianttime //- .Options `*-o uncompress=true*`::: Uncompress and probe files -- + == Encodings, serializations and hashes In addition to binary formats fq also supports various encodings and serialization formats. At the moment fq does not have any dedicated argument for serialization formats but raw string input `-R` slurp `-s` and raw string output `-r` can make things easier. The combination `-Rs` will read all inputs into one string (same as jq). Note that `from*` functions output jq values and `to*` functions take jq values as input so in some cases not all information will be properly preserved. For example, the element and attribute order might change and text and comment nodes might move or be merged. https://github.com/mikefarah/yq[yq] might be a better tool if that is needed. Some example usages: [source,shell] ---- # read yml (format is probed, use -d yaml to force) and do some query $ fq '...' file.yml # convert YAML to JSON # note the -r for raw string output, without it a JSON string with escaped JSON would be output $ fq -r 'tojson({indent:2})' file.yml # add token to URL $ echo -n "https://host.org" | fq -Rsr 'from_url | .user.username="token" | to_url' https://token@host.org # top 3 hosts in src or href attributes: # -d to decode as html, can't be probed as html5 parsers always produce some parse tree # [...] to start collecting values into an array # .. | ."@src"?, ."@href"? | values, recurse and try (?) to get src and href attributes and filter out nulls # from_url.host | values, parse as url and filter out those without a host # count to count unique values, returns [[key, count], ...] # reverse sort by count and pick first 3 # map [key, count] tuples into {key: key, value: count} # from_entries, convert into object $ curl -s https://www.discogs.com/ | fq -d html '[.. | ."@src"?, ."@href"? | values | from_url.host | values] | count | sort_by(-.[1])[0:3] | map({key: .[0], value: .[1]}) | from_entries' { "blog.discogs.com": 9, "st.discogs.com": 10, "www.discogs.com": 14 } # shows how serialization functions can be used on any string, how to transform values and output some other format # read and decode zip file and start an interactive REPL $ fq -i . <(curl -sL https://github.com/stefangabos/world_countries/archive/master.zip) # select from interesting xml file zip> .local_files[] | select(.file_name == "world_countries-master/data/countries/en/world.xml").uncompressed | repl # convert xml into jq value > .local_files[95].uncompressed string> from_xml | repl # sort countries by name and select the first one >> object> .countries.country | sort_by(."@name") | first | repl # see what current input is >>> object> . { "@alpha2": "af", "@alpha3": "afg", "@id": "4", "@name": "Afghanistan" } # remove "@" prefix from keys and convert to YAML and print it >>> object> with_entries(.key |= .[1:]) | to_yaml | print alpha2: af alpha3: afg id: "4" name: Afghanistan # exit all REPLs back to shell >>> object> ^D >> object> ^D > .local_files[95].uncompressed string> ^D zip> ^D ---- === XML and HTML * `from_xml`/`from_xml($opts)` Parse XML into jq value. `$opts` are: ** `{seq: true}` preserve element ordering if more than one sibling. + ** `{array: true}` use nested `[name, attributes, children]` arrays to represent elements. Attributes will be `null` if none and children will be `[]` if none, this is to make it easier to work with as the array always has 3 values. `to_xml` does not require this. + * `from_html`/`from_html($opts)` Parse HTML into jq value. + Similar to `from_xml` but parses html5 in non-script mode. Will always have a `html` root with `head` and `body` elements. + `$opts` are: ** `{array: true}` use nested arrays to represent elements. + ** `{seq: true}` preserve element ordering if more than one sibling. + * `to_xml`/`to_xml($opts)` Serialize jq value into XML. + Assumes object representation if input is an object, and nested arrays if input is an array. + Will automatically add a root `doc` element if jq value has more than one root element. + If a `#seq` is found on at least one element all siblings will be sorted by sequence number. Attributes are always sorted. + `$opts` are: ** `{indent: number}` indent child elements. XML elements can be represented as jq value in two ways, as objects (inspired by https://github.com/clbanning/mxj[mxj] and https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html[xml.com's Converting Between XML and JSON]) or nested arrays. Both representations are lossy and might lose ordering of elements, text nodes and comments. In object representation `from_xml`, `from_html` and `to_xml` support `{seq: true}` option to parse/serialize `{"#seq": }` attributes to preserve element sibling ordering. The object version is denser and convenient to query, the nested arrays version is probably easier to use when generating XML. Let's assume `$xml` is this XML document as a string: [source,xml] ---- text text ---- With object representation an element is represented as: ** Attributes as `@` prefixed `@` keys. ** Text nodes as `#text`. ** Comment nodes as `#comment` keys. ** For explicit sibling ordering `#seq` keys with a number, can be negative, assumed zero if missing. ** Child element with only text as `` key with text as value. ** Child element with more than just text as `` key with value an object. ** Multiple child element siblings with same name as `` key with value as array with strings and objects. [source,shell] ---- > $xml | from_xml { "doc": { "child": [ { "@attr": "1" }, { "#text": "text", "@attr": "2" } ], "other": "text" } } ---- With nested array representation, an array with these values `["", {attributes...}, [children...]]`. - Index 0 is an element name. - Index 1 object attributes (including `#text` and `#comment` keys). - Index 2 array of child elements. [source,shell] ---- > $xml | from_xml({array: true}) [ "doc", null, [ [ "child", { "attr": "1" }, [] ], [ "child", { "#text": "text", "attr": "2" }, [] ], [ "other", { "#text": "text" }, [] ] ] ] ---- Parse and include `#seq` attributes if needed: [source,shell] ---- > $xml | from_xml({seq:true}) { "doc": { "child": [ { "#seq": 0, "@attr": "1" }, { "#seq": 1, "#text": "text", "@attr": "2" } ], "other": { "#seq": 2, "#text": "text" } } } ---- Select values in ``, remove ``, add a `` element, serialize to xml with 2 space indent and print the string [source,shell] ---- > $xml | from_xml.doc | del(.child) | .new = "abc" | {root: .} | to_xml({indent: 2}) | println abc text ---- === JSON * `fromjson` Parse JSON into jq value. * `tojson`/`tojson($opts)` Serialize jq value into JSON. `$opts` are: ** `{indent: number}` Indent depth. * `from_jsonl` Parse JSON lines into jq array. * `to_jsonl` Serialize jq array into JSONL. === jq-flavoured JSON * `from_jq` Parse jq-flavoured JSON into jq value. * `to_jq`/`to_jq($opts)` Serialize jq value into jq-flavoured JSON. jq-flavoured JSON has optional key quotes, `#` comments and can have trailing comma in objects. `$opts` are: ** `{indent: number}` Indent depth. Note that `fromjson` and `tojson` use different naming conventions as they originate from jq's standard library. === YAML * `from_yaml` Parse YAML into jq value. * `to_yaml`/`to_yaml($opts)` Serialize jq value into YAML. `$opts` are: ** `{indent: number}` Indent depth. === TOML * `from_toml` Parse TOML into jq value. * `to_toml`/`to_toml($opts)` Serialize jq value into TOML. `$opts` are: ** `{indent: number}` Indent depth. === CSV * `from_csv`/`from_csv($opts)` Parse CSV into jq value. + To work with tab separated values you can use `from_csv({comma: "\t"})` or `fq -d csv -o 'comma="\t"'`. + `$opts` are: ** `{comma: string}` field separator, default ",". + ** `{comment: string}` comment line character, default "#". + * `to_csv`/`to_csv($opts)` Serialize jq value into CSV. `$opts` are: ** `{comma: string}` field separator, default ",". + === XML entities - `from_xmlentities` Decode XML entities. - `to_xmlentities` Encode XML entities. === URL - `from_urlpath` Decode URL path component. - `to_urlpath` Encode URL path component. Whitespace as %20. - `from_urlencode` Decode URL query encoding. - `to_urlencode` Encode URL to query encoding. Whitespace as "+". - `from_urlquery` Decode URL query into object. For duplicate keys value will be an array. - `to_urlquery` Encode object into query string. - `from_url` Decode URL into object. + -- [source,shell] ---- > "schema://user:pass@host/path?key=value#fragment" | from_url { "fragment": "fragment", "host": "host", "path": "/path", "query": { "key": "value" }, "rawquery": "key=value", "scheme": "schema", "user": { "password": "pass", "username": "user" } } ---- -- - `to_url` Encode object into URL string. === PEM - `from_pem` Decode PEM to binary. - `to_pem`/`to_pem($label)` Encode binary to PEM. `$label` is the PEM header label. === Hex and base64 - `from_hex` Decode hex string to binary. - `to_hex` Encode binary into hex string. - `from_base64`/`from_base64($opts)` Decode base64 encodings into binary. `$opts` are: - `{encoding:string}` encoding variant: `std` (default), `url`, `rawstd` or `rawurl` - `to_base64`/`to_base64($opts)` Encode binary into base64 encodings. `$opts` are: - `{encoding:string}` encoding variant: `std` (default), `url`, `rawstd` or `rawurl` === Hash functions - `to_md4` Hash binary using md4. - `to_md5` Hash binary using md5. - `to_sha1` Hash binary using sha1. - `to_sha256` Hash binary using sha256. - `to_sha512` Hash binary using sha512. - `to_sha3_224` Hash binary using sha3 224. - `to_sha3_256` Hash binary using sha3 256. - `to_sha3_384` Hash binary using sha3 384. - `to_sha3_512` Hash binary using sha3 512. === Text encodings - `to_iso8859_1` Encode string as ISO8859-1 into binary. - `from_iso8859_1` Decode binary as ISO8859-1 into string. - `to_utf8` Encode string as UTF8 into binary. - `from_utf8` Decode binary as UTF8 into string. - `to_utf16` Encode string as UTF16 into binary. - `from_utf16` Decode binary as UTF16 into string. - `to_utf16le` Encode string as UTF16 little-endian into binary. - `from_utf16le` Decode binary as UTF16 little-endian into string. - `to_utf16be` Encode string as UTF16 big-endian into binary. - `from_utf16be` Decode binary as UTF16 big-endian into string. == Interactive REPL The REPL can be useful in some scenarios: - When decoding is slow you can reuse the decode result. - Dig thru a file using sub-REPL to cut down on typing. - Use auto-completion to speed up typing. [source,shell] ---- # start REPL with no (null) input $ fq -i null> # same as $ fq -ni null> # in the REPL you will see a prompt indicating current input and you can type a jq expression to evaluate. # start REPL with one file as input $ fq -i . doc/file.mp3 mp3> # basic arithmetic and jq expressions mp3> 1+1 2 mp3> 1, 2, 3 | . * 2 2 4 6 mp3> [1, 2, 3] | add 6 # "." is the identity function which just returns current input, the mp3 file. mp3> . # access the first frame in the mp3 file mp3> .frames[0] # start a new nested REPL with first frame as input mp3> .frames[0] | repl # prompt shows "path" to current input and that it's an mp3_frame. # Ctrl-D to exit REPL or to shell if last REPL > .frames[0] mp3_frame> ^D # "jq" value of layer in first frame mp3> .frames[0].header.layer | tovalue 3 mp3> .frames[0].header.layer * 2 6 # symbolic value, same as "jq" value mp3> .frames[0].header.layer | tosym 3 # actual underlying decoded value mp3> .frames[0].header.layer | toactual 1 # description of value mp3> .frames[0].header.layer | todescription "MPEG Layer 3" mp3> ^D $ # back to shell ---- Use Ctrl-D to exit and Ctrl-C to interrupt current evaluation. == Examples === Basic usage fq tries to behave the same way as jq as much as possible, so you can do: [source,shell] ---- fq . file fq < file cat file | fq fq . < file fq . *.png *.mp3 fq '.frames[0]' *.mp3 fq '.frames[-1] | tobytes' file.mp3 > last_frame ---- === Common usages [source,shell] ---- # recursively display decode tree but truncate long arrays fq d file # same as fq display file # display all bytes for each value fq dd file # same as fq 'd({array_truncate: 0, string_truncate: 0, display_bytes: 0})' file # display 200 bytes for each value fq 'd({display_bytes: 200})' file # recursively display decode tree without truncating fq da file # same as fq 'd({array_truncate: 0, string_truncate: 0})' file # display a specific decode tree one level fq '.path[1].to.value' file # display a specific decode tree all levels fq '.path[1].to.value | d' file fq '.path[1].to.value | dd' file fq '.path[1].to.value | da' file # recursively and verbosely display decode tree fq dv file # same as fq 'd({array_truncate: 0, string_truncate: 0, verbose: true})' file # JSON representation for whole file fq tovalue file # or use -V (--value-output) that does tovalue automatically fq -V . file # or -Vr if the value is a string and you want a "raw" string fq -Vr .path.to.string file # JSON but raw bit fields truncated fq -o bits_format=truncate tovalue file # JSON but raw bit fields as md5 hex string fq -o bits_format=md5 tovalue file # JSON but raw bit fields as byte arrays fq -o bits_format=byte_array tovalue file # look up a path fq '.some[1].path' file # look up a path and output JSON fq -V '.some[1].path' file # can be a query that outputs multiple values # this outputs first and last value in .some array and .path, three values in total fq -V '.some[0,-1], .path' file # grep whole tree by value fq 'grep("^prefix")' file fq 'grep(123)' file # grep whole tree by condition fq 'grep_by(. >= 100 and . <= 200)' file # recursively look for values fulfilling some condition fq '.. | select(.type=="trak")?' file fq 'grep_by(.type=="trak")' file # grep_by(f) is an alias for .. | select(f)?, that is: recurse, select and ignore errors # recursively look for decode value roots for a format fq '.. | select(format=="jpeg")' file # can also use grep_by fq 'grep_by(format=="jpeg")' file # recursively look for first decode value root for a format fq 'first(.. | select(format=="jpeg"))' file fq 'first(grep_by(format=="jpeg"))' file # decode file as mp4 and return a result even if there are some errors fq -d mp4 file.mp4 # decode file as mp4 and also ignore validity assertions fq -o force=true -d mp4 file.mp4 ---- == Links https://github.com/wader/fq[github.com/wader/fq] == Copyright See LICENSE file in source tree. == See also *jq*(1) ifdef::backend-html5[] ++++ ++++ endif::[] --- ### Doc Fq.1.Tmpl (`doc/fq.1.tmpl.adoc`) = fq(1) Mattias Wadman v$FQ_VERSION :doctype: manpage :manmanual: FQ :mansource: FQ :toc: left :toclevels: 3 :sectanchors: :reproducible: :description: fq - jq for binary formats. Tool, language and decoders for working with binary formats == Name fq - tool, language and decoders for working with binary formats == Synopsis `*fq [_OPTIONS_] [--] [_EXPRESSION_] [_FILE_...]*` == Description **fq** is a tool, language, and set of decoders for working with binary formats and data. In most cases it behaves and feels similar to https://jqlang.github.io/jq/[jq] and it also uses the same expression language. To get the most out of fq it's recommended to learn more about jq. It features a structural hex viewer, nested format decoding, slicing and concatenating binary data, bit-level decoding and an interactive REPL with auto-completion. [source,shell] ---- # Evaluate "d" (display) for file.mp4 $ fq d file.mp4 # structural hex output # Evaluate "da" (display all) for file.mp4 $ fq da file.mp4 # structural hex output # JSON for file.mp4 $ fq -V . file.mp4 # JSON output # Evaluate ".boxes[0].type" for all *.mp4 files # -Vr to output value without quotes $ fq -Vr '.boxes[0].type' *.mp4 ftyp # Evaluate "1+2" without reading any input $ fq -n 1+2 3 ---- For more usage examples see the examples section at the end of the documentation. == Options ////jq-eval ( _opt_cli_opts | to_entries | sort_by(.value.long) | .[] | .value as $v | ([$v.long, $v.short | select(. != null)] | join("*`, `*") ) as $flags | if $v.bool then "`*\($flags)*`::" elif $v.pairs then "`*\($flags) \($v.pairs)*`::" elif $v.string then "`*\($flags) \($v.string)*`::" elif $v.array then "`*\($flags) \($v.array)*`::" elif $v.object then "`*\($flags) \($v.object)*`::" elif $v.positional then "`*\($flags)*`::" else error("unknown option type \($v)") end , " \($v.description)" , ( select($v.long == "--option" ) | ( def _type_to_human: { "boolean": "true|false" , "csv_ranges_array": "ranges=string,..." , "csv_kv_obj": "key=value,..." }[.] // .; ( _opt_options | to_entries | sort_by(.key) | .[] | . as $opt | select(.value.internal | not) | "`*-o \(.key)=\(.value.type | _type_to_human)*`:::" , " \(.value.description)" , if .value.values then ( .value.values | to_entries | sort_by(.key) | .[] | ( "`*-o \($opt.key)=\(.key)*`::::" , " \(.value)" ) ) else empty end , "" ) ) ) , "" ) ////jq-eval == Configuration === Init files To add your own functions you can use `init.jq` that is read from: macOS:: `$HOME/Library/Application Support/fq/init.jq` or + `$HOME/.config/fq/init.jq` Linux and BSD:: `$HOME/.config/fq/init.jq` Windows:: `%AppData%\fq\init.jq` === Environment `*NO_COLOR*`:: Non-empty string disables color output. `*CLIUNICODE*`:: Enables use of unicode output characters. `*COMPLETION_TIMEOUT*`:: REPL completion timeout in seconds. `*NO_DECODE_PROGRESS*`:: Disables decode progress indicator. == Exit status `*0*` Success + `*2*` Argument error or input IO error + `*3*` Expression compile error + `*4*` Decode error + `*5*` Expression error == Expression === Syntax See *jq*(1) for syntax details. But here are some common beginner gotchas: - Functions that take no arguments are called using `name` instead of `name()`. - Arguments are separated by `;` instead of `,`. Comma is used to concatenate output streams. To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a single argument `1, 2`, a filter that outputs `1` and then `2`, to `f`. - Expressions can return or "output" zero or more values. This is how iteration etc is done, `1, 2` outputs `1` then `2`. - Similar to shell pipelines, implicit input and output are used and piped together using `|`. `.` is used to refer to current input. Ex `1 | . + 2` outputs `3`, `1, 2 | . + 2` outputs `3` and `4`. - In the jq manual and other jq related documentation you might see `name/2`, this means the function `name` takes two arguments (arity). === Additional features fq uses an extended variant of the jq language with a few extra features: * Arbitrary-precision integers and arithmetic. * Supports raw strings using back-ticks. String interpolation and codepoint escapes are not processed. + ---- `hello \("world")\ud83c\udf0d` ---- results in the string `hello \("world")\ud83c\udf0d` or as JSON `"hello \\(\"world\")\\ud83c\\udf0d"`. In contrast + ---- "hello \("world")\ud83c\udf0d" ---- results in the string `hello world🌍` or as JSON `"hello world🌍"`. + * Supports more number bases in integer literals. ** `0xabcd` (Hexadecimal). ** `0o125715` (Octal). ** `0b1010101111001101` (Binary). ** Grouping using underscore `0xab_cd`. * Binary and decode value types, see below. * Try include using an ending question mark `include "file?";` that doesn't fail if file is missing or has errors. * Some values can act as an object with keys even when they are arrays, numbers etc. * There can be keys hidden from `keys` and `[]`. * Some values are readonly and can't be updated or will convert to JSON on update. * Mixing `--args` and `--jsonargs` does not behave the same as in jq. === Additional functions `*band*`, `*bor*`, `*bxor*`, `*bsl*`, `*bsr*`, `*bnot*`:: Bitwise operations as functions. Work the same as jq's math functions. Functions that take one argument use input, `1 | bnot`, and functions with more arguments ignore the input and use formal arguments `bsl(1; 3)`. `*chunk($size)*`:: Split array or string into `$size` length chunks. Last chunk might be shorter. `*count*`, `*count_by(f)*`:: Like `group` but outputs array of `[value, count]` pairs. `*delta*`, `*delta_by(f)*`:: Array with differences between consecutive elements. `delta` is the same as `delta_by(.b - .a)`. `*diff($a; $b)*`:: Produce a diff between `$a` and `$b`. Differences are represented as an object `{a: , b: }`. `*expr_to_path*`:: Converts from a string `".key[1]"` to a path value `["key", 1]`. `*grep_by(f)*`:: Recursively select using a filter and ignore any errors. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`. This is the same as doing `.. | select(f)?`. `*group*`:: Group values, same as `group_by(.)`. `*path_to_expr*`:: Converts a path value `["key", 1]` to a string `".key[1]"`. `*paste*`:: Read string from stdin until ^D. Useful for pasting text. Ex: `paste | from_pem | asn1_ber | repl` read from stdin then decode and start a new sub-REPL with result. `*streaks*`, `*streaks_by(f)*`:: Like `group` but groups streaks based on condition. `*repl*`, `*repl($opts)*`:: Nested REPL. Must be last in a pipeline. `repl` can "slurp" outputs, ex: `1, 2, 3 | repl`, and supports options, ex: `[1,2,3] | repl({compact: true})`. `*slurp("")*`:: Slurp outputs and saves them to `$name`. Must be last in the pipeline. Will be available as a global array `$name`. Ex `1,2,3 | slurp("a")`, `$a[]` same as `spew("a")`. `*spew*`, `*spew("")*`:: Outputs all or a specific slurp. Ex: `spew("a")`. `*println*`, `*print*`:: Print string or compact JSON to stdout with and without new line. `*printerrln*`, `*printerr*`:: Print string or compact JSON to stderr with and without new line. `*hexdump*`, `*hexdump($opts)*`, `*hd*`, `*hd($opts)*`:: Display binary as a hex dump. `*format*`:: Format name for a decode value. Ex: `first(grep_by(format == "jpeg"))`. === Decode value A decode value is the type returned from decoding a format and used to represent values produced by a decoder. It can be seen as representing any standard jq type but with some additional properties attached. Each decode value has these properties: * Bit range in the input. Can be used as a binary using `tobytes`, `tobytesrange`, `tobits` and `tobitsrange`. * If scalar type, an actual value: ** This is the decoded representation of the bits, a number, string, bool etc. ** Can be accessed using `toactual`. * If scalar type, an optional symbolic value: ** Is usually a mapping of the actual to symbolic value, ex: map number to a string value. ** Can be accessed using `tosym`. * An optional description: ** Can be accessed using `todescription`. * `parent` is the parent decode value * `parents` is all the parent decode values * `topath` is the jq path for the decode value * `torepr` converts decode value to its representation if possible The value of a decode value is the symbolic value if available and otherwise the actual value. To explicitly access the value use `tovalue`. In most expressions this is not needed as it will be done automatically. === Decode value functions `*root*`:: Root decode value for decode value. `*buffer_root*`:: Root decode value of sub buffer for decode value. `*format_root*`:: Root decode value of nested format for decode value. `*parent*`:: Parent decode value for decode value. `*parents*`:: Outputs all parent decode values from decode value. `*topath*`:: Path for decode value. Use `path_to_expr` to get a string representation. `*tovalue*`, `*tovalue($opts)*`:: Symbolic, if available, or actual value for decode value. `*toactual*`, `*toactual($opts)*`:: Actual value for decode value. `*tosym*`, `*tosym($opts)*`:: Symbolic value for decode value. `*todescription*`:: Description for decode value. `*torepr*`:: Converts decode value into what it represents. For example converts msgpack decode value into a value representing its JSON representation. `*decode*`, `*decode($name)*`, `*decode($name; $opts)*`:: Decode binary as a format. `$name` is format name, `$opts` is format options. Ex: `... | decode("mp4")`. Note that all decoders are available as functions. Ex: `... | mp4` is the same as `... | decode("mp4").` `tobytes`, `tobytesrange`, `tobits` and `tobitsrange` on a decode value will return the raw source bits as a binary. === Binary Binary type is used to store raw bits or bytes. Raw bits will act as zero bits padded strings in standard jq expressions. Use `tobits` and `tobytes` to create them from decode value, string, number or binary array. `tobytes` will if needed zero pad most significant bits to be byte aligned. There is also `tobitsrange` and `tobytesrange` which do the same thing but will preserve source range when displayed. - `"string" | tobytes` produces a binary with UTF8 bytes. - `1234 | tobits` produces a binary with the unsigned big-endian integer 1234 with enough bits to represent the number. Use `tobytes` to get the same but with enough bytes to represent the number. This is different to how numbers work inside binary arrays where they are limited to 0-255. - `["abc", 123, ...] | tobytes` produces a binary from a binary array. See <> below. - `.[index]` accesses bit or byte at index `index`. Index is in units. - `[0x12, 0x34, 0x56] | tobytes[1]` is `0x34` - `[0x12, 0x34, 0x56] | tobits[3]` is `1` - `.[start:]`, `.[start:end]` or `.[:end]` is normal jq slice syntax and will slice the binary from `start` to `end`. `start` and `end` are in units. - `[0x12, 0x34, 0x56] | tobytes[1:2]` will be a binary with the byte `0x34` - `[0x12, 0x34, 0x56] | tobits[4:12]` will be a binary with the byte `0x23` - `[0x12, 0x34, 0x56] | tobits[4:20]` will be a binary with the bytes `0x23`, `0x45` - `[0x12, 0x34, 0x56] | tobits[4:20] | tobytes[1:]` will be a binary with the byte `0x45` - Both `.[index]` and `.[start:end]` support negative indices to index from end. - `explode` outputs an array with all bytes or bits as integers. === Binary functions `*grep($v)*`, `*grep($v; $flags)*`, `*vgrep($v)*`, `*vgrep($v; $flags)*`, `*bgrep($v)*`, `*bgrep($v; $flags)*`:: Recursively match `$v`. `$v` is a scalar to match, where a string is treated as a regexp. A binary will match exact bytes. `$flags` arguments are regexp flags with additional flag "b" that will treat each byte in the input binary as a code point. This makes it possible to match exact bytes. `*fgrep($v)*`, `*fgrep($v; $flags)*`:: Recursively match field name in a decode value. `*tobits*`:: Transform input to binary with bit as unit and don't preserve source range. `*tobitsrange*`:: Transform input to binary with bit as unit and preserve source range. `*tobytes*`:: Transform input to binary with byte as unit and don't preserve source range. `*tobytesrange*`:: Transform input to binary with byte as unit and preserve source range. `*open*`:: Open file for reading. Returns a binary. Ex: `"file" | open | decode`, `"file" | open | tostring`. The standard jq functions `test`, `match`, `capture`, `scan` and `split` also work on binary values, matching against the raw bytes. === Binary array [[binary_array]] Binary array is a value "shape" and not a new type. It's an array of numbers, strings, binaries or other binary arrays. They can be used as input to `tobits`, `tobytes` or other functions that accept a binary as input. - Number is a byte with value 0-255 - String as UTF8 bytes - Binary as is - Binary array used recursively Binary arrays are similar to and inspired by https://www.erlang.org/doc/man/erlang.html#type-iolist[Erlang iolist]. Some examples: - `[0, 123, 255] | tobytes` will be binary with 3 bytes 0, 123 and 255. - `[0, [123, 255]] | tobytes` same as above. - `[0, 1, 1, 0, 0, 1, 1, 0 | tobits] | tobytes` will be binary with 1 byte, 0x66. - `[(.a | tobytes[-10:]), 255, (.b | tobits[:10])] | tobytes` the concatenation of the last 10 bytes of `.a`, byte of value 255 and the first 10 bits of `.b`. === Differences to jq See https://github.com/itchyny/gojq#difference-to-jq[gojq's differences to jq]. === Naming inconsistencies jq's naming convention is a bit inconsistent. Some standard library functions are named `tojson` while others `from_entries`. fq follows this tradition but tries to use `snake_case` unless there is a good reason. Here are all the non-snake_case functions added by fq. Most of them deal with decode and binary values which are new "primitive" types: - `toactual` - `tobits` - `tobitsrange` - `tobytes` - `tobytesrange` - `todescription` - `topath` - `torepr` - `tosym` - `tovalue` == Display output `display` or `d` is the main function for displaying values and is also the function that will be used if no other output function is explicitly used. If its input is a decode value it will output a dump and tree structure or otherwise it will output as JSON. Below are some usage examples: The first and second examples do the same thing, inputting `"hello"` to `display`. image::display_json.svg[] In the next few examples we select out the first "edit list" box in an mp4 file and display it in various ways. By default, display will only show the root level: image::display_decode_value.svg[] First row shows a ruler with byte offset into the line and jq path for the value. The columns are: - Start address for the line. For example we see that `size` starts at `0xd5c` (row) + `0x09` (column) = `0xd65`. - Hex representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it. - ASCII representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it. - Tree structure of decoded value, symbolic value and description. Notation: - `{}` value is an object that might have nested values. - `[start:end]` value is an array with index starting at `start` and ending at `end` (exclusive). With `display` or `d` it will recursively show the whole tree: image::display_decode_value_d.svg[] Same but verbose `dv`: image::display_decode_value_dv.svg[] In verbose mode bit ranges and array element names are shown. Bit ranges use `[.]-[.]` as notation where `.` is left out if byte aligned. For example `type` starts at byte `0xd69` bit `0` (`.0` is left out) and ends at `0xd6d` bit `0` (exclusive) and has a size of `4` bytes. This verbosely displays the header of the second frame in an mp3 file which has a bunch of non-byte-aligned fields: image::display_decode_value_dv2.svg[] Here the `sync` pattern starts at `0xb79` (bit `0`) and ends at `0xb7a.3` (exclusive) and has a size of `1` byte and `3` bits, `11` bits in total (`8+3`). There are also some other `display` aliases: - `da` is `display({array_truncate: 0, string_truncate: 0})` don't truncate arrays and strings. - `dd` is `display({array_truncate: 0, string_truncate: 0, display_bytes: 0})` don't truncate arrays and strings, show all raw bytes. - `dv` is `display({array_truncate: 0, string_truncate: 0, verbose: true})` don't truncate arrays and strings and display verbosely. - `ddv` is `display({array_truncate: 0, string_truncate: 0, display_bytes: 0, verbose: true})` don't truncate arrays and strings, show all raw bytes and display verbosely. == Formats By default fq will try to automatically determine input format. In some cases this might fail or is not possible, then a format can be specified using `-d NAME`. It's possible sometimes to force decode and get a partial or broken result using `-o force=true`. [source,shell] ---- # decode as msgpack $ fq -d msgpack d file # force decode as msgpack $ fq -d msgpack -o force=true d file # see msgpack format help $ fq -h msgpack # list supported formats $ fq -h formats ---- === Format options Some formats have their own options that can be set using `-o`. For example the `mp4` format has a `decode_samples` option that controls if individual samples should be decoded. To disable it one can do `fq -o decode_samples=false . file.mp4`. See format list for options. === Format functions In addition to using `-d` all format decoders are also available as normal jq functions. Each format provides multiple functions: ``:: Decode and return a decode value even on error. Ex: `... | mp4` `($options)`:: Same as above with format options. Ex: `... | mp4({decode_samples: false})` `from_`:: Decode or throw on error. Ex: `... | from_mp4` `from_($options)`:: Same as above with format options. Ex: `... | from_mp4({decode_samples: false})` Example usage: [source,shell] ---- # decode jpeg found inside some other format $ fq '.some[].query | jpeg' file # decode jpeg at byte range 100-200 $ fq -d bytes '.[100:200] | jpeg' file ---- === Supported formats ////jq-eval ( formats | to_entries | sort_by(.key) | .[] | . as {key: $key, value: $f} | [ ( _registry.files[][] | select(.name=="\($key).md").data | markdown | _markdown_to_asciidoc | . ) , if .value.decode_in_arg then ( "" , ".Options" , ( $f.decode_in_arg | to_entries[] | "`*-o \(.key)=\(.value | tojson)*`:::" , " \($f.decode_in_arg_doc[.key] | if . == "" then "No description" end)" , "" ) ) else empty end ] as $c | "`*\($key)*`::" , " \(.value.description)" , if $c != [] then ( "+" , "--" , $c[] , "--" , "+" ) else empty end , "" ) ////jq-eval == Encodings, serializations and hashes In addition to binary formats fq also supports various encodings and serialization formats. At the moment fq does not have any dedicated argument for serialization formats but raw string input `-R` slurp `-s` and raw string output `-r` can make things easier. The combination `-Rs` will read all inputs into one string (same as jq). Note that `from*` functions output jq values and `to*` functions take jq values as input so in some cases not all information will be properly preserved. For example, the element and attribute order might change and text and comment nodes might move or be merged. https://github.com/mikefarah/yq[yq] might be a better tool if that is needed. Some example usages: [source,shell] ---- # read yml (format is probed, use -d yaml to force) and do some query $ fq '...' file.yml # convert YAML to JSON # note the -r for raw string output, without it a JSON string with escaped JSON would be output $ fq -r 'tojson({indent:2})' file.yml # add token to URL $ echo -n "https://host.org" | fq -Rsr 'from_url | .user.username="token" | to_url' https://token@host.org # top 3 hosts in src or href attributes: # -d to decode as html, can't be probed as html5 parsers always produce some parse tree # [...] to start collecting values into an array # .. | ."@src"?, ."@href"? | values, recurse and try (?) to get src and href attributes and filter out nulls # from_url.host | values, parse as url and filter out those without a host # count to count unique values, returns [[key, count], ...] # reverse sort by count and pick first 3 # map [key, count] tuples into {key: key, value: count} # from_entries, convert into object $ curl -s https://www.discogs.com/ | fq -d html '[.. | ."@src"?, ."@href"? | values | from_url.host | values] | count | sort_by(-.[1])[0:3] | map({key: .[0], value: .[1]}) | from_entries' { "blog.discogs.com": 9, "st.discogs.com": 10, "www.discogs.com": 14 } # shows how serialization functions can be used on any string, how to transform values and output some other format # read and decode zip file and start an interactive REPL $ fq -i . <(curl -sL https://github.com/stefangabos/world_countries/archive/master.zip) # select from interesting xml file zip> .local_files[] | select(.file_name == "world_countries-master/data/countries/en/world.xml").uncompressed | repl # convert xml into jq value > .local_files[95].uncompressed string> from_xml | repl # sort countries by name and select the first one >> object> .countries.country | sort_by(."@name") | first | repl # see what current input is >>> object> . { "@alpha2": "af", "@alpha3": "afg", "@id": "4", "@name": "Afghanistan" } # remove "@" prefix from keys and convert to YAML and print it >>> object> with_entries(.key |= .[1:]) | to_yaml | print alpha2: af alpha3: afg id: "4" name: Afghanistan # exit all REPLs back to shell >>> object> ^D >> object> ^D > .local_files[95].uncompressed string> ^D zip> ^D ---- === XML and HTML * `from_xml`/`from_xml($opts)` Parse XML into jq value. `$opts` are: ** `{seq: true}` preserve element ordering if more than one sibling. + ** `{array: true}` use nested `[name, attributes, children]` arrays to represent elements. Attributes will be `null` if none and children will be `[]` if none, this is to make it easier to work with as the array always has 3 values. `to_xml` does not require this. + * `from_html`/`from_html($opts)` Parse HTML into jq value. + Similar to `from_xml` but parses html5 in non-script mode. Will always have a `html` root with `head` and `body` elements. + `$opts` are: ** `{array: true}` use nested arrays to represent elements. + ** `{seq: true}` preserve element ordering if more than one sibling. + * `to_xml`/`to_xml($opts)` Serialize jq value into XML. + Assumes object representation if input is an object, and nested arrays if input is an array. + Will automatically add a root `doc` element if jq value has more than one root element. + If a `#seq` is found on at least one element all siblings will be sorted by sequence number. Attributes are always sorted. + `$opts` are: ** `{indent: number}` indent child elements. XML elements can be represented as jq value in two ways, as objects (inspired by https://github.com/clbanning/mxj[mxj] and https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html[xml.com's Converting Between XML and JSON]) or nested arrays. Both representations are lossy and might lose ordering of elements, text nodes and comments. In object representation `from_xml`, `from_html` and `to_xml` support `{seq: true}` option to parse/serialize `{"#seq": }` attributes to preserve element sibling ordering. The object version is denser and convenient to query, the nested arrays version is probably easier to use when generating XML. Let's assume `$xml` is this XML document as a string: [source,xml] ---- text text ---- With object representation an element is represented as: ** Attributes as `@` prefixed `@` keys. ** Text nodes as `#text`. ** Comment nodes as `#comment` keys. ** For explicit sibling ordering `#seq` keys with a number, can be negative, assumed zero if missing. ** Child element with only text as `` key with text as value. ** Child element with more than just text as `` key with value an object. ** Multiple child element siblings with same name as `` key with value as array with strings and objects. [source,shell] ---- > $xml | from_xml { "doc": { "child": [ { "@attr": "1" }, { "#text": "text", "@attr": "2" } ], "other": "text" } } ---- With nested array representation, an array with these values `["", {attributes...}, [children...]]`. - Index 0 is an element name. - Index 1 object attributes (including `#text` and `#comment` keys). - Index 2 array of child elements. [source,shell] ---- > $xml | from_xml({array: true}) [ "doc", null, [ [ "child", { "attr": "1" }, [] ], [ "child", { "#text": "text", "attr": "2" }, [] ], [ "other", { "#text": "text" }, [] ] ] ] ---- Parse and include `#seq` attributes if needed: [source,shell] ---- > $xml | from_xml({seq:true}) { "doc": { "child": [ { "#seq": 0, "@attr": "1" }, { "#seq": 1, "#text": "text", "@attr": "2" } ], "other": { "#seq": 2, "#text": "text" } } } ---- Select values in ``, remove ``, add a `` element, serialize to xml with 2 space indent and print the string [source,shell] ---- > $xml | from_xml.doc | del(.child) | .new = "abc" | {root: .} | to_xml({indent: 2}) | println abc text ---- === JSON * `fromjson` Parse JSON into jq value. * `tojson`/`tojson($opts)` Serialize jq value into JSON. `$opts` are: ** `{indent: number}` Indent depth. * `from_jsonl` Parse JSON lines into jq array. * `to_jsonl` Serialize jq array into JSONL. === jq-flavoured JSON * `from_jq` Parse jq-flavoured JSON into jq value. * `to_jq`/`to_jq($opts)` Serialize jq value into jq-flavoured JSON. jq-flavoured JSON has optional key quotes, `#` comments and can have trailing comma in objects. `$opts` are: ** `{indent: number}` Indent depth. Note that `fromjson` and `tojson` use different naming conventions as they originate from jq's standard library. === YAML * `from_yaml` Parse YAML into jq value. * `to_yaml`/`to_yaml($opts)` Serialize jq value into YAML. `$opts` are: ** `{indent: number}` Indent depth. === TOML * `from_toml` Parse TOML into jq value. * `to_toml`/`to_toml($opts)` Serialize jq value into TOML. `$opts` are: ** `{indent: number}` Indent depth. === CSV * `from_csv`/`from_csv($opts)` Parse CSV into jq value. + To work with tab separated values you can use `from_csv({comma: "\t"})` or `fq -d csv -o 'comma="\t"'`. + `$opts` are: ** `{comma: string}` field separator, default ",". + ** `{comment: string}` comment line character, default "#". + * `to_csv`/`to_csv($opts)` Serialize jq value into CSV. `$opts` are: ** `{comma: string}` field separator, default ",". + === XML entities - `from_xmlentities` Decode XML entities. - `to_xmlentities` Encode XML entities. === URL - `from_urlpath` Decode URL path component. - `to_urlpath` Encode URL path component. Whitespace as %20. - `from_urlencode` Decode URL query encoding. - `to_urlencode` Encode URL to query encoding. Whitespace as "+". - `from_urlquery` Decode URL query into object. For duplicate keys value will be an array. - `to_urlquery` Encode object into query string. - `from_url` Decode URL into object. + -- [source,shell] ---- > "schema://user:pass@host/path?key=value#fragment" | from_url { "fragment": "fragment", "host": "host", "path": "/path", "query": { "key": "value" }, "rawquery": "key=value", "scheme": "schema", "user": { "password": "pass", "username": "user" } } ---- -- - `to_url` Encode object into URL string. === PEM - `from_pem` Decode PEM to binary. - `to_pem`/`to_pem($label)` Encode binary to PEM. `$label` is the PEM header label. === Hex and base64 - `from_hex` Decode hex string to binary. - `to_hex` Encode binary into hex string. - `from_base64`/`from_base64($opts)` Decode base64 encodings into binary. `$opts` are: - `{encoding:string}` encoding variant: `std` (default), `url`, `rawstd` or `rawurl` - `to_base64`/`to_base64($opts)` Encode binary into base64 encodings. `$opts` are: - `{encoding:string}` encoding variant: `std` (default), `url`, `rawstd` or `rawurl` === Hash functions - `to_md4` Hash binary using md4. - `to_md5` Hash binary using md5. - `to_sha1` Hash binary using sha1. - `to_sha256` Hash binary using sha256. - `to_sha512` Hash binary using sha512. - `to_sha3_224` Hash binary using sha3 224. - `to_sha3_256` Hash binary using sha3 256. - `to_sha3_384` Hash binary using sha3 384. - `to_sha3_512` Hash binary using sha3 512. === Text encodings - `to_iso8859_1` Encode string as ISO8859-1 into binary. - `from_iso8859_1` Decode binary as ISO8859-1 into string. - `to_utf8` Encode string as UTF8 into binary. - `from_utf8` Decode binary as UTF8 into string. - `to_utf16` Encode string as UTF16 into binary. - `from_utf16` Decode binary as UTF16 into string. - `to_utf16le` Encode string as UTF16 little-endian into binary. - `from_utf16le` Decode binary as UTF16 little-endian into string. - `to_utf16be` Encode string as UTF16 big-endian into binary. - `from_utf16be` Decode binary as UTF16 big-endian into string. == Interactive REPL The REPL can be useful in some scenarios: - When decoding is slow you can reuse the decode result. - Dig thru a file using sub-REPL to cut down on typing. - Use auto-completion to speed up typing. [source,shell] ---- # start REPL with no (null) input $ fq -i null> # same as $ fq -ni null> # in the REPL you will see a prompt indicating current input and you can type a jq expression to evaluate. # start REPL with one file as input $ fq -i . doc/file.mp3 mp3> # basic arithmetic and jq expressions mp3> 1+1 2 mp3> 1, 2, 3 | . * 2 2 4 6 mp3> [1, 2, 3] | add 6 # "." is the identity function which just returns current input, the mp3 file. mp3> . # access the first frame in the mp3 file mp3> .frames[0] # start a new nested REPL with first frame as input mp3> .frames[0] | repl # prompt shows "path" to current input and that it's an mp3_frame. # Ctrl-D to exit REPL or to shell if last REPL > .frames[0] mp3_frame> ^D # "jq" value of layer in first frame mp3> .frames[0].header.layer | tovalue 3 mp3> .frames[0].header.layer * 2 6 # symbolic value, same as "jq" value mp3> .frames[0].header.layer | tosym 3 # actual underlying decoded value mp3> .frames[0].header.layer | toactual 1 # description of value mp3> .frames[0].header.layer | todescription "MPEG Layer 3" mp3> ^D $ # back to shell ---- Use Ctrl-D to exit and Ctrl-C to interrupt current evaluation. == Examples === Basic usage fq tries to behave the same way as jq as much as possible, so you can do: [source,shell] ---- fq . file fq < file cat file | fq fq . < file fq . *.png *.mp3 fq '.frames[0]' *.mp3 fq '.frames[-1] | tobytes' file.mp3 > last_frame ---- === Common usages [source,shell] ---- # recursively display decode tree but truncate long arrays fq d file # same as fq display file # display all bytes for each value fq dd file # same as fq 'd({array_truncate: 0, string_truncate: 0, display_bytes: 0})' file # display 200 bytes for each value fq 'd({display_bytes: 200})' file # recursively display decode tree without truncating fq da file # same as fq 'd({array_truncate: 0, string_truncate: 0})' file # display a specific decode tree one level fq '.path[1].to.value' file # display a specific decode tree all levels fq '.path[1].to.value | d' file fq '.path[1].to.value | dd' file fq '.path[1].to.value | da' file # recursively and verbosely display decode tree fq dv file # same as fq 'd({array_truncate: 0, string_truncate: 0, verbose: true})' file # JSON representation for whole file fq tovalue file # or use -V (--value-output) that does tovalue automatically fq -V . file # or -Vr if the value is a string and you want a "raw" string fq -Vr .path.to.string file # JSON but raw bit fields truncated fq -o bits_format=truncate tovalue file # JSON but raw bit fields as md5 hex string fq -o bits_format=md5 tovalue file # JSON but raw bit fields as byte arrays fq -o bits_format=byte_array tovalue file # look up a path fq '.some[1].path' file # look up a path and output JSON fq -V '.some[1].path' file # can be a query that outputs multiple values # this outputs first and last value in .some array and .path, three values in total fq -V '.some[0,-1], .path' file # grep whole tree by value fq 'grep("^prefix")' file fq 'grep(123)' file # grep whole tree by condition fq 'grep_by(. >= 100 and . <= 200)' file # recursively look for values fulfilling some condition fq '.. | select(.type=="trak")?' file fq 'grep_by(.type=="trak")' file # grep_by(f) is an alias for .. | select(f)?, that is: recurse, select and ignore errors # recursively look for decode value roots for a format fq '.. | select(format=="jpeg")' file # can also use grep_by fq 'grep_by(format=="jpeg")' file # recursively look for first decode value root for a format fq 'first(.. | select(format=="jpeg"))' file fq 'first(grep_by(format=="jpeg"))' file # decode file as mp4 and return a result even if there are some errors fq -d mp4 file.mp4 # decode file as mp4 and also ignore validity assertions fq -o force=true -d mp4 file.mp4 ---- == Links https://github.com/wader/fq[github.com/wader/fq] == Copyright See LICENSE file in source tree. == See also *jq*(1) ifdef::backend-html5[] ++++ ++++ endif::[] --- ### Doc Presentations Bts2022 Abstract (`doc/presentations/bts2022/abstract.txt`) fq is inspired by the well known jq tool and language and allows you to work with binary formats the same way you would using jq. In addition it can also present data similar to a hex viewer, transform, slice and concatenate binary data, supports nested formats and has an interactive REPL with auto-completion. It was originally designed to query, inspect and debug codecs and metadata in media files and containers like mp4, flac, mp3, jpeg. But has since been extended to support a variety of formats like executables, packet captures including TCP reassembly and serialization formats like ASN1 BER, Avro, CBOR, protobuf and a lot more. In summary it aims to be something like jq, hexdump, dd and gdb combined into one. --- --- METRICS --- - Files Extracted: 25 - Estimated Token Budget: ~42113 tokens - Canonical Reference: https://codewiki.google/github.com/wader/fq