Lib
jc.lib
Table of Contents
jc - JSON Convert lib module
all_parser_info
def all_parser_info(
documentation: bool = False,
show_hidden: bool = False,
show_deprecated: bool = False) -> List[jc.jc_types.ParserInfoType]Returns a list of dictionaries that includes metadata for all parser
modules. By default only non-hidden, non-deprecated parsers are
returned.
Parameters:
documentation: (boolean) include parser docstrings if `True`
show_hidden: (boolean) also show parsers marked as hidden
in their info metadata.
show_deprecated: (boolean) also show parsers marked as
deprecated in their info metadata.get_help
def get_help(parser_mod_name: Union[str, module]) -> NoneShow help screen for the selected parser.
This function will accept module_name, cli-name, and
--argument-name variants of the module name string as well as a
parser module object.
get_parser
def get_parser(parser_mod_name: Union[str, module]) -> moduleReturn the parser module object and check that the module is a valid
parser module.
Parameters:
parser_mod_name: (string or Name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
If a Module is given and the Module
is a valid parser Module, then the
same Module is returned.Returns:
Module: the parser Module objectRaises:
ModuleNotFoundError: If the Module is not found or is not a valid
parser Module, then a ModuleNotFoundError
exception is raised.parse
def parse(
parser_mod_name: Union[str, module],
data: Union[str, bytes, Iterable[str]],
quiet: bool = False,
raw: bool = False,
ignore_exceptions: Optional[bool] = None,
**kwargs
) -> Union[Dict[str, Any], List[Dict[str, Any]], Iterator[Dict[str, Any]]]Parse the data (string or bytes) using the supplied parser (string or
module object).
This function provides a high-level API to simplify parser use. This
function will call built-in parsers and custom plugin parsers.
Example (standard parsers):
>>> import jc
>>> date_obj = jc.parse('date', 'Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022Example (streaming parsers):
>>> import jc
>>> ping_gen = jc.parse('ping_s', ping_output.splitlines())
>>> for item in ping_gen:
>>> print(f'Response time: {item["time_ms"]} ms')
Response time: 102 ms
Response time: 109 ms
...To get a list of available parser module names, use parser_mod_list().
Alternatively, a parser module object can be supplied:
>>> import jc
>>> jc_date = jc.get_parser('date')
>>> date_obj = jc.parse(jc_date, 'Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022You can also use the parser modules directly via get_parser():
>>> import jc
>>> jc_date = jc.get_parser('date')
>>> date_obj = jc_date.parse('Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022Finally, you can access the low-level parser modules manually:
>>> import jc.parsers.date
>>> date_obj = jc.parsers.date.parse('Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022Though, accessing plugin parsers directly is a bit more cumbersome, so
this higher-level API is recommended. Here is how you can access plugin
parsers without this API:
>>> import os
>>> import sys
>>> import jc.appdirs
>>> data_dir = jc.appdirs.user_data_dir('jc', 'jc')
>>> local_parsers_dir = os.path.join(data_dir, 'jcparsers')
>>> sys.path.append(local_parsers_dir)
>>> import my_custom_parser
>>> my_custom_parser.parse('command_data')Parameters:
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
A Module object can also be passed
directly or via `get_parser()`
data: (string or data to parse (string or bytes for
bytes or standard parsers, iterable of
iterable) strings for streaming parsers)
raw: (boolean) output preprocessed JSON if `True`
quiet: (boolean) suppress warning messages if `True`
ignore_exceptions: (boolean) ignore parsing exceptions if `True`
(streaming parsers only)Returns:
Standard Parsers: Dictionary or List of Dictionaries
Streaming Parsers: Generator Object containing Dictionariesparser_info
def parser_info(parser_mod_name: Union[str, module],
documentation: bool = False) -> jc.jc_types.ParserInfoTypeReturns a dictionary that includes the parser module metadata.
Parameters:
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name as well
as a parser module object.
documentation: (boolean) include parser docstring if `True`parser_mod_list
def parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]Returns a list of all available parser module names.
plugin_parser_mod_list
def plugin_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]Returns a list of plugin parser module names. This function is a
subset of parser_mod_list().
slurpable_parser_mod_list
def slurpable_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]Returns a list of slurpable parser module names. This function is a
subset of parser_mod_list().
standard_parser_mod_list
def standard_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]Returns a list of standard parser module names. This function is a
subset of parser_mod_list() and does not contain any streaming
parsers.
streaming_parser_mod_list
def streaming_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]Returns a list of streaming parser module names. This function is a
subset of parser_mod_list().
Readme
jc
JC - JSON Convert
This package converts the output of many standard unix command line tools
and file-types to dictionaries and lists of dictionaries.
Interactive Documentation
Using jc in your python programs:
>>> help('jc')
>>> help('jc.lib')
>>> jc.get_help('parser_module_name')Developing jc parsers:
>>> help('jc.utils')
>>> help('jc.streaming')
>>> help('jc.parsers.universal')Online Documentation
Latest
https://github.com/kellyjonbrazil/jc/tree/master/docs
Specific Version
https://github.com/kellyjonbrazil/jc/tree/v<full_version_number>/docs
Replace
<full_version_number>- e.g.1.18.0:
Specific versions can also be selected by tag in the Github branch dropdown
menu.
Usage Example
>>> import subprocess
>>> import jc
>>>
>>> cmd_output = subprocess.check_output(['dig', 'example.com'],
text=True)
>>> data = jc.parse('dig', cmd_output)
>>> data
[{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', ...}]Alternatively, you can bypass the high-level API and call the parser
modules directly:
>>> import subprocess
>>> import jc
>>>
>>> jc_dig = jc.get_parser('dig')
>>> cmd_output = subprocess.check_output(['dig', 'example.com'],
text=True)
>>> data = jc_dig.parse(cmd_output)
>>> data
[{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', ...}]or
>>> import subprocess
>>> import jc.parsers.dig
>>>
>>> cmd_output = subprocess.check_output(['dig', 'example.com'],
text=True)
>>> data = jc.parsers.dig.parse(cmd_output)
>>> data
[{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', ...}]Available Functions
Use help(jc.lib) for details.
parse
parse(
parser_module_name: str,
data: str | bytes | Iterable
) -> dict | list[dict] | Iterable[dict]High-level API to easily access the parser. This API will find both
built-in parsers and local plugin parsers.
get_parser
get_parser(
parser_module_name: str
) -> ModuleTypeGet a parser Module object so you can use it directly.
parser_info
parser_info(
parser_module_name: str,
documentation: bool = False
) -> dictGet the metadata for a particular parser.
all_parser_info
all_parser_info(documentation: bool = False) -> list[dict]Get the metadata for all parsers.
get_help
get_help(parser_module_name: str) -> NoneConvenience function to display the help screen for a parser using
its module name.
parser_mod_list
parser_mod_list() -> list[str]Get a list of all available parser module names to be used inparse(), parser_info(), and get_help().
plugin_parser_mod_list
plugin_parser_mod_list() -> list[str]Get a list of plugin parser module names to be used inparse(), parser_info(), and get_help(). This list is a subset ofparser_mod_list().
standard_parser_mod_list
standard_parser_mod_list() -> list[str]Get a list of standard parser module names to be used inparse(), parser_info(), and get_help(). This list is a subset ofparser_mod_list() and does not contain any streaming parsers.
streaming_parser_mod_list
streaming_parser_mod_list() -> list[str]Get a list of streaming parser module names to be used inparse(), parser_info(), and get_help(). This list is a subset ofparser_mod_list().
Streaming
jc.streaming
Table of Contents
jc - JSON Convert streaming utils
add_jc_meta
def add_jc_meta(func: ~F) -> ~FDecorator for streaming parsers to add stream_success andstream_error objects. This simplifies the yield lines in the
streaming parsers.
With the decorator on parse():
# successfully parsed line:
yield output_line if raw else _process(output_line)
# unsuccessfully parsed line:
except Exception as e:
yield raise_or_yield(ignore_exceptions, e, line)Without the decorator on parse():
# successfully parsed line:
if raw:
yield stream_success(output_line, ignore_exceptions)
else:
stream_success(_process(output_line), ignore_exceptions)
# unsuccessfully parsed line:
except Exception as e:
yield stream_error(raise_or_yield(ignore_exceptions, e, line))In all cases above:
output_line: (Dict) successfully parsed line yielded as a dict
e: (BaseException) exception object as the first value
of the tuple if the line was not successfully parsed.
line: (str) string of the original line that did not
successfully parse.
ignore_exceptions: (bool) continue processing lines and ignore
exceptions if `True`.raise_or_yield
def raise_or_yield(ignore_exceptions: bool, e: BaseException,
line: str) -> Tuple[BaseException, str]Return the exception object and line string if ignore_exceptions isTrue. Otherwise, re-raise the exception from the exception object with
an annotation.
stream_error
def stream_error(e: BaseException, line: str) -> Dict[str, Any]Return an error _jc_meta field.
stream_success
def stream_success(output_line: Dict[str, Any],
ignore_exceptions: bool) -> Dict[str, Any]Add _jc_meta object to output line if ignore_exceptions=True
streaming_input_type_check
def streaming_input_type_check(data: Iterable[Union[str, bytes]]) -> NoneEnsure input data is an iterable, but not a string or bytes. RaisesTypeError if not.
streaming_line_input_type_check
def streaming_line_input_type_check(line: str) -> NoneEnsure each line is a string. Raises TypeError if not.
Utils
jc.utils
Table of Contents
jc - JSON Convert utils
compatibility
def compatibility(mod_name: str,
compatible: List[str],
quiet: bool = False) -> NoneChecks for the parser's compatibility with the running OS platform and
prints a warning message to STDERR if not compatible andquiet=False.
Parameters:
mod_name: (string) __name__ of the calling module
compatible: (list) sys.platform name(s) compatible with
the parser. compatible options:
linux, darwin, cygwin, win32, aix, freebsd
quiet: (bool) suppress compatibility message if `True`Returns:
None - just prints output to STDERRconvert_size_to_int
def convert_size_to_int(size: str,
binary: bool = False,
posix_mode: bool = False,
decimal_bias: bool = False) -> Optional[int]Parse a human readable data size and return the number of bytes.
Parameters:
size: (string) The human readable file size to parse.
binary: (boolean) `True` to use binary multiples of bytes
(base-2) for ambiguous unit symbols and names,
`False` to use decimal multiples of bytes (base-10).
posix_mode: (boolean) Treat one-letter units (k, m, g, etc.) as
binary.
decimal_bias: (boolean) `True` to treat slightly ambiguous two-
letter unit symbols ending in "i" (e.g. Ki, Gi) to
use decimal multiples of bytes (base-10). `False`
(default) to use binary multiples of bytes.Returns:
integer/None Integer if successful conversion, otherwise NoneThis function knows how to parse sizes in bytes, kilobytes, megabytes,
gigabytes, terabytes and petabytes. Some examples:
>>> convert_size_to_int('42')
42
>>> convert_size_to_int('13b')
13
>>> convert_size_to_int('5 bytes')
5
>>> convert_size_to_int('1 KB')
1000
>>> convert_size_to_int('1 kilobyte')
1000
>>> convert_size_to_int('1 KiB')
1024
>>> convert_size_to_int('1 Ki')
1024
>>> convert_size_to_int('1 Ki', decimal_bias=True)
1000
>>> convert_size_to_int('1 KB', binary=True)
1024
>>> convert_size_to_int('1.5 GB')
1500000000
>>> convert_size_to_int('1.5 GB', binary=True)
1610612736convert_to_bool
def convert_to_bool(value: object) -> boolConverts string, integer, or float input to boolean by checking
for 'truthy' values.
Parameters:
value: (string/integer/float) Input valueReturns:
True/False False unless a 'truthy' number or string is found
('y', 'yes', 'true', '1', 1, -1, etc.)convert_to_float
def convert_to_float(value: object) -> Optional[float]Converts string and int input to float. Strips all non-numeric
characters from strings.
Parameters:
value: (string/integer) Input valueReturns:
float/None Float if successful conversion, otherwise Noneconvert_to_int
def convert_to_int(value: object) -> Optional[int]Converts string and float input to int. Strips all non-numeric
characters from strings.
Parameters:
value: (string/float) Input valueReturns:
integer/None Integer if successful conversion, otherwise Noneerror_message
def error_message(message_lines: List[str]) -> NonePrints an error message to STDERR for fatal issues. The first line is
prepended with 'jc: Error - ' and subsequent lines are indented.
Wraps text as needed based on the terminal width.
Parameters:
message: (list) list of string linesReturns:
None - just prints output to `STDERR`has_data
def has_data(data: Union[str, bytes]) -> boolChecks if the string input contains data. If there are any
non-whitespace characters then return True, else return False.
For bytes, returns True if there is any data.
Parameters:
data: (string, bytes) input to check whether it contains dataReturns:
Boolean `True` if input string (data) contains non-whitespace
characters, otherwise `False`. For bytes data, returns
`True` if there is any data, otherwise `False`.input_type_check
def input_type_check(data: object) -> NoneEnsure input data is a string. Raises TypeError if not.
is_compatible
def is_compatible(compatible: List[str]) -> boolReturns True if the parser is compatible with the running OS platform.
line_slice
def line_slice(
data: Union[str, Iterable[str], TextIO, bytes, NoneType],
slice_start: Optional[int] = None,
slice_end: Optional[int] = None
) -> Union[str, Iterable[str], TextIO, bytes, NoneType]Slice input data by lines - lazily, if possible.
Accepts a string (for normal parsers) or an iterable (for streaming
parsers). Uses normal start/stop slicing values, but will always slice
on lines instead of characters. Positive slices will use less memory as
the function will attempt to lazily iterate over the input. A negative
slice parameter will force the function to read in all of the data and
then slice, which will use more memory.
Parameters:
data: (string or iterable) - input to slice by lines
slice_start: (int) - starting line
slice_end: (int) - ending lineReturns:
string if input is a string.
iterable of strings if input is an iterable (for streaming parsers)
normalize_key
def normalize_key(data: str) -> strNormalize a key name by shifting to lower-case and converting special
characters to underscores.
Special characters are defined as space and the following:
!"#$%&'()*+,-./:;<=>?@[\]^`{|}~This is a lossy algorithm. Repeating and trailing underscores are
removed.
Parameters:
data: (string) Input valueReturns:
stringremove_quotes
def remove_quotes(data: str) -> strRemove single or double quotes surrounding a string. If no quotes are
found then the string is returned unmodified.
Parameters:
data: (string) Input valueReturns:
stringwarning_message
def warning_message(message_lines: List[str]) -> NonePrints warning message to STDERR for non-fatal issues. The first line
is prepended with 'jc: Warning - ' and subsequent lines are indented.
Wraps text as needed based on the terminal width.
Parameters:
message: (list) list of string linesReturns:
None - just prints output to STDERR