File: README.md
Archive Notice
The entire history of this repository is now contained in https://github.com/gitpython-developers/GitPython, where it is now maintained.
For issues, please use its issue tracker as well.
Motivation
When reading from many possibly large files in a fashion similar to random access, it is usually the fastest and most efficient to use memory maps.
Although memory maps have many advantages, they represent a very limited system resource as every map uses one file descriptor, whose amount is limited per process. On 32 bit systems, the amount of memory you can have mapped at a time is naturally limited to theoretical 4GB of memory, which may not be enough for some applications.
Limitations
- System resources (file-handles) are likely to be leaked! This is due to the library authors reliance on a deterministic
__del__()destructor. - The memory access is read-only by design.
Overview
Smmap wraps an interface around mmap and tracks the mapped files as well as the amount of clients who use it. If the system runs out of resources, or if a memory limit is reached, it will automatically unload unused maps to allow continued operation.
To allow processing large files even on 32 bit systems, it allows only portions of the file to be mapped. Once the user reads beyond the mapped region, smmap will automatically map the next required region, unloading unused regions using a LRU algorithm.
Although the library can be used most efficiently with its native interface, a Buffer implementation is provided to hide these details behind a simple string-like interface.
For performance critical 64 bit applications, a simplified version of memory mapping is provided which always maps the whole file, but still provides the benefit of unloading unused mappings on demand.
Prerequisites
- Python 3.7+
- OSX, Windows or Linux
The package was tested on all of the previously mentioned configurations.
Installing smmap
Its easiest to install smmap using the pip program:
$ pip install smmapAs the command will install smmap in your respective python distribution, you will most likely need root permissions to authorize the required changes.
If you have downloaded the source archive, the package can be installed by running the setup.py script:
$ python setup.py installIt is advised to have a look at the Usage Guide for a brief introduction on the different database implementations.
Homepage and Links
The project is home on github at https://github.com/gitpython-developers/smmap .
The latest source can be cloned from github as well:
- git://github.com/gitpython-developers/smmap.git
For support, please use the git-python mailing list:
Issues can be filed on github:
A link to the pypi page related to this repository:
License Information
smmap is licensed under the New BSD License.
File: doc/source/api.rst
.. _api-label:
#############
API Reference
#############
Mapped Memory Managers
.. automodule:: smmap.mman
:members:
:undoc-members:
Buffers
.. automodule:: smmap.buf
:members:
:undoc-members:
Utilities
.. automodule:: smmap.util
:members:
:undoc-members:
File: doc/source/changes.rst
#########
Changelog
#########
v5.0.3
- declare support for Python 3.13
For more, see https://github.com/gitpython-developers/smmap/compare/v5.0.3...v5.0.2
v5.0.2
- remove a usage of mktemp
v5.0.1
- Added support for Python 3.12
v6.0.0
YANKED
- Dropped support 3.6 and 3.7
- Declared support for Python 3.11 and 3.12
v5.0.0
- Dropped support 3.5
- Added support for Python 3.10
v4.0.0
- Dropped support for Python 2.7 and 3.4
- Added support for Python 3.7, 3.8, and 3.9
- Removed unused exc.MemoryManagerError and exc.RegionCollectionError
v3.0.5
- Restored Python 2 support removed in v3.0.2
- Changed release signature key to 27C50E7F590947D7273A741E85194C08421980C9.
See https://keybase.io/byronbates for proof of ownership.
v3.0.4
- Signed release (with correct key this time)
v3.0.2
- Signed release
- Switched to GitHub Actions for CI
v3.0.1
- Switched back to the smmap package name on PyPI and fixed the smmap2 mirror package
(#44 <https://github.com/gitpython-developers/smmap/issues/44>_) - Fixed setup.py
long_descriptionrendering
(#40 <https://github.com/gitpython-developers/smmap/pull/40>_)
v0.9.0
- Fixed issue with resources never being freed as mmaps were never closed.
- Client counting is now done manually, instead of relying on pyton's reference count
v0.8.5
- Fixed Python 3.0-3.3 regression, which also causes smmap to become about 3 times slower depending on the code path. It's related to this bug (http://bugs.python.org/issue15958), which was fixed in python 3.4
v0.8.4
- Fixed Python 3 performance regression
v0.8.3
- Cleaned up code and assured it works sufficiently well with python 3
v0.8.1
- A single bugfix
v0.8.0
- Initial Release
File: doc/source/index.rst
.. smmap documentation master file, created by
sphinx-quickstart on Wed Jun 8 15:14:25 2011.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.
Welcome to smmap's documentation!
smmap is a pure python implementation of a sliding memory map to help unifying memory mapped access on 32 and 64 bit systems and to help managing resources more efficiently.
Contents:
.. toctree::
:maxdepth: 2
intro
tutorial
api
changes
Indices and tables
- :ref:
genindex - :ref:
modindex - :ref:
search
File: doc/source/tutorial.rst
.. _tutorial-label:
###########
Usage Guide
###########
This text briefly introduces you to the basic design decisions and accompanying classes.
Design
Per application, there is MemoryManager which is held as static instance and used throughout the application. It can be configured to keep your resources within certain limits.
To access mapped regions, you require a cursor. Cursors point to exactly one file and serve as handles into it. As long as it exists, the respective memory region will remain available.
For convenience, a buffer implementation is provided which handles cursors and resource allocation behind its simple buffer like interface.
Memory Managers
There are two types of memory managers, one uses static windows, the other one uses sliding windows. A window is a region of a file mapped into memory. Although the names might be somewhat misleading as technically windows are always static, the sliding version will allocate relatively small windows whereas the static version will always map the whole file.
The static manager does nothing more than keeping a client count on the respective memory maps which always map the whole file, which allows to make some assumptions that can lead to simplified data access and increased performance, but reduces the compatibility to 32 bit systems or giant files.
The sliding memory manager therefore should be the default manager when preparing an application for handling huge amounts of data on 32 bit and 64 bit platforms::
import smmap
# This instance should be globally available in your application
# It is configured to be well suitable for 32-bit or 64 bit applications.
mman = smmap.SlidingWindowMapManager()
# the manager provides much useful information about its current state
# like the amount of open file handles or the amount of mapped memory
mman.num_file_handles()
mman.mapped_memory_size()
# and many more ...Cursors
Cursors are handles that point onto a window, i.e. a region of a file mapped into memory. From them you may obtain a buffer through which the data of that window can actually be accessed::
import smmap.test.lib
fc = smmap.test.lib.FileCreator(1024*1024*8, "test_file")
# obtain a cursor to access some file.
c = mman.make_cursor(fc.path)
# the cursor is now associated with the file, but not yet usable
assert c.is_associated()
assert not c.is_valid()
# before you can use the cursor, you have to specify a window you want to
# access. The following just says you want as much data as possible starting
# from offset 0.
# To be sure your region could be mapped, query for validity
assert c.use_region().is_valid() # use_region returns self
# once a region was mapped, you must query its dimension regularly
# to assure you don't try to access its buffer out of its bounds
assert c.size()
c.buffer()[0] # first byte
c.buffer()[1:10] # first 9 bytes
c.buffer()[c.size()-1] # last byte
# its recommended not to create big slices when feeding the buffer
# into consumers (e.g. struct or zlib).
# Instead, either give the buffer directly, or use pythons buffer command.
buffer(c.buffer(), 1, 9) # first 9 bytes without copying them
# you can query absolute offsets, and check whether an offset is included
# in the cursor's data.
assert c.ofs_begin() < c.ofs_end()
assert c.includes_ofs(100)
# If you are over out of bounds with one of your region requests, the
# cursor will be come invalid. It cannot be used in that state
assert not c.use_region(fc.size, 100).is_valid()
# map as much as possible after skipping the first 100 bytes
assert c.use_region(100).is_valid()
# You can explicitly free cursor resources by unusing the cursor's region
c.unuse_region()
assert not c.is_valid()Now you would have to write your algorithms around this interface to properly slide through huge amounts of data.
Alternatively you can use a convenience interface.
Buffers
To make first use easier, at the expense of performance, there is a Buffer implementation which uses a cursor underneath.
With it, you can access all data in a possibly huge file without having to take care of setting the cursor to different regions yourself::
# Create a default buffer which can operate on the whole file
buf = smmap.SlidingWindowMapBuffer(mman.make_cursor(fc.path))
# you can use it right away
assert buf.cursor().is_valid()
buf[0] # access the first byte
buf[-1] # access the last ten bytes on the file
buf[-10:]# access the last ten bytes
# If you want to keep the instance between different accesses, use the
# dedicated methods
buf.end_access()
assert not buf.cursor().is_valid() # you cannot use the buffer anymore
assert buf.begin_access(offset=10) # start using the buffer at an offset
# it will stop using resources automatically once it goes out of scopeDisadvantages
Buffers cannot be used in place of strings or maps, hence you have to slice them to have valid input for the sorts of struct and zlib. A slice means a lot of data handling overhead which makes buffers slower compared to using cursors directly.